Pith. sign in

REVIEW 3 major objections 6 minor 50 references

Mobile LLM efficiency is limited by framework offloading, a prefill–decode phase split, and host–accelerator scheduling waste—not raw NPU peak compute.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 12:13 UTC pith:RIJOGX7Y

load-bearing objection Solid first cross-layer NPU measurement for mobile LLMs; the phase-split and host-side waste results are real, but the 54.8% figure is an estimate from composed knobs, not a joint end-to-end run. the 3 major comments →

arxiv 2607.05475 v1 pith:RIJOGX7Y submitted 2026-07-06 cs.AR cs.AI

Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference

Guanyu Cai , Ruiming Tian , Lang Yang , Zhouhong Ren , Jinliang Yuan , Lingkun Li , Jiliang Wang This is my paper
classification cs.AR cs.AI
keywords Mobile LLMPerformance MeasurementEnergy EfficiencyNPUPowerBenchprefill-decode phase splitresource schedulingon-device inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Putting large language models on phones is attractive for privacy and latency, but the real limit is hardware inefficiency that prior device-level studies could not see. This paper runs the first cross-layer measurement across five mainstream inference frameworks and three backends (CPU, GPU, NPU), using a new tool, PowerBench, that attributes energy to each compute unit rather than the whole phone. It shows that framework choices (how work is offloaded, how operators scale, whether activations are quantized) open gaps of up to about 10–15 imes on NPUs; that NPUs dominate the compute-heavy prefill stage while CPUs win the memory-bound decode stage; and that host-side polling, NPU sleep timing, thread placement, and frequency defaults can waste tens of percent of energy even when arithmetic is offloaded. An energy-oriented NPU configuration is estimated to cut energy by up to roughly 55% on three real datasets. Anyone building private, always-on phone agents needs these numbers, because battery life and responsiveness hinge on them.

Core claim

Mobile LLM efficiency is jointly set by three layers: framework design (partial vs full-graph offload, operator scaling for large vs small shapes, and activation quantization), a sharp phase split in which NPUs excel at compute-bound prefilling but lose to CPUs on memory-bound decoding because static, fixed-shape NPU graphs clash with short, dynamic decode kernels and preallocated long contexts, and host–accelerator scheduling (RPC polling, NPU sleep latency, thread–core affinity, and DVFS) that can waste up to about 40% of energy. Backend-specific attribution via PowerBench makes these costs visible; an energy-oriented best-practice NPU setup is estimated to reduce energy by up to 54.8% acr

What carries the argument

PowerBench — a lightweight, framework-agnostic instrumentation library that maps Qualcomm power-zone counters to SoC, CPU, GPU, and NPU energy — is the mechanism that turns coarse device-level measurements into per-backend attribution, exposing host waste and phase-dependent rankings.

Load-bearing premise

The paper assumes that savings measured when tuning knobs one at a time (polling interval, NPU sleep latency, low fixed CPU frequency, affinity) add up to the reported ~54% end-to-end energy cuts on full datasets without large unmodeled interactions, thermal effects, or quality side effects.

What would settle it

Rerun the paper’s energy-optimized GENIE settings (20 µs RPC polling, 65535 µs NPU sleep, lowest fixed CPU frequency, full-graph QNN) on MathQA, RoleBench, and LongBench under the same thermal protocol and PowerBench rails; if total energy does not fall by roughly 53–55% versus default, or if quality degrades beyond the paper’s reported perplexity deltas, the composition claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Full-graph NPU offload with vendor MatMul kernels can beat partial offload and custom kernels by large factors in prefill while decode gaps shrink.
  • Runtimes should not bind an entire request to one backend: prefill belongs on the NPU and decode usually on the CPU (with GPU as a near-equivalent auxiliary).
  • Short RPC polling and longer NPU sleep, plus lowered host CPU frequency, recover tens of percent energy with little throughput loss on NPU paths.
  • Activation quantization can more than double throughput and cut energy by half or more, but can raise perplexity sharply on smaller models.
  • Default thread counts and DVFS leave large headroom; matching threads to cores and choosing mid or low host frequencies often improves energy for accelerator-bound work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • On-device agent stacks will need phase-aware multi-backend schedulers if battery life is to stay usable under continuous tool use.
  • Static-graph NPU toolchains likely need multi-graph context windows or dynamic shapes to close the decode gap without giving up prefill speed.
  • Marketing claims that NPUs are “energy efficient” are incomplete without host-side attribution; whole-device meters systematically hide the waste this work measures.
  • The same prefill–decode split and host-orchestrator overhead pattern is likely to appear on other mobile AI accelerators, not only the platforms studied here.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper presents a cross-layer measurement study of on-device LLM inference spanning five frameworks (llama.cpp, MNN, MLC-LLM, MLLM, GENIE) and three backends (CPU, GPU, NPU) on four Qualcomm Snapdragon phones. The authors introduce PowerBench, a QPT/powercap-based instrumentation library that attributes energy to SoC, CPU, GPU, and NPU rails rather than whole-device battery stats. From a controlled benchmark of 256-token prefill/decode (and related microbenchmarks), they report three main findings: (1) framework gaps are strongly amplified on NPU due to offloading granularity, MatMul scaling, and activation quantization; (2) a phase split in which NPUs dominate compute-bound prefill while CPUs (and often GPUs) win memory-bound decode, linked to static large-context NPU graphs; and (3) host–accelerator scheduling waste from RPC polling, NPU sleep latency, thread–core affinity, and DVFS, with per-knob energy savings of tens of percent. They then propose an energy-oriented NPU configuration (full-graph QNN offload, short RPC polling, long NPU sleep, low fixed CPU frequency) and estimate up to 54.8% energy reduction on MathQA, RoleBench, and LongBench.

Significance. If the measurement results hold, the paper fills a clear gap relative to prior on-device LLM studies that largely omit NPU paths, cross-framework NPU comparison, and backend-specific energy. PowerBench is a concrete methodological contribution that makes host-side waste under accelerator offload visible. The phase-split result and the operator-scaling / partial-offload explanations for NPU framework gaps are actionable for framework and runtime designers, and the scheduling knobs (polling, sleep, affinity, cross-backend DVFS) are underexplored for mobile NPU LLM inference. The work is empirical rather than theoretical; its value is breadth, controlled protocol, and fine-grained energy attribution. Strengths include multi-device/multi-model coverage, a documented experimental protocol (thermal floor, airplane mode, warm-up, repeats), and explicit labeling of the end-to-end savings as estimates rather than fully measured outcomes.

major comments (3)
  1. §8 and Table 8 present the flagship 53.6–54.8% energy reductions as composed effects of independently swept knobs (RPC polling 20 µs, NPU sleep 65535 µs, lowest fixed CPU frequency; §7.1–7.3, Figs. 9–13), applied to full dataset token streams. The text does not report a single joint PowerBench run that enables all knobs together on MathQA/RoleBench/LongBench under the same thermal protocol. Because polling, sleep, DVFS, and static context-window graphs can interact under growing KV cache and thermal load, the composed percentages remain extrapolations. Either measure the full configuration end-to-end on at least one dataset, or demote/qualify the abstract and §8 claim so that only per-knob measured savings are headline numbers and composition is clearly marked as an upper-bound estimate with stated independence assumptions.
  2. Tables 5 and 9 (and several figure-backed comparisons, e.g. GENIE 1463.7 vs llama.cpp 115.1 tokens/s NPU prefill) drive the central “framework gaps amplified on NPU” and phase-split claims, yet report only means with no trial variance, std, or min/max despite §3.4 stating ≥3 recorded trials. For gaps of 2×–15× this may not reverse rankings, but for closer decode comparisons (CPU vs GPU, NPU vs CPU) and for energy per token, missing dispersion weakens reproducibility and the strength of “CPU dominates decode.” Add error bars or variance columns for the primary throughput/energy cells used in the three insights.
  3. Finding 4 (§6.1, Fig. 7) attributes NPU decode weakness to large preallocated static graphs (e.g., context 4096). The recommended best practice in §8 still “build[s] a computation graph with a context length that is suitable for the target workload” without specifying how that length is chosen for mixed short/long traces, nor quantifying residual overhead when the graph is oversized relative to actual context. Given that Table 8’s LongBench vs MathQA/RoleBench latency signs differ partly by prefill/decode mix, clarify the context-window policy used in the estimate and, if possible, report sensitivity of energy/throughput to graph size under the tuned polling/sleep settings.
minor comments (6)
  1. Abstract claims framework gaps “reaching up to 10×” on NPU; §5.2 reports GENIE vs llama.cpp prefill as over 15× (1463.7 vs 115.1 tokens/s). Align the abstract with the measured maximum or state the comparison basis.
  2. Table 5 omits several rows present in appendix Table 9 (e.g., CPU GENIE); caption “Averaged Throughput and Energy” is slightly misleading if values are per-device means of trials rather than averages across devices. Clarify aggregation.
  3. All devices are Qualcomm Snapdragon with QNN/Hexagon. §3.1 justifies this for framework coverage, but the title and abstract generalize to “Your NPU.” A short limitations paragraph on vendor specificity (MediaTek/Apple NPU stacks) would set scope cleanly.
  4. Activation quantization (§5.3, Table 7) shows large PPL degradation on Llama-3.2-1B (+30.36%) vs small on 3B. The energy best-practice path emphasizes full-graph QNN and scheduling; state whether recommended deployments assume W4 or W4A16 and whether accuracy is in scope for the 54.8% estimate.
  5. Figure 1 and some multi-panel figures (e.g., Fig. 10 heatmaps) are dense; ensure axis labels and “Default setting” markers remain legible in print. Fig. 3’s “llama-shape / qwen-shape” should define N and the exact MatMul dimensions in the caption.
  6. Minor consistency: abstract “up to 40% energy waste” vs body ranges (30.9–37.8% polling, 44.6–50.9% sleep). Prefer one consistent summary range tied to a cited figure.

Circularity Check

0 steps flagged

No circularity: empirical measurement paper whose energy/throughput claims are instrumented, not definitionally forced by fitted parameters or self-citation.

full rationale

This is a systems measurement study. Prefill/decode throughput is defined from wall-clock token timing under a standardized harness; SoC/CPU/GPU/NPU energy is read from Qualcomm PMIC counters via PowerBench. The three main findings (framework gaps on NPU, prefill–decode phase split, scheduling waste) are comparative observations across frameworks, backends, and knobs, not algebraic derivations of a target quantity from parameters that already encode that quantity. The flagship 53.6–54.8% energy reduction is explicitly labeled an estimate obtained by applying independently swept knobs (RPC polling, NPU sleep, CPU frequency, affinity) to dataset token mixes (Table 8 / §8); that is an extrapolation risk, not circularity by construction. There is no uniqueness theorem, no self-definitional scale, no fitted parameter renamed as a prediction, and no load-bearing self-citation chain that forces the result. Self-citations to related mobile-LLM work are ordinary background and are not used to prove the measurements. Score 0 is therefore the correct outcome.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

As a systems measurement study, the load-bearing content is empirical protocol and instrumentation fidelity rather than free theoretical parameters. The main unproved inputs are trust in Qualcomm powercap/QPT counters, representativeness of the chosen phones/frameworks/models, and the composition of microbenchmark savings into dataset-level energy estimates. No new physical entities are postulated.

free parameters (4)
  • RPC polling interval (energy-optimal setting) = 20 µs
    Chosen from a discrete sweep (5–65535 µs); best-practice uses 20 µs. Not a fit to a scientific law, but a hand-selected operating point that the headline energy claim depends on.
  • NPU sleep latency (energy-optimal setting) = 65535 µs
    Chosen from a discrete sweep; best-practice uses 65535 µs versus framework defaults (~40 µs). Directly drives the large NPU energy reductions claimed in §7–8.
  • CPU frequency level under NPU execution = lowest fixed level
    Fixed to the lowest level in the energy-oriented configuration; savings vs default DVFS are measured, not predicted from a model.
  • Static NPU context window / graph shape = large fixed context (e.g. 4096)
    Decode graphs preallocate a large context (e.g., 4096). This modeling choice is central to the claimed NPU decode weakness and residual headroom.
axioms (5)
  • domain assumption Qualcomm Power Telemetry / powercap counters accurately attribute cumulative energy to SoC, CPU clusters, GPU, and NPU rails for the measured intervals.
    §3.3 bases all backend-specific energy claims on these counters; no independent validation against external power meters is reported in the text.
  • domain assumption Qualcomm Snapdragon phones with QNN/Hexagon NPU paths are representative enough of mobile LLM NPU behavior for the stated conclusions.
    §3.1 restricts the testbed to four Qualcomm devices; MediaTek/Apple NPUs are discussed only via peak TOPS tables, not measured.
  • domain assumption Prefill is predominantly compute-bound GEMM and decode is predominantly memory-bound GEMV/KV-cache traffic under the studied models and lengths.
    §2.1 and §6 use this phase model to explain backend ranking reversal; it is standard but still an assumption about the measured regimes.
  • ad hoc to paper Default high-performance framework settings plus the authors’ standardized token injection protocol yield fair cross-framework comparisons.
    §3.2–3.4 and §4; fairness depends on harness choices that frameworks do not all expose identically.
  • ad hoc to paper Independent knob savings (polling, sleep, affinity, DVFS) can be combined into the Table 8 end-to-end energy estimates without large antagonistic interactions.
    §8 presents estimated dataset-level impact from the composed best-practice configuration.
invented entities (1)
  • PowerBench no independent evidence
    purpose: Framework-agnostic instrumentation library for backend-specific energy deltas and throughput-standardized runs.
    Introduced as a software tool, not a physical postulate; independent evidence would be public code and external validation of its energy attribution.

pith-pipeline@v1.1.0-grok45 · 34844 in / 3704 out tokens · 35445 ms · 2026-07-11T12:13:53.128386+00:00 · methodology

0 comments
read the original abstract

Deploying Large Language Models (LLMs) on mobile devices enhances privacy and reduces latency, but is severely bottlenecked by hardware inefficiency. We present the first comprehensive, cross-layer measurement study of mobile LLM inference, uniquely spanning five mainstream frameworks (e.g., llama.cpp, GENIE) and three hardware backends (CPU, GPU, NPU). To enable this analysis, we develop PowerBench, a fine-grained profiling tool that provides the first backend-specific energy attribution, moving beyond traditional device-level measurements. Our study yields three critical insights: (1) Framework-induced performance gaps are substantially amplified on NPUs, reaching up to 10x using custom operators due to divergent offloading and quantization strategies. (2) We identify a distinct phase split where NPUs excel at compute-bound prefilling, while CPUs outperform all other backends in memory-bound decoding. This is driven by the NPU's preference for large, fixed-shape workloads, which conflicts with the small-kernel, dynamic nature of decoding. (3) Backend-specific profiling uncovers substantial scheduling headroom missed by prior work. Suboptimal thread configurations, uncoordinated NPU sleep latencies, and CPU polling intervals result in up to 40% energy waste. Leveraging these findings, we present an energy-oriented best-practice configuration for mobile LLM inference. We estimate that this configuration could reduce energy consumption by up to 54.8% on the NPU backend across three datasets.

Figures

Figures reproduced from arXiv: 2607.05475 by Guanyu Cai, Jiliang Wang, Jinliang Yuan, Lang Yang, Lingkun Li, Ruiming Tian, Zhouhong Ren.

Figure 1
Figure 1. Figure 1: Overview of our cross-layer measurement. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of our evaluation infrastructure. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of matmul cycles between the QNN implementation used by GENIE and llama.cpp. 5 RQ1: Framework Gaps Mobile inference frameworks share a common pipeline, in￾cluding graph conversion, quantization, backend selection, operator offloading, and runtime optimization. The bench￾mark overview in Section 4 shows that framework gaps are particularly large on NPUs, especially in prefill. We find three major… view at source ↗
Figure 4
Figure 4. Figure 4: The impact of activation quantization on [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Operator speedup breakdown for W4 vs W4A16 of Llama-3.2-3B. Percentages denote each op￾erator’s contribution to the total compute cycles. The speedup is concentrated rather than uniform [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: CPU and GPU Decode Throughput Across Devices for Each Model [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Decode throughput of the QNN NPU backend [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Backend decode energy on Xiaomi 17 with Qwen2.5-1.5B. shortens the time the NPU remains unnecessarily active, while the extra host polling overhead is very small. NPU sleep latency [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Impact of CPU–NPU invocation parameters on throughput and energy consumption. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Impact of CPU thread masking on throughput (top row) and system energy (bottom row) across backends [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Throughput and energy under frequency scaling in llama.cpp for Llama 3.2 1B decode with 256 tokens. [PITH_FULL_IMAGE:figures/full_fig_p011_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Impact of NPU voltage-corner scaling on throughput and energy during inference. [PITH_FULL_IMAGE:figures/full_fig_p012_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Impact of CPU frequency scaling on through [PITH_FULL_IMAGE:figures/full_fig_p012_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 18 linked inside Pith

  1. [1]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tai, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Sid- dhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro- Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gau- rav Mishra, Adams Yu, Vincent Zhao, Yanp...

    2024

  2. [2]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. InProceedings of NeurIPS, 2023

    2023

  3. [3]

    On-Device Language Models: A Comprehensive Review

    Jiajun Xu, Zhiyuan Li, Wei Chen, Qun Wang, Xin Gao, Qi Cai, and Ziyuan Ling. On-Device Language Models: A Comprehensive Review. arXiv preprint arXiv:2409.00088, September 2024

    Pith/arXiv arXiv 2024

  4. [4]

    AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation.arXiv preprint arXiv:2412.18116, May 2025

    Hao Wen, Shizuo Tian, Borislav Pavlov, Wenjie Du, Yixuan Li, Ge Chang, Shanhui Zhao, Jiacheng Liu, Yunxin Liu, Ya-Qin Zhang, and Yuanchun Li. AutoDroid-V2: Boosting SLM-based GUI Agents via Code Generation.arXiv preprint arXiv:2412.18116, May 2025

    Pith/arXiv arXiv 2025

  5. [5]

    AutoDroid: LLM-powered Task Automation in Android

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. AutoDroid: LLM-powered Task Automation in Android. InProceedings of ACM MobiCom, 2024

    2024

  6. [6]

    Fast On-device LLM Inference with NPUs

    Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Gang Huang, Meng- wei Xu, and Xuanzhe Liu. Fast On-device LLM Inference with NPUs. arXiv preprint arXiv:2407.05858, December 2024

    Pith/arXiv arXiv 2024

  7. [7]

    PowerInfer-2: Fast large language model inference on a smartphone.arXiv preprint arXiv:2406.06282, 2024

    Zhenliang Xue, Yixin Song, Zeyu Mi, Xinrui Zheng, Yubin Xia, and Haibo Chen. PowerInfer-2: Fast large language model inference on a smartphone.arXiv preprint arXiv:2406.06282, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices .IEEE Transactions on Mobile Computing, 24(08): 7059–7073, August 2025

    Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices .IEEE Transactions on Mobile Computing, 24(08): 7059–7073, August 2025

    2025

  9. [9]

    Characterizing Mobile SoC for Accelerat- ing Heterogeneous LLM Inference

    Le Chen, Dahu Feng, Erhu Feng, Yingrui Wang, Rong Zhao, Yubin Xia, Pinjie Xu, and Haibo Chen. Characterizing Mobile SoC for Accelerat- ing Heterogeneous LLM Inference. InProceedings of the ACM SOSP, 2025

    2025

  10. [10]

    MNN-AECS: Energy Optimization for LLM Decoding on Mobile Devices via Adaptive Core Selection.arXiv preprint arXiv:2506.19884, June 2025

    Zhengxiang Huang, Chaoyue Niu, Zhaode Wang, Jiarui Xue, Hanming Zhang, Yugang Wang, Zewei Xin, Xiaotang Jiang, Chengfei Lv, Fan Wu, and Guihai Chen. MNN-AECS: Energy Optimization for LLM Decoding on Mobile Devices via Adaptive Core Selection.arXiv preprint arXiv:2506.19884, June 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Charlie Hu, Qiang Xu, Jian Li, and Haibing Guan

    Zongpu Zhang, Pranab Dash, Y. Charlie Hu, Qiang Xu, Jian Li, and Haibing Guan. Dissecting the Impact of Mobile DVFS Governors on LLM Inference Performance and Energy Efficiency.arXiv preprint arXiv:2507.02135, July 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Lane, and Mengwei Xu

    Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Wei Liu, Jian Luan, Xiwen Zhang, Nicholas D. Lane, and Mengwei Xu. Demys- tifying Small Language Models for Edge Deployment. InProceedings of ACL, 2025

    2025

  13. [13]

    Large Language Models on Mobile Devices: Measurements, Analysis, and Insights

    Xiang Li, Zhenyan Lu, Dongqi Cai, Xiao Ma, and Mengwei Xu. Large Language Models on Mobile Devices: Measurements, Analysis, and Insights. InProceedings of the Workshop on Edge and Mobile Foundation Models, 2024

    2024

  14. [14]

    Large Language Models on Mobile Devices: A Measurement Study of Single- and Multi-Instance Execution

    Qingzhe Guo, Tu Ouyang, and An Wang. Large Language Models on Mobile Devices: A Measurement Study of Single- and Multi-Instance Execution. InProceedings of the 2nd International Workshop on Edge and Mobile Foundation Models, 2025

    2025

  15. [15]

    MELTing Point: Mobile Evaluation of Language Transform- ers

    Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto, and Hamed Haddadi. MELTing Point: Mobile Evaluation of Language Transform- ers. InProceedings of ACM MobiCom, 2024

    2024

  16. [16]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InProceedings of NeurIPS, 2017

    2017

  17. [17]

    GQA: Training generalized multi-query transformer models from multi-head checkpoints

    Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of EMNLP, 2023

    2023

  18. [18]

    Splitwise: Efficient gen- erative llm inference using phase splitting

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, Íñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient gen- erative llm inference using phase splitting. InProceedings of ISCA, 2025

    2025

  19. [19]

    URL https: //nanoreview.net/en/soc-list/rating

    Smartphone Processors Ranking List [2025] - NanoReview. URL https: //nanoreview.net/en/soc-list/rating. Guanyu Cai, Ruiming Tian, Lang Yang, Zhouhong Ren, Jinliang Yuan, Lingkun Li, and Jiliang Wang

    2025

  20. [20]

    Qualcomm AI Engine Direct SDK | Qualcomm Developer,

    Qualcomm Technologies, Inc. Qualcomm AI Engine Direct SDK | Qualcomm Developer, . URL https://www.qualcomm.com/developer/ software/qualcomm-ai-engine-direct-sdk

  21. [21]

    Hexagon NPU SDK | Qualcomm Devel- oper,

    Qualcomm Technologies, Inc. Hexagon NPU SDK | Qualcomm Devel- oper, . URL https://www.qualcomm.com/developer/software/hexagon- npu-sdk

  22. [22]

    llama.cpp, 2023

    Georgi Gerganov. llama.cpp, 2023. URL https://github.com/ggml- org/llama.cpp

    2023

  23. [23]

    Mnn: A universal and efficient inference engine

    Xiaotang Jiang, Huan Wang, Yiliu Chen, Ziqi Wu, Lichuan Wang, Bin Zou, Yafeng Yang, Zongyang Cui, Yu Cai, Tianhang Yu, Chengfei Lv, and Zhihua Wu. Mnn: A universal and efficient inference engine. arXiv preprint arXiv:2002.12418, 2020

    Pith/arXiv arXiv 2002

  24. [24]

    Mlc-llm, 2023

    MLC Team. Mlc-llm, 2023. URL https://github.com/mlc-ai/mlc-llm

    2023

  25. [25]

    mllm: Fast multimodal llm on mobile devices,

    Ruyi Yi, Xiangyu Li, et al. mllm: Fast multimodal llm on mobile devices,

  26. [26]

    URL https://github.com/UbiquitousLearning/mllm

  27. [27]

    Gen ai inference extensions

    Qualcomm Technologies, Inc. Gen ai inference extensions. https://www.qualcomm.com/developer/software/gen-ai-inference- extensions, 2026. Accessed: 2026-03-14

    2026

  28. [28]

    EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding.IEEE Transactions on Mobile Computing, 24(4):3256–3273, April 2025

    Daliang Xu, Wangsong Yin, Hao Zhang, Xin Jin, Ying Zhang, Shiyun Wei, Mengwei Xu, and Xuanzhe Liu. EdgeLLM: Fast On-Device LLM Inference With Speculative Decoding.IEEE Transactions on Mobile Computing, 24(4):3256–3273, April 2025

    2025

  29. [29]

    android_kernel_oneplus_sm8850: Qualcomm power telemetry and powercap driver sources

    OnePlusOSS. android_kernel_oneplus_sm8850: Qualcomm power telemetry and powercap driver sources. https: //github.com/OnePlusOSS/android_kernel_oneplus_sm8850/tree/ 6504e3d0385a951de4848bc81ce19ce8f8145dbe/drivers/powercap/ qcom, 2026. Commit 6504e3d0385a951de4848bc81ce19ce8f8145dbe, accessed 2026-03-13

    2026

  30. [30]

    Scaling LLM Test-Time Compute with Mobile NPU on Smartphones.arXiv preprint arXiv:2509.23324, September 2025

    Zixu Hao, Jianyu Wei, Tuowei Wang, Minxing Huang, Huiqiang Jiang, Shiqi Jiang, Ting Cao, and Ju Ren. Scaling LLM Test-Time Compute with Mobile NPU on Smartphones.arXiv preprint arXiv:2509.23324, September 2025

    arXiv 2025

  31. [31]

    MathQA: Towards interpretable math word problem solving with operation-based formalisms

    Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. InPro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long ...

    2019

  32. [32]

    Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models

    Noah Wang, Zy Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Jian Yang, et al. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. InFindings of the Association for Computa- tional Linguistics: ACL 2024, 2024

    2024

  33. [33]

    Longbench: A bilingual, multitask benchmark for long context under- standing

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhid- ian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context under- standing. InProceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), 2024

    2024

  34. [34]

    Mo- bileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases

    Rithesh Murthy, Liangwei Yang, Juntao Tan, Tulika Manoj Awal- gaonkar, Yilun Zhou, Shelby Heinecke, Sachin Desai, Jason Wu, Ran Xu, Sarah Tan, Jianguo Zhang, Zhiwei Liu, Shirley Kokane, Zuxin Liu, Ming Zhu, Huan Wang, Caiming Xiong, and Silvio Savarese. Mo- bileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases. arXiv preprint arXiv:2406.10290, June 2024

    Pith/arXiv arXiv 2024

  35. [35]

    lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Models

    Haoxin Wang, Xiaolong Tu, Hongyu Ke, Huirong Chai, Dawei Chen, and Kyungtae Han. lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Models. InProceedings of ACM/IEEE SEC, 2025

    2025

  36. [36]

    GPTQ: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, October 2022

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers.arXiv preprint arXiv:2210.17323, October 2022

    Pith/arXiv arXiv 2022

  37. [37]

    AWQ: Activation-aware weight quantization for llm compression and acceleration.arXiv preprint arXiv:2306.00978, June 2023

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei- Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-aware weight quantization for llm compression and acceleration.arXiv preprint arXiv:2306.00978, June 2023

    Pith/arXiv arXiv 2023

  38. [38]

    SparseGPT: Massive language models can be accurately pruned in one-shot.arXiv preprint arXiv:2301.00774, January 2023

    Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot.arXiv preprint arXiv:2301.00774, January 2023

    Pith/arXiv arXiv 2023

  39. [39]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with io-awareness.arXiv preprint arXiv:2205.14135, May 2022

    Pith/arXiv arXiv 2022

  40. [40]

    FlashAttention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, July 2023

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning.arXiv preprint arXiv:2307.08691, July 2023

    Pith/arXiv arXiv 2023

  41. [41]

    Flashinfer: Efficient and customizable attention engine for llm inference serving.Proceedings of Machine Learning and Systems, 7, 2025

    Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, et al. Flashinfer: Efficient and customizable attention engine for llm inference serving.Proceedings of Machine Learning and Systems, 7, 2025

    2025

  42. [42]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention.arXiv preprint arXiv:2309.06180, September 2023

    Pith/arXiv arXiv 2023

  43. [43]

    Gonzalez, Ion Stoica, and Matei Zaharia

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, Joseph E. Gonzalez, Ion Stoica, and Matei Zaharia. H 2O: Heavy-hitter oracle for efficient generative inference of large language models.arXiv preprint arXiv:2306.14048, June 2023

    Pith/arXiv arXiv 2023

  44. [44]

    Fast inference from transformers via speculative decoding.arXiv preprint arXiv:2211.17192, November 2022

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding.arXiv preprint arXiv:2211.17192, November 2022

    Pith/arXiv arXiv 2022

  45. [45]

    PowerInfer: Fast large language model serving with a consumer-grade GPU

    Yixin Song, Zeyu Mi, Haotong Xie, and Haibo Chen. PowerInfer: Fast large language model serving with a consumer-grade GPU. In Proceedings of the ACM SOSP, 2024

    2024

  46. [46]

    WIP: Efficient LLM prefilling with mobile NPU

    Daliang Xu, Hao Zhang, Liming Yang, Ruiqi Liu, Mengwei Xu, and Xuanzhe Liu. WIP: Efficient LLM prefilling with mobile NPU. In Proceedings of the Workshop on Edge and Mobile Foundation Models, 2024

    2024

  47. [47]

    m 22LLM: A Multi-Dimensional Optimization Framework for LLM Inference on Mobile Devices .IEEE Transactions on Parallel & Distributed Systems, 36(10):2014–2029, Oc- tober 2025

    Kaiyuan Liu, Xiaobo Zhou, and Li Li. m 22LLM: A Multi-Dimensional Optimization Framework for LLM Inference on Mobile Devices .IEEE Transactions on Parallel & Distributed Systems, 36(10):2014–2029, Oc- tober 2025

    2014

  48. [48]

    Mobile foundation model as firmware

    Jinliang Yuan, Chen Yang, Dongqi Cai, Shihe Wang, Xin Yuan, Zeling Zhang, Xiang Li, Dingge Zhang, Hanzi Mei, Xianqing Jia, Shangguang Wang, and Mengwei Xu. Mobile foundation model as firmware. In Proceedings of ACM MobiCom, 2024

    2024

  49. [49]

    LP-Spec: Lever- aging LPDDR PIM for efficient LLM mobile speculative inference with architecture-dataflow co-optimization.arXiv preprint arXiv:2508.07227, August 2025

    Siyuan He, Zhantong Zhu, Yandong He, and Tianyu Jia. LP-Spec: Lever- aging LPDDR PIM for efficient LLM mobile speculative inference with architecture-dataflow co-optimization.arXiv preprint arXiv:2508.07227, August 2025

    Pith/arXiv arXiv 2025

  50. [50]

    Dynamic sparse attention on mobile SoCs.arXiv preprint arXiv:2508.16703, 2025

    Wangsong Yin, Daliang Xu, Mengwei Xu, Gang Huang, and Xu- anzhe Liu. Dynamic sparse attention on mobile SoCs.arXiv preprint arXiv:2508.16703, 2025. Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference Table 9: Averaged Throughput and Energy across Devices, Models, Frameworks, and Backends with 256 tokens. Model ...

    Pith/arXiv arXiv 2025