Pith. sign in

REVIEW 4 major objections 6 minor 17 references

Efficient Deployment of Vision-Language Models on Mobile Devices: A Case Study on OnePlus 13R

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Mobile VLM deployment frameworks consistently run the whole inference pipeline on the CPU, leave the phone's GPU and NPU idle, and only a GPU-offloading runtime cuts power by an order of magnitude.

desk verdict Useful first cross-framework mobile VLM benchmark, but the latency numbers contradict each other and the device spec is ambiguous; worth engaging with major revisions. read the letter →

arxiv 2507.08505 v2 pith:ALR2RBT3 submitted 2025-07-11 cs.LG

Pablo Robin Guerrero , Yueyang Pan , Sanidhya Kashyap This is my paper · ORCID
classification cs.LG
keywords mobileVLMdeploymenton-deviceinferencehardwareutilizationCPUGPUNPUschedulingenergyefficiencyllama.cppMLC-Impmllm
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish how well today's frameworks can actually deploy vision-language models on a modern smartphone, benchmarking llama.cpp, MLC-Imp, and mllm on a OnePlus 13R running LLaVA-1.5 7B, MobileVLM-3B, and Imp-v1.5 3B. Its central finding is that in three of four configurations the entire pipeline, including the autoregressive decoder, runs on the ARM CPU cores, hitting 600–800 percent load, 10–12 W draw, and 88–95°C die temperatures, while the Adreno 740 GPU sits idle and the Hexagon NPU is never invoked. The exception, MLC-Imp running Imp-v1.5 3B, offloads image encoding, attention, and MLP layers to the GPU, dropping power to about 1.3 W and temperature to about 60°C while remaining latency-bound by one-token-at-a-time CPU decoding. The paper also claims that runtime scheduling decisions rival model size in importance: the same LLaVA-1.5 7B takes 82 seconds under llama.cpp but 174 seconds under mllm. If the results hold, efficient on-device multimodal inference is primarily an accelerator-mapping and scheduler problem rather than a model-compression problem.

What carries the argument

The load-bearing apparatus is the measurement and decomposition scheme: each inference is split into four stages, model load, image (slice) encoding, prompt evaluation, and token generation, with CPU, GPU, and NPU utilization sampled every 100 ms from root-accessible /proc and /sys files, aligned to vendor power-supply and thermal-zone sensors, and synchronized with screen recordings. Stage-level timing is what lets the paper attribute the bottleneck to GEMM-heavy prompt evaluation and GEMV-heavy sequential decoding, and the contrast between CPU-only runtimes (llama.cpp, mllm) and a GPU-offloading runtime (MLC-Imp) is what isolates the effect of accelerator mapping from model size. The identity of these four stages and the utilization traces carry the entire argument.

What would settle it

Re-run the same four model–framework configurations on a single documented OnePlus 13R while logging accelerator activity with the vendor's sub-millisecond hardware counters rather than 100 ms /proc samples: if the Adreno 740 or Hexagon NPU shows sustained busy time during prompt evaluation or token generation, the central accelerators-idle claim is refuted. A second decisive check is reproducibility on another SoC, because if the CPU-bound pattern and the MLC-Imp power drop (1.3 W versus 10–12 W) do not reproduce on a different chipset, the conclusions are device-specific rather than general.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that mobile VLM frameworks are bottlenecked by runtime scheduling and accelerator mapping, not by model size: every tested deployment leaves the Hexagon NPU unused, three of four leave the Adreno 740 mostly idle, and the CPU alone carries LLaVA-1.5 7B to 82–174 seconds of latency, 10–12 W of power, and an 88–95°C die plateau. When MLC-Imp routes Imp-v1.5 3B to the GPU, image encoding saturates the accelerator above 90 percent busy while the CPU stays under 120 percent load, power falls to about 1.3 W, and temperature peaks near 60°C, an order-of-magnitude energy reduction that the paper attributes to kernel offloading rather than model size. The paper further shows that framework-level choices can double or halve the speed of the identical model (82 s for llama.cpp versus 174 s for mllm on LLaVA-1.5 7B), and it names four optimization levers: GPU flash-attention and MLP offload, INT8 Hexagon decoding, mixed-precision KV caches, and overlapping image encoding with prompt evaluation.

Load-bearing premise

The accelerator-idle conclusion stands on the assumption that the custom monitoring scripts sampling /proc and /sys at 100 ms intervals, together with the vendor power and thermal sensors, accurately capture what the GPU and NPU are doing and what the device draws, a premise the paper never validates against independent counters.

Editorial extensions

If this is right

  • Moving attention and MLP kernels to the Adreno GPU at FP16 should roughly halve prompt-evaluation latency and cut power about four-fold, the paper estimates.
  • Offloading the final projection stage to the Hexagon NPU in INT8 could shed another 1–2 W from CPU-bound workloads.
  • Mixed-precision (FP8 or INT4) KV caches could reduce memory footprint by roughly 40 percent, while overlapping GPU image encoding with CPU prompt evaluation could hide 2–3 seconds of latency on mobile-scale models.
  • At one query per minute, a CPU-only LLaVA deployment would drain a 5000 mAh battery in about eight hours, whereas the GPU-offloaded Imp path would last nearly two days.
  • Since runtime scheduler and thread-affinity decisions produced a 2× latency gap on the identical model, framework engineering offers as much leverage as quantization and pruning for on-device VLMs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own hardware description is internally inconsistent: Section 3.3 names a Snapdragon 8 Gen 2 with 12 GB RAM, while the Limitations section, the Conclusion, and the usage plots reference a Snapdragon 8 Gen 3 with a 16 GB pool, so the quantitative results should be read against a confirmed device specification before being generalized.
  • The 100 ms sampling interval from /proc and /sys could miss short accelerator bursts, so a vendor profiler with sub-millisecond hardware counters is the natural check on whether the GPU is truly idle during the phases the paper reports as CPU-bound.
  • The total absence of NPU usage across all frameworks points to an open systems problem: a runtime that exposes the Hexagon NPU's INT8 compute to the sequential decoder would test the largest untapped accelerator in the paper's data.
  • The CPU-bound decoding pattern likely generalizes beyond Adreno hardware, since any SoC whose NPU lacks a mature GEMV path would show the same profile; the paper's four levers could thereby serve as a template for other vendor stacks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper reports a measurement study of three VLM deployment frameworks (llama.cpp, MLC-Imp, and mllm) running three vision-language models (LLaVA-1.5 7B, MobileVLM-3B, Imp-v1.5-3B) on a OnePlus 13R smartphone. The authors measure inference latency, CPU/GPU/NPU utilization, power, temperature, and user responsiveness, and claim that current frameworks consistently over-utilize the CPU during token generation while leaving the GPU and NPU largely idle, with only the MLC-Imp + Imp-v1.5-3B path successfully offloading compute to the Adreno GPU. The paper concludes that runtime scheduling and accelerator mapping, rather than model size alone, are the dominant factors in deployment efficiency, and proposes future optimizations such as GPU flash-attention, NPU decoding, mixed-precision KV caches, and stage overlap.

Significance. If the measurements are reliable, the paper provides a useful empirical data point: it compares multiple mobile VLM runtimes on the same device under system-level metrics, an area with little published data. The central observation that CPU-bound token generation dominates and that the NPU is entirely unused across all frameworks is practically actionable for runtime developers, and the reported power/thermal contrast between CPU-only and GPU-offloaded paths is striking. The authors also state an intent to release their profiling pipeline, which would support reproducibility. However, the quantitative support for these claims is currently undermined by internal inconsistencies and missing statistical detail, so the contribution cannot be fully assessed in its present form.

major comments (4)
  1. [§4.2, Table 2] The latency numbers in Table 2 are mutually inconsistent. For the llama.cpp + LLaVA-1.5 7B row, the listed stages sum to 150,809.3 ms, but the same row reports a Total of 82,194.9 ms, and §4.2 states "over 101 seconds total latency." The table also includes an "Image decoding" stage (63,724.0 ms) that is not defined in §4.2 or §3.5, which lists only load, slice-encode, prompt-eval, and token-eval phases. Since the paper's central comparison (prompt evaluation dominates; mllm doubles latency; CPU-only stacks exceed 100 s) is built on these numbers, the authors must reconcile the per-stage accounting with wall-clock totals, define all stages, and report the actual measured values consistently.
  2. [§3.3 vs §5.3 and §6] The hardware configuration is described inconsistently. Section 3.3 states the device is powered by a Snapdragon 8 Gen 2 with 12 GB of LPDDR5X RAM, while §5.3 and §6 say the phone is "Snapdragon 8 Gen 3-based," and §4.3 repeatedly refers to a "16 GB pool." If the same device was not used throughout, the cross-framework comparisons are invalid; if it was, the specification must be corrected. Please clarify the exact SoC and RAM configuration and ensure that all sections, including the figures, refer to the same hardware.
  3. [§3.5 and §4.3] Although §3.5 states each benchmark is repeated five times and averages are reported, no variance information appears anywhere in the paper. None of the tables or prose report standard deviations, min/max ranges, or per-run values. As a result, the reader cannot judge whether the headline differences (e.g., 82 s vs 174 s for LLaVA-1.5 7B, or 10 W vs 1.3 W) are significant relative to run-to-run noise. Please add error bars or ranges to all reported metrics, and describe the conditions that were held fixed versus those that varied across runs.
  4. [§3.5 and §4.3] The monitoring methodology is not sufficient to substantiate the central GPU/NPU idle conclusions. The scripts sample /proc and /sys at 100 ms intervals and rely on vendor power/thermal sensors, but the paper provides no validation that these counters accurately capture GPU and NPU activity, nor any discussion of whether 100 ms sampling could miss short accelerator bursts. The claim that "the Adreno 740 GPU remains at 0%" across entire runs is load-bearing and should be cross-checked with an independent profiler (e.g., Qualcomm's Snapdragon Profiler or GPU vendor counters). Please document exactly which files and counters were read, the sampling resolution, and any validation performed.
minor comments (6)
  1. [Abstract and §1] The abstract and introduction speak of "four representative runtimes" but then list only three frameworks (llama.cpp, MLC-Imp, and mllm); the conclusion later refers to "four mobile-oriented VLM stacks," which counts model-framework pairs. Please align the wording.
  2. [§4.2 and Table 3] Token counts are inconsistent: §4.2 reports token generation of 69 tokens for llama.cpp and 51 for mllm, while Table 3 lists average answer lengths of 30 tokens for both LLaVA stacks. If these are different quantities (e.g., staged timings versus final answer tokens), the distinction should be stated.
  3. [§4.3] The paragraph following Figure 3 contains a garbled sentence: "Taken together, Taken the phone remains entirely CPU-bound: ..." This appears to be an editorial leftover and should be cleaned up.
  4. [§5.2] In item (iii), "reduce memory usage by around 40" is missing the percent sign; it should read "by around 40%."
  5. [§4.4] The battery-life estimates ("one query per minute ... roughly eight hours ... versus almost two days") are presented without showing the underlying calculation. Please include the assumed energy per query and battery capacity so the estimate is reproducible.
  6. [Abstract and §1] The paper promises an open-source benchmarking pipeline and detailed profiling results, but no repository URL or availability statement appears in the manuscript. Please add a data/code availability section.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a direct measurement study whose conclusions are empirical observations, not derivations from fitted inputs or self-cited premises.

full rationale

The paper's central claims—CPU over-utilization during token generation, GPU/NPU underutilization, and GPU saturation during image encoding—are supported by direct profiling measurements of latency, utilization, power, and temperature. There is no fitted parameter that is later renamed as a prediction, no equation that reduces to its own input, and no uniqueness or ansatz imported through self-citation. The references to PowerInfer [3] and mllm-NPU [4] are background context, not load-bearing evidence, and the authors of those cited works do not overlap with the present authors in a way that makes the argument circular. The reproducibility statement about releasing the benchmarking pipeline is self-referential but is not used as evidence for any measured claim. The paper does contain internal inconsistencies—for example, Table 2's stage latencies for llama.cpp+LLaVA-1.5 7B sum to 150,809.3 ms while the row total is 82,194.9 ms and the prose reports 'over 101 seconds total latency', and the hardware description (Snapdragon 8 Gen 2, 12 GB) conflicts with the Limitations and Conclusion (Snapdragon 8 Gen 3, 16 GB pool referenced in utilization plots)—but these are correctness and reproducibility concerns about measurement reporting, not circular reasoning. A measurement artifact from 100 ms sampling or vendor counters would undermine validity, but it would not make the conclusion equivalent to its inputs by construction. No circular step can be quoted and exhibited, so the appropriate verdict is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters, fitted constants, or invented entities appear in this measurement study. The axioms listed are the domain assumptions about instrumentation accuracy, workload representativeness, and statistical adequacy that the conclusions depend on. The device specification inconsistency (Gen 2 vs Gen 3, 12 vs 16 GB) is another unverified premise, but it is captured in the weakest_assumption field rather than as a separate axiom.

assumptions (3)
  • domain assumption Root-accessible /proc and /sys counters plus vendor sensors give an accurate picture of CPU, GPU, and NPU utilization, power, and temperature at 100ms resolution.
    Section 3.5 describes this monitoring setup; the central conclusions about accelerator underutilization depend on these counters being correct and frequent enough to catch short bursts.
  • domain assumption The three selected models and the single-image prompt are representative of mobile VLM workloads.
    Section 3.2 selects LLaVA-1.5 7B, MobileVLM-3B, and Imp-v1.5-3B as representative; generalization to other VLMs and to multi-image or multilingual inputs is assumed rather than demonstrated.
  • domain assumption Five repeated runs without reported variance are sufficient to support the quantitative comparisons.
    Section 3.5 reports averages over five runs but gives no error bars; the paper's comparisons between frameworks rely on point estimates being stable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Deployment of Vision-Language Models on Mobile Devices: A Case Study on OnePlus 13R." pith.science (2026). https://pith.science/paper/ALR2RBT3

@misc{pith2026250708505,
  author       = {Pith},
  title        = {Pith review of: Efficient Deployment of Vision-Language Models on Mobile Devices: A Case Study on OnePlus 13R},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ALR2RBT3}},
  note         = {Machine review of arXiv:2507.08505}
}
read the original abstract

Vision-Language Models (VLMs) offer promising capabilities for mobile devices, but their deployment faces significant challenges due to computational limitations and energy inefficiency, especially for real-time applications. This study provides a comprehensive survey of deployment frameworks for VLMs on mobile devices, evaluating llama.cpp, MLC-Imp, and mllm in the context of running LLaVA-1.5 7B, MobileVLM-3B, and Imp-v1.5 3B as representative workloads on a OnePlus 13R. Each deployment framework was evaluated on the OnePlus 13R while running VLMs, with measurements covering CPU, GPU, and NPU utilization, temperature, inference time, power consumption, and user experience. Benchmarking revealed critical performance bottlenecks across frameworks: CPU resources were consistently over-utilized during token generation, while GPU and NPU accelerators were largely unused. When the GPU was used, primarily for image feature extraction, it was saturated, leading to degraded device responsiveness. The study contributes framework-level benchmarks, practical profiling tools, and an in-depth analysis of hardware utilization bottlenecks, highlighting the consistent overuse of CPUs and the ineffective or unstable use of GPUs and NPUs in current deployment frameworks.

Figures

Figures reproduced from arXiv: 2507.08505 by the authors.

Figure 1
Figure 1. CPU–GPU utilisation while running LLAVA-1.5 7B on a OnePlus 13R.Left: llama.cpp drives CPU load to ≈ 600 % (8 cores) for most of the 90 s run; the Adreno 740 stays idle.Right: mllm peaks even higher (≈ 750 %) yet still leaves the GPU unused.Shaded regions mark pipeline stages (slice-encode, load, prompt-eval, token-eval). Taken together, Taken the phone remains entirely CPU-bound: the GPU/NPU sit idle, CPU load peak… view at source ↗
Figure 2
Figure 2. CPU versus GPU utilisation while running [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. CPU versus GPU utilisation while running [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Peak package power (left axis) and die temperature (right axis) for each framework–model [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 7 canonical work pages

  1. [1]

    Densefusion-1m: Merging vision experts for comprehensive multimodal perception

    Kaiwen Zhou, Yujie Wang, Ke Li, Chen Qian, and Cewu Lu. Densefusion-1m: Merging vision experts for comprehensive multimodal perception. arXiv preprint arXiv:2407.08303, 2024

    arXiv 2024

  2. [2]

    Large multimodal agents: A survey.arXiv preprint arXiv:2402.15116, 2024

    Yujie Wang, Yujie Yu, Jing Liu, and Hanwang Wang. Large multimodal agents: A survey.arXiv preprint arXiv:2402.15116, 2024

    arXiv 2024

  3. [3]

    Powerinfer-2: Fast large language model inference on a smartphone

    Xiangning Chen, Yifan Zhang, Lian Xu, Yuhan Wang, Shiyao Li, Hanrui Liu, and Song Han. Powerinfer-2: Fast large language model inference on a smartphone. arXiv preprint arXiv:2406.06282, 2024

    arXiv 2024

  4. [4]

    Empowering 1000 tokens/second on-device llm prefilling with mllm-npu

    Weizhen Li, Wendi Zhou, Ying Xu, and Song Han. Empowering 1000 tokens/second on-device llm prefilling with mllm-npu. arXiv preprint arXiv:2407.05858, 2024

    arXiv 2024

  5. [5]

    Swapmoe: Serving off-the-shelf moe-based large language models with tunable memory budget

    Jianwei Zhao, Junnan Liu, Yao Cheng, Fuchun Yu, Lingjun Xie, and Mingxing Tan. Swapmoe: Serving off-the-shelf moe-based large language models with tunable memory budget. arXiv preprint arXiv:2308.15030, 2023

    arXiv 2023

  6. [6]

    llama.cpp: Efficient inference of llama models

    Georgi Gerganov. llama.cpp: Efficient inference of llama models. https://github.com/ ggerganov/llama.cpp, 2023. Accessed: 2024-06-05

    work page 2023

  7. [7]

    Zhou Yu. Mlc-imp. https://github.com/MILVLG/mlc-imp, 2024. Accessed: 2024-03-01

    work page 2024

  8. [8]

    mllm: On-device multimodal llm inference framework

    Ubiquitous Learning Lab. mllm: On-device multimodal llm inference framework. https: //github.com/UbiquitousLearning/mllm, 2024. Accessed: 2024-06-05

    work page 2024

Show all 17 references
  1. [9]

    Llava: Visual instruction tuning

    Haotian Liu, Chunyuan Zhang, Xizi Hu, Yuwei Wang, Yi Tay, Zhenyu Yang, Yichong Yu, Hangbo Li, Dale Schuurmans, Qu Gu, et al. Llava: Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2024

    2024 arXiv

  2. [10]

    Mobilevlm: An efficient vision-language model for mobile devices

    Haoyu Li, Yujie Yang, Yun Zhang, and Kai Tang. Mobilevlm: An efficient vision-language model for mobile devices. https://github.com/SCUTlihaoyu/MobileVLM, 2024. Ac- cessed: 2024-06-05

    2024

  3. [11]

    Imp: Highly capable large multimodal models for mobile devices

    Jianwei Zhang, Ziheng Wang, and Tianqi Chen. Imp: Highly capable large multimodal models for mobile devices. arXiv preprint arXiv:2405.12107, 2024

    2024 arXiv

  4. [12]

    Gptq: Accurate post-training quantization for generative pretrained transformers

    Elias Frantar, Pierre Stock, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pretrained transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  5. [13]

    Awq: Activation-aware weight quanti- zation for llms

    Ji Lin, Zhenyu Tang, Xiangning Zhang, and Song Han. Awq: Activation-aware weight quanti- zation for llms. arXiv preprint arXiv:2306.00978, 2023

    2023 arXiv

  6. [14]

    Learning both weights and connections for efficient neural networks

    Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural networks. In Advances in Neural Information Processing Systems (NIPS) , 2015

    2015

  7. [15]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  8. [16]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 10

    2022

  9. [17]

    Deepseek-vl 2: Mixture- of-experts vision-language models for advanced multimodal understanding

    Zirui Cao, Zhuohan Li, Haotian Zhang, Yuxuan Zhou, Yuxuan He, Yifan Zhang, Yu Huang, Yuxiao Zheng, Zihang Sun, Xu Han, Junkai Shi, Weijie Wang, Yutao Chen, Xin Wang, Liyuan Liu, Xiangning Lin, Puzhao Li, Ke Liu, Song Han, and Wei Dai. Deepseek-vl 2: Mixture- of-experts vision-...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.