Pith. sign in

REVIEW 3 major objections 5 minor 56 references

ConsumerBench: Benchmarking Generative AI Applications on End-User Devices

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ConsumerBench argues that benchmarking generative AI on end-user devices must include concurrent multi-application interference, since exclusive-access benchmarks miss the starvation and underutilization that dominate real use.

desk verdict A genuinely useful and open-sourced benchmark for concurrent on-device GenAI, with plausible headline findings that are presented without sensitivity analysis; deserves peer review after a robustness pass. read the letter →

arxiv 2506.17538 v1 pith:AUPHFLQN submitted 2025-06-21 cs.DC cs.AIcs.LGcs.OS

Yile Gu , Rohan Kadekodi , Hoang Nguyen , Keisuke Kamahori , Yiyu Liu , Baris Kasikci This is my paper
classification cs.DCcs.AIcs.LGcs.OS
keywords generativeAIbenchmarkingend-userdevicesconcurrentGPUworkloadsservicelevelobjectivesresourcesharingon-deviceinferencemulti-applicationworkflowsSLOattainment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ConsumerBench argues that current GenAI benchmarks, which run one model alone on a dedicated GPU, do not represent the end-user device experience, where chatbots, captioning, and image generation compete for a single GPU. The paper introduces a configurable framework that turns a user's application list, request patterns, SLOs, and dependencies into a workflow DAG (a directed acyclic graph of dependent tasks), executes the applications concurrently, and reports both per-application SLO attainment and system-level utilization. In experiments on a consumer GPU, the framework finds that greedy resource allocation lets image-generation kernels starve live-captioning's small kernels, raising average end-to-end caption latency by 12.4x and causing almost all caption requests to miss their 2-second target. Static GPU partitioning removes that starvation but reserves GPU slices rigidly, underutilizing the device and causing image generation to miss its SLO even though spare compute exists. The authors conclude that on-device GenAI systems need SLO-aware scheduling, dynamic resource partitioning, and kernels designed for consumer GPU architectures.

What carries the argument

The machinery is a DAG-based execution engine: a user configuration is compiled into a directed acyclic graph whose nodes are application setup, execution, and cleanup, and whose edges encode dependencies; an orchestrator then runs the graph under a chosen resource-sharing strategy. For each application the framework measures SLO attainment on latency metrics (time to first token and time per output token for the chatbot, per-denoising-step time for image generation, per-segment time for live captions) and compares them against user-set thresholds. A system monitor records how much of the GPU is reserved (SMACT) versus actually occupied (SMOCC), which is the diagnostic that exposes low-occupancy kernels in image generation and caption decoding. The contrast between greedy allocation and static partitioning is the experiment that carries the main argument: it isolates starvation (greedy) from underutilization (static partitioning) as the two failure modes of resource sharing on end-user devices.

What would settle it

Run the same three-application concurrent workload on the same GPU but change one plausibly realistic parameter—for example, set LiveCaptions' SLO to 4 seconds per segment (the value the paper itself uses on Apple Silicon) while keeping the 2-second arrival rate—and count SLO misses under greedy allocation; if LiveCaptions then meets most SLOs, the starvation conclusion is an artifact of the chosen threshold. Alternatively, replace ImageGen's generic attention kernel with a low-register custom kernel that raises SM occupancy and check whether LiveCaptions' decode slowdown disappears under greedy allocation.

Watch

Extended reading notes

Core claim

The central claim is that the performance of GenAI on end-user devices can only be understood under concurrent, multi-application execution, and that naive resource-sharing strategies fail the SLOs users actually experience. With three latency-sensitive applications sharing one GPU, the paper shows that greedy first-come-first-served allocation is unfair: LiveCaptions' decoder phase runs 30x slower than under exclusive access because its small kernels wait behind ImageGen's large kernels, producing a 12.4x increase in average request latency and near-total SLO misses. Static GPU partitioning, by contrast, prevents the stalling but rigidly gives each app 33% of the GPU even when one app finishes, leaving ImageGen unable to use idle capacity and missing its 1-second-per-step SLO. The paper also shows that sharing one model across applications through an inference server with a static configuration (a large attention key-value cache for a background research agent) makes the latency-sensitive chatbot miss its SLO for about 40% of requests, because CPU-side attention becomes a bottleneck. The paper presents ConsumerBench as the first benchmark to include user-defined collaborative workflows and to pair application-level SLO metrics with system-level GPU/CPU utilization, memory bandwidth, and power measurements.

Load-bearing premise

The results depend on the paper's hand-picked SLO thresholds (1s time-to-first-token and 0.25s per output token for the chatbot, 1s per denoising step for image generation, 2s per audio segment for live captions) and its request arrival pattern (a 2-second audio segment every 2 seconds); if real users' SLOs or workloads differ, the reported SLO-miss rates and the comparison between greedy allocation and partitioning could change.

Editorial extensions

If this is right

  • If ConsumerBench's results hold, any future benchmark for on-device GenAI should include concurrent workloads; exclusive-access scores will overstate both latency and SLO attainment for lightweight apps.
  • System designers should replace static GPU partitioning with dynamic, SLO-aware partitioning, since fixed slices both waste idle capacity and cause SLO misses after other apps finish.
  • Inference servers that share one model across applications need per-application configurability; a single static key-value cache setting can force one app's requests to miss their SLO nearly half the time.
  • Model and kernel developers should treat concurrency as a design constraint: kernels with high register or shared-memory pressure reduce SM occupancy and make applications disproportionately vulnerable to interference.
  • SLO-aware scheduling could recover the 45% end-to-end workflow time saved by greedy allocation while avoiding LiveCaptions starvation, because the workflow experiments show greedy is faster overall but unfair to the captioning task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This suggests a direct test not run in the paper: patching only the Whisper decoder kernels to raise SM occupancy, without changing the scheduler, should reduce LiveCaptions' 30x decode slowdown under greedy allocation—if kernel occupancy, not scheduler fairness, is the main lever.
  • The same workflow-DAG method could be applied to integrated GPUs and phones, where memory bandwidth is shared with the CPU; the starvation-versus-underutilization trade-off may be even sharper there, but the paper does not provide data for such devices.
  • Because the paper reports that greedy allocation consumes more instantaneous power but finishes the workflow 45% faster, total energy per task may be lower under greedy allocation despite its unfairness; the paper hints at this but does not compute total energy.
  • An SLO-aware scheduler that gives LiveCaptions priority only when its small kernels are ready, while letting ImageGen fill idle capacity, would be a concrete next system; ConsumerBench's metrics (SMACT versus SMOCC) provide the monitoring hooks to build and validate it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents ConsumerBench, a benchmarking framework for evaluating generative AI applications on end-user devices under concurrent multi-application execution. ConsumerBench lets users define applications, models, SLOs, and workflow dependencies in YAML, then orchestrates execution under different GPU sharing strategies (greedy allocation, static MPS-based partitioning, and shared inference servers), while collecting application-level metrics (latency, SLO attainment) and system-level metrics (GPU/CPU utilization, memory, power). The evaluation on an RTX 6000 server and an Apple M1 Pro laptop covers four applications: Chatbot, DeepResearch, ImageGen, and LiveCaptions. The headline findings are that greedy GPU allocation starves lightweight latency-sensitive applications (LiveCaptions shows a 12.4x average latency increase and almost all requests miss SLO), static GPU partitioning underutilizes the GPU and causes ImageGen to miss SLOs despite available capacity, and naive model sharing via an inference server with a large KV cache degrades Chatbot SLO attainment.

Significance. If the findings are robust, ConsumerBench fills a real gap: existing benchmarks assume exclusive hardware access, while end-user devices run multiple GenAI applications concurrently on a single GPU. The open-sourced implementation and the custom workflow support are concrete strengths, as is the effort to connect application-level SLO misses to system-level metrics such as SM occupancy (SMACT/SMOCC). The paper makes falsifiable empirical observations and offers practical insights about kernel design and SLO-aware scheduling. However, the significance of the headline 'reveals' claims is currently tempered by the limited experimental scope and by the fact that the central SLO-attainment results depend on a single, hand-picked set of thresholds without sensitivity analysis. The framework itself is a reasonable contribution; the paper's broader conclusions about greedy vs. static partitioning are not yet established as general properties of end-user devices.

major comments (3)
  1. [§4.2 and Table 1] The SLO attainment results are threshold-sensitive, and no sensitivity analysis is provided. The LiveCaptions workload sends a 2-second audio segment every 2 seconds with a 2-second SLO, so the deadline equals the inter-arrival period, leaving zero queueing slack; this configuration is predisposed to SLO misses under any form of added latency, and indeed the exclusive-GPU baseline already misses 2% of segments (§4.1). The ImageGen SLO of 1 second per denoising step is only about 1.8x the exclusive-GPU step time (~560 ms, Fig. 4b), so once MPS degrades step time beyond 1 second the 'misses SLO despite available capacity' result follows automatically. The authors should either add a sensitivity analysis varying the SLO multipliers and request arrival patterns, or explicitly reframe the conclusions as point-in-parameter-space demonstrations rather than general properties of resource-sharing strategies.
  2. [§4.2 and §4.2.1] The quantitative claims appear to be based on a single run per configuration; no confidence intervals, standard deviations, or repeated trials are reported. This is load-bearing for statements such as '12.4× increase in average end-to-end request latency' and 'misses its SLO for approximately 40% of its requests', especially because the paper itself notes high variance for Chatbot-KVCache-CPU. At minimum, the authors should report the number of runs and provide mean ± std or percentile ranges for the headline latency and SLO-attainment numbers.
  3. [§4.2, Static GPU Partitioning paragraph] The claim that static partitioning causes ImageGen to 'miss SLOs despite available compute capacity' rests on the stairstep SMACT/SMOCC pattern in Fig. 5a, but the paper does not quantify the idle GPU capacity available during ImageGen's execution. To make the causal link concrete, the authors should measure the fraction of time ImageGen has pending work while its MPS partition is idle and show that this idle time is directly responsible for the SLO miss, rather than relying solely on the 1-second threshold being crossed.
minor comments (5)
  1. [§2.1] In the second paragraph, 'credibleon-deviceinference' is missing spaces and should read 'credible on-device inference'.
  2. [Figure 3] The SLO attainment bars in Fig. 3b are not clearly labeled for GPU vs. CPU; a legend or direct axis labels would make the 0% vs. 98% values interpretable.
  3. [§3.3, LiveCaptions] The 2-second segment / 2-second SLO description leaves unclear whether the SLO includes HTTP and network overhead and whether segments are processed sequentially; please clarify the pipeline and state that this is a zero-slack workload by construction.
  4. [Appendix C] The Apple Silicon experiments use a different image model (SD-v1-4 vs SD-3.5) and a 4-second LiveCaptions SLO; the authors should explicitly state that these results are not directly comparable with the NVIDIA experiments because the configuration is not held fixed.
  5. [Section 6, Limitations] The limitations section does not mention the sensitivity of the results to the chosen SLO values or the absence of repeated runs; both should be acknowledged as limitations of the current experimental study.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: ConsumerBench reports direct measurements of application latency and SLO attainment, with no fitted parameters, derivations, or load-bearing self-citations that reduce to the paper's own outputs.

full rationale

This paper is an empirical benchmarking study. It introduces a framework, runs four GenAI applications on real hardware under exclusive, greedy, partitioned, and shared-server configurations, and reports measured latencies, SLO attainment rates, and system-level utilization metrics. There is no derivation chain in which an output is used as an input: SLO values are user-specified configuration constants (e.g., 1s TTFT, 0.25s TPOT, 1s per denoising step, 2s per LiveCaptions segment) cited to external human-factors and on-device diffusion studies, not parameters fitted to the experimental outcomes. The headline findings about starvation under greedy allocation and underutilization under static partitioning are directly observed comparisons of measured request latencies against these fixed thresholds. The Apple Silicon appendix explicitly adjusts the LiveCaptions SLO to 4s and states the change, so the comparison remains transparent rather than being made circular by construction. The paper's repeated claims to 'reveal' inefficiencies are generalized interpretations of measurements, but the measurements themselves are not produced by fitting or by self-citation. Threshold sensitivity and workload representativeness are legitimate experimental-design concerns, but they are not circularity under the stated criteria, which require that a claimed derivation reduce by definition or by a self-citation chain to its own inputs. No such reduction is present in the text, and no load-bearing argument rests on the authors' prior work. The analysis therefore finds no significant circularity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper's conclusions rest on hand-chosen SLO values, workload patterns, and a single hardware platform, plus the assumption that the selected applications represent the on-device GenAI landscape. There are no fitted physical parameters or invented entities. These configurable choices are not inherently flawed for a benchmark, but they should be validated against real usage traces before treating the findings as general truths.

free parameters (5)
  • Chatbot SLO thresholds = TTFT 1s, TPOT 0.25s
    Chosen from human reading speed [29]; all Chatbot SLO attainment results depend on these hand-picked values.
  • ImageGen SLO threshold = 1s per denoising step
    Set in Table 1 based on diffusion model studies; affects ImageGen SLO attainment.
  • LiveCaptions SLO threshold = 2s per segment (4s on Apple Silicon)
    Set to match 2s audio chunking; changed to 4s on Apple Silicon with no detailed justification, weakening comparability.
  • GPU partitioning fraction = 33% per app (MPS)
    Equally divides GPU among three apps; the results for static partitioning depend on this choice.
  • Workload request patterns = Not fully specified for concurrency experiments
    The paper does not report request arrival rates or counts for the main concurrency experiment (Fig. 5), so the observed interference may depend on undisclosed workload intensity.
assumptions (5)
  • domain assumption The chosen four applications (Chatbot, DeepResearch, ImageGen, LiveCaptions) are representative of GenAI applications on end-user devices.
    The framework's conclusions about system inefficiencies are drawn from this small set; stated in Section 1 and Table 1 without evidence of representativeness.
  • domain assumption The SLOs derived from human reading speed [29] and step-time targets are appropriate proxies for user experience.
    SLO attainment is the primary application-level metric; if these thresholds are not meaningful, the SLO miss results lose significance. Table 1.
  • domain assumption NVIDIA DCGM and MPS behave as documented on consumer-grade GPUs.
    The paper relies on DCGM for SMACT/SMOCC measurements and MPS for static partitioning on an RTX 6000; DCGM is designed for datacenters (footnote 1). Section 3.2.
  • domain assumption The RTX 6000 (Turing, 24GB) is representative of consumer-grade GPUs for end-user devices.
    All primary experiments use a single Quadro RTX 6000; the paper generalizes to 'consumer-grade GPU architectures' despite this being a workstation card. Section 4.
  • domain assumption The DAG workflow configuration in Fig. 23 reflects a realistic content creation workflow.
    The end-to-end workflow timing comparison (Section 4.3) assumes the task dependencies and request counts are realistic; they are chosen by the authors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ConsumerBench: Benchmarking Generative AI Applications on End-User Devices." pith.science (2026). https://pith.science/paper/AUPHFLQN

@misc{pith2026250617538,
  author       = {Pith},
  title        = {Pith review of: ConsumerBench: Benchmarking Generative AI Applications on End-User Devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AUPHFLQN}},
  note         = {Machine review of arXiv:2506.17538}
}
read the original abstract

The recent shift in Generative AI (GenAI) applications from cloud-only environments to end-user devices introduces new challenges in resource management, system efficiency, and user experience. This paper presents ConsumerBench, a comprehensive benchmarking framework designed to evaluate the system efficiency and response time of GenAI models running on end-user devices. Unlike existing benchmarks that assume exclusive model access on dedicated GPUs, ConsumerBench simulates realistic multi-application scenarios executing concurrently on constrained hardware. Furthermore, ConsumerBench supports customizable workflows that simulate complex tasks requiring coordination among multiple applications. ConsumerBench captures both application-level metrics, including latency and Service Level Objective (SLO) attainment, and system-level metrics like CPU/GPU utilization and memory bandwidth. Through extensive experiments, ConsumerBench reveals inefficiencies in resource sharing, unfair scheduling under greedy allocation, and performance pitfalls of static model server configurations. The paper also provides practical insights for model developers and system designers, highlighting the benefits of custom kernels tailored to consumer-grade GPU architectures and the value of implementing SLO-aware scheduling strategies.

Figures

Figures reproduced from arXiv: 2506.17538 by the authors.

Figure 1
Figure 1. The overall design of CONSUMERBENCH. ✞ ☎ 1 Analysis (DeepResearch): 2 model: Llama-3.2-3B 3 num_requests: 1 4 device: cpu 5 Creating Cover Art (ImageGen): 6 model: SD-3.5-Medium-Turbo 7 num_requests: 5 8 device: gpu 9 slo: 1s 10 Generating Captions (LiveCaptions): 11 model: Whisper-Large-V3-Turbo 12 num_requests: 1 13 device: gpu 14 ... ✝ ✆ (a) Task Definition ✞ ☎ 1 analysis_1: 2 uses: Analysis 3 cover_art: 4 uses: … view at source ↗
Figure 2
Figure 2. Example YAML configuration to define application tasks as well as user workflows. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. (a) Latencies normalized to SLO requirements and (b) SLO attainment for Chatbot, Image [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: GPU utilization of each application running exclusively on the GPU. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Application performance & GPU util using greedy resource allocation and GPU partitioning. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of Chatbot performance and resource utilization with GPU vs. CPU KV cache. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: E2E latency & SLO attainment for content-creation workflow w/ and w/o GPU partitioning. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Running applications exclusively on the GPU. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Running applications exclusively on the CPU. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Running applications concurrently on the GPU. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Normalized latency of running larger applications concurrently using greedy allocation [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: GPU metrics of running larger applications concurrently. [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: CPU metrics and power usage of running larger applications concurrently. [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: GPU metrics of running applications with static model sharing via inference servers. [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: CPU metrics and power usage of running applications with static model sharing via [PITH_FULL_IMAGE:figures/full_fig_p018_15.png]
Figure 16
Figure 16. Figure 16: GPU metrics of running the digital content creation workflow. [PITH_FULL_IMAGE:figures/full_fig_p019_16.png]
Figure 17
Figure 17. Figure 17: CPU metrics and power usage of running the digital content creation workflow. [PITH_FULL_IMAGE:figures/full_fig_p019_17.png]
Figure 18
Figure 18. Figure 18: (a) Latencies normalized to SLO requirements and (b) SLO attainment for Chatbot, Image [PITH_FULL_IMAGE:figures/full_fig_p020_18.png]
Figure 19
Figure 19. Figure 19: Metrics of running applications exclusively and concurrently on the Apple Silicon. [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 22
Figure 22. Figure 22: Fig. 21c and Fig. 21f shows the GPU utilization and power usage of the workflow. This [PITH_FULL_IMAGE:figures/full_fig_p020_22.png]
Figure 20
Figure 20. Figure 20: Normalized latency and SLO attainment for Chatbot and Chatbot-KVCache-CPU on the [PITH_FULL_IMAGE:figures/full_fig_p021_20.png]
Figure 21
Figure 21. Figure 21: Metrics of running Chatbot, Chatbot-KVCache-CPU and content creation workflow on [PITH_FULL_IMAGE:figures/full_fig_p021_21.png]
Figure 22
Figure 22. Figure 22: E2E latency & SLO attainment for content-creation workflow running on the Apple [PITH_FULL_IMAGE:figures/full_fig_p021_22.png]
Figure 23
Figure 23. Figure 23: Full YAML configuration of the content creation workflow. [PITH_FULL_IMAGE:figures/full_fig_p022_23.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 39 canonical work pages

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    Llama 3.2 3b: Multilingual large language model

    Meta AI. Llama 3.2 3b: Multilingual large language model. https://huggingface.co/ meta-llama/Llama-3.2-3B, 2024. Accessed: 2025-05-11

    work page 2024

  3. [3]

    Gemini nano with the google ai edge sdk, 2025

    Android Developers. Gemini nano with the google ai edge sdk, 2025

    work page 2025

  4. [4]

    Introducing apple intelligence, the personal intelligence system that puts powerful generative models at the core of iphone, ipad, and mac, 2024

    Apple Inc. Introducing apple intelligence, the personal intelligence system that puts powerful generative models at the core of iphone, ipad, and mac, 2024

    work page 2024

  5. [5]

    Macbook pro 14- and 16-inch – technical specifications, 2024

    Apple Inc. Macbook pro 14- and 16-inch – technical specifications, 2024. Accessed: 2025-05- 16

    work page 2024

  6. [6]

    Stable diffusion web ui, 2022

    AUTOMATIC1111. Stable diffusion web ui, 2022. Accessed: 2025-05-11

    work page 2022

  7. [7]

    Litellm: Python sdk and proxy server for unified llm access, 2024

    BerriAI. Litellm: Python sdk and proxy server for unified llm access, 2024. Accessed: 2025-05-11

    work page 2024

  8. [8]

    Lawrence Zitnick

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015

    2015

Show all 56 references
  1. [9]

    Intel ® performance counter monitor

    Intel Corporation. Intel ® performance counter monitor. https://github.com/intel/pcm,

  2. [10]

    Nvidia data center gpu manager (dcgm)

    NVIDIA Corporation. Nvidia data center gpu manager (dcgm). https://github.com/ NVIDIA/DCGM, 2025. Accessed: 2025-05-12

    2025

  3. [11]

    Nvidia multi-process service (mps)

    NVIDIA Corporation. Nvidia multi-process service (mps). https://docs.nvidia.com/ deploy/mps/index.html, 2025. Accessed: 2025-05-12

    2025

  4. [12]

    Earnings- 21: A practical benchmark for asr in the wild.arXiv preprint arXiv:2104.11348, 2021

    Miguel Del Rio, Natalie Delworth, Ryan Westerman, Michelle Huang, Nishchal Bhandari, Joseph Palakapilly, Quinten McNamara, Joshua Dong, Piotr Zelasko, and Miguel Jetté. Earnings- 21: A practical benchmark for asr in the wild.arXiv preprint arXiv:2104.11348, 2021

    2021 arXiv

  5. [13]

    Mobile-bench: An evaluation benchmark for llm-based mobile agents, 2024

    Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, and Shuo Shang. Mobile-bench: An evaluation benchmark for llm-based mobile agents, 2024

    2024

  6. [14]

    Qlora: Efficient finetuning of quantized llms, 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms, 2023

    2023

  7. [15]

    Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023

    2023

  8. [16]

    llama.cpp: Llm inference in c/c++, 2023

    Georgi Gerganov and ggml-org contributors. llama.cpp: Llm inference in c/c++, 2023. Accessed: 2025-05-11

    2023

  9. [17]

    Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models, 2024

    Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models, 2024

    2024

  10. [18]

    A critical review of large language models: Sensitivity, bias, and the path toward specialized ai.Quantitative Science Studies, 5(3):736–756, 2024

    Arash Hajikhani and Carolyn Cole. A critical review of large language models: Sensitivity, bias, and the path toward specialized ai.Quantitative Science Studies, 5(3):736–756, 2024

    2024

  11. [19]

    Microsecond-scale preemption for concurrent GPU-accelerated DNN inferences

    Mingcong Han, Hanze Zhang, Rong Chen, and Haibo Chen. Microsecond-scale preemption for concurrent GPU-accelerated DNN inferences. In16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 539–558, Carlsbad, CA, July 2022. USENIX Association. 10

    2022

  12. [20]

    Hybrid slm and llm for edge- cloud collaborative inference

    Zixu Hao, Huiqiang Jiang, Shiqi Jiang, Ju Ren, and Ting Cao. Hybrid slm and llm for edge- cloud collaborative inference. InProceedings of the Workshop on Edge and Mobile Foundation Models, EdgeFM ’24, page 36–41, New York, NY , USA, 2024. Association for Computing Machinery

    2024

  13. [21]

    Searching for mobilenetv3

    Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. InProceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019

    2019

  14. [22]

    Running average power limit (rapl) energy reporting

    Intel Corporation. Running average power limit (rapl) energy reporting. https: //www.intel.com/content/www/us/en/developer/articles/technical/software- security-guidance/advisory-guidance/running-average-power-limit-energy- reporting.html, 2022. Accessed: 2025-05-14

    2022

  15. [23]

    Benchmarking edge ai platforms for high-performance ml inference, 2024

    Rakshith Jayanth, Neelesh Gupta, and Viktor Prasanna. Benchmarking edge ai platforms for high-performance ml inference, 2024

    2024

  16. [24]

    Melting point: Mobile evaluation of language transformers, 2024

    Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto, and Hamed Haddadi. Melting point: Mobile evaluation of language transformers, 2024

    2024

  17. [25]

    Api-bank: A comprehensive benchmark for tool-augmented llms, 2023

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms, 2023

    2023

  18. [26]

    Snapfusion: Text-to-image diffusion model on mobile devices within two seconds.Advances in Neural Information Processing Systems, 36:20662–20678, 2023

    Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds.Advances in Neural Information Processing Systems, 36:20662–20678, 2023

    2023

  19. [27]

    Palmbench: A comprehensive benchmark of compressed large language models on mobile platforms, 2025

    Yilong Li, Jingyu Liu, Hao Zhang, M Badri Narayanan, Utkarsh Sharma, Shuai Zhang, Pan Hu, Yijing Zeng, Jayaram Raghuram, and Suman Banerjee. Palmbench: A comprehensive benchmark of compressed large language models on mobile platforms, 2025

    2025

  20. [28]

    Awq: Activation-aware weight quantization for llm compression and acceleration, 2024

    Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for llm compression and acceleration, 2024

    2024

  21. [29]

    Andes: Defining and enhancing quality-of-experience in llm-based text streaming services

    Jiachen Liu, Jae-Won Chung, Zhiyu Wu, Fan Lai, Myungjin Lee, and Mosharaf Chowdhury. Andes: Defining and enhancing quality-of-experience in llm-based text streaming services. arXiv preprint arXiv:2404.16283, 2024

    2024 arXiv

  22. [30]

    Agentbench: Evaluating llms as agents.arXiv preprint arXiv: 2308.03688, 2023

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench:...

    2023 arXiv

  23. [31]

    Turning whisper into real-time transcription system

    Dominik Macháˇcek, Raj Dabre, and Ondˇrej Bojar. Turning whisper into real-time transcription system. InProceedings of the 13th International Joint Conference on Natural Language Pro- cessing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computation...

    2023

  24. [32]

    Aios: Llm agent operating system.arXiv preprint arXiv:2403.16971, 2024

    Kai Mei, Xi Zhu, Wujiang Xu, Wenyue Hua, Mingyu Jin, Zelong Li, Shuyuan Xu, Ruosong Ye, Yingqiang Ge, and Yongfeng Zhang. Aios: Llm agent operating system.arXiv preprint arXiv:2403.16971, 2024

    2024 arXiv

  25. [33]

    Mo- bileaibench: Benchmarking llms and lmms for on-device use cases, 2024

    Rithesh Murthy, Liangwei Yang, Juntao Tan, Tulika Manoj Awalgaonkar, Yilun Zhou, Shelby Heinecke, Sachin Desai, Jason Wu, Ran Xu, Sarah Tan, Jianguo Zhang, Zhiwei Liu, Shirley Kokane, Zuxin Liu, Ming Zhu, Huan Wang, Caiming Xiong, and Silvio Savarese. Mo- bileaibench: Benchmar...

    2024

  26. [34]

    Green contexts - cuda driver api.https://docs.nvidia.com/cuda/cuda-driver- api/group__CUDA__GREEN__CONTEXTS.html, 2025

    NVIDIA. Green contexts - cuda driver api.https://docs.nvidia.com/cuda/cuda-driver- api/group__CUDA__GREEN__CONTEXTS.html, 2025. Accessed: 2025-04-30. 11

    2025

  27. [35]

    Nvidia multi-instance gpu (mig)

    NVIDIA. Nvidia multi-instance gpu (mig). https://www.nvidia.com/en-us/ technologies/multi-instance-gpu/, 2025. Accessed: 2025-05-15

    2025

  28. [36]

    NVIDIA, August 2018

    NVIDIA Corporation.NVIDIA Quadro RTX 6000 GPU. NVIDIA, August 2018. Data sheet, retrieved May 13, 2025

    2018

  29. [37]

    Nvidia management library (nvml).https://developer.nvidia.com/ management-library-nvml, 2025

    NVIDIA Corporation. Nvidia management library (nvml).https://developer.nvidia.com/ management-library-nvml, 2025. Accessed: 2025-05-14

    2025

  30. [38]

    Seungeun Oh, Jinhyuk Kim, Jihong Park, Seung-Woo Ko, Tony Q. S. Quek, and Seong-Lyun Kim. Uncertainty-aware hybrid inference with on-device small and remote large language models, 2025

    2025

  31. [39]

    Attention u-net: Learning where to look for the pancreas

    Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, Ben Glocker, and Daniel Rueckert. Attention u-net: Learning where to look for the pancreas. InMedical Imaging with Deep Lea...

    2018

  32. [40]

    Openai api, 2020

    OpenAI. Openai api, 2020. Accessed: 2025-05-11

    2020

  33. [41]

    Memgpt: Towards llms as operating systems

    Charles Packer, Vivian Fang, Shishir_G Patil, Kevin Lin, Sarah Wooders, and Joseph_E Gonza- lez. Memgpt: Towards llms as operating systems. 2023

    2023

  34. [42]

    Snapdragon x elite product brief.https://www.qualcomm.com/ content/dam/qcomm-martech/dm-assets/documents/Product-Brief-Snapdragon- X-Elite.pdf, October 2023

    Qualcomm Technologies, Inc. Snapdragon x elite product brief.https://www.qualcomm.com/ content/dam/qcomm-martech/dm-assets/documents/Product-Brief-Snapdragon- X-Elite.pdf, October 2023

    2023

  35. [43]

    Robust speech recognition via large-scale weak supervision, 2022

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022

    2022

  36. [44]

    Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B

    Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, It...

    2020

  37. [45]

    ‘smolagents‘: a smol library to build great agentic systems

    Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. ‘smolagents‘: a smol library to build great agentic systems. https: //github.com/huggingface/smolagents, 2025

    2025

  38. [46]

    Specialist or generalist? instruction tuning for specific NLP tasks

    Chufan Shi, Yixuan Su, Cheng Yang, Yujiu Yang, and Deng Cai. Specialist or generalist? instruction tuning for specific NLP tasks. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages...

    2023

  39. [47]

    Orion: Interference-aware, fine-grained gpu sharing for ml applications

    Foteini Strati, Xianzhe Ma, and Ana Klimovic. Orion: Interference-aware, fine-grained gpu sharing for ml applications. InProceedings of the Nineteenth European Conference on Computer Systems, EuroSys ’24, page 1075–1092, New York, NY , USA, 2024. Association for Computing Machinery

    2024

  40. [48]

    Stable diffusion 3.5 medium turbo, 2024

    TensorArt Studios. Stable diffusion 3.5 medium turbo, 2024. Accessed: 2025-05-11

    2024

  41. [49]

    Large language model performance benchmarking on mobile platforms: A thorough evaluation.arXiv preprint arXiv:2410.03613, 2024

    Jie Xiao, Qianyi Huang, Xu Chen, and Chen Tian. Large language model performance benchmarking on mobile platforms: A thorough evaluation.arXiv preprint arXiv:2410.03613, 2024

    2024

  42. [50]

    Crab: Cross-environment agent benchmark for multimodal language model agents, 2024

    Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, Anjie Yang, Zhaoxuan Jin, Jianbo Deng, Philip Torr, Bernard Ghanem, and Guohao Li. Crab: Cross-environment agent benchmark for multimodal language ...

    2024

  43. [51]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  44. [52]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proc...

    2018

  45. [53]

    Faaswap: Slo-aware, gpu-efficient serverless inference via model swapping, 2024

    Minchen Yu, Ao Wang, Dong Chen, Haoxuan Yu, Xiaonan Luo, Zhuohao Li, Wei Wang, Ruichuan Chen, Dapeng Nie, and Haoran Yang. Faaswap: Slo-aware, gpu-efficient serverless inference via model swapping, 2024

    2024

  46. [54]

    Mobilediffusion: Instant text-to-image generation on mobile devices, 2024

    Yang Zhao, Yanwu Xu, Zhisheng Xiao, Haolin Jia, and Tingbo Hou. Mobilediffusion: Instant text-to-image generation on mobile devices, 2024

    2024

  47. [55]

    brainstorm

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-1m: A large-scale real-world llm conversation dataset, 2024. 13 Appendices A Data and C...

    2024

  48. [2025]

    Accessed: 2025-05-12

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.