Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Inference performance evaluation for LLMs on edge devices with a novel benchmarking framework and metric

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new metric, MBU, measures how well edge devices use memory bandwidth when running quantized LLMs, and benchmarks reveal GPU-accelerated runs can be fast but badly inaccurate.

desk verdict Worth a desk read for the CPU/Metal benchmark data, but the MBU 'novel metric' is a standard roofline ratio and the OpenCL GPU rows on two platforms are effectively garbage-in. read the letter →

arxiv 2508.11269 v1 pith:ZYUSPJUW submitted 2025-08-15 cs.PF

Hao Chen , Cong Tian , Zixuan He , Bin Yu , Yepang Liu , Jialun Cao This is my paper · ORCID
classification cs.PF
keywords LLMinferenceedgecomputingmemorybandwidthutilizationbenchmarkingmodelquantizationGPUaccelerationperplexityKVcache
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ELIB, a benchmarking tool for large language model inference on edge devices, and a new metric, MBU (Model Bandwidth Utilization), defined as the ratio of achieved memory bandwidth to the hardware's peak memory bandwidth. The central argument is that LLM token generation on edge hardware is memory-bound, so MBU is a normalized, platform-independent measure of how efficiently a device uses its available memory bandwidth when running a given quantized model; higher MBU should track higher FLOPS and throughput. Using three edge platforms (an IoT board, an Android phone, and a laptop) and five quantized LLaMA models, the paper reports MBU values and derives heuristics for optimizing MBU through batch size, sequence length, and KV cache management, subject to memory capacity and latency constraints. The paper's own accuracy data show that GPU acceleration on the board and phone produced perplexity scores around 54-68, roughly ten times worse than CPU scores of 4-6, which means those GPU runs were not performing correct inference despite reporting high FLOPS, throughput, and MBU.

What carries the argument

The MBU metric (Model Bandwidth Utilization): MBU = achieved memory bandwidth / peak memory bandwidth, with achieved memory bandwidth = (total model parameter size + KV cache size) / time per output token. It is a normalized ratio meant to capture how efficiently an LLM uses the memory system during decoding, the presumed bottleneck. ELIB's Model-Graph-Kernel runtime, which separates model, graph, and kernel layers, is the supporting mechanism that lets the same benchmarking flow run across CPU and GPU backends on different operating systems.

What would settle it

Re-run the GPU benchmarks on the board and phone with a correctness sanity check: decode a fixed prompt and compare the output tokens (or exact logits) to the CPU output. If the GPU outputs diverge or perplexity stays above ~20 on WikiText-2, the reported GPU FLOPS, throughput, and MBU values do not represent valid LLM inference.

Watch

Extended reading notes

Core claim

The paper's central claim is that MBU—the ratio of achieved memory bandwidth to peak memory bandwidth, where achieved bandwidth is (model parameter size + KV cache size) divided by time per output token—is a valid normalized metric for comparing LLM inference across heterogeneous edge hardware and quantization schemes. In a memory-bound decoding process, the authors argue, MBU is the quantity that actually determines token throughput, and optimizing it (by increasing batch size, tuning sequence length, and compressing the KV cache) is the right way to improve inference performance without raising raw FLOPS. The paper also claims that MBU optimization is constrained by memory capacity and tot

Load-bearing premise

The load-bearing assumption is that the GPU benchmark results on the board and phone are valid LLM inference measurements; if those GPU runs are actually producing near-random text (perplexity 54-68 vs CPU 4-6), then their FLOPS, throughput, and MBU numbers are not describing LLM inference performance.

Editorial extensions

If this is right

  • If MBU is correct, edge platforms and quantized models become comparable by a single normalized number, independent of raw hardware speed.
  • Optimizing MBU—larger batch sizes, tuned sequence lengths, compressed KV caches—should improve token throughput even when FLOPS stay constant.
  • The benchmark's accuracy data imply that high FLOPS, throughput, and MBU on a GPU path are not sufficient: perplexity must be checked, because two of the three devices' GPU runs were near-random.
  • The heuristics (batch size, sequence length, KV cache management) provide a practical starting point for tuning LLM deployment on new edge hardware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • MBU as defined is essentially per-token bytes moved (model + KV size divided by time per token) normalized by peak bandwidth, so on a memory-bound workload it ranks configurations similarly to throughput normalized by peak bandwidth; the novelty is the normalization, not the raw measurement.
  • The near-random perplexity on the GPU paths suggests those MBU and FLOPS numbers should not be interpreted as LLM inference performance; a fair comparison would either exclude those configurations or fix the GPU path first.
  • A testable extension: decrease quantized model size enough or increase batch size enough to enter the compute-bound regime, where the MBU-throughput correlation should weaken; if it does not, the memory-bound assumption behind MBU is incomplete.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ELIB, a benchmarking framework for LLM inference on edge devices, and proposes a new metric, Model Bandwidth Utilization (MBU), defined as the ratio of achieved to peak memory bandwidth. ELIB is deployed on three platforms (NanoPI RK3588, Xiaomi Snapdragon 778, MacBook Air M2) using llama.cpp and five GGML quantized LLaMA-7B variants. The paper reports FLOPS, throughput, latency, perplexity, and MBU, and derives heuristics about maximizing MBU and the constraints that limit it. The central empirical contribution is the claim that MBU is a normalized, comparable metric that correlates with throughput and FLOPS across edge platforms, and that GPU acceleration improves throughput and MBU on all devices.

Significance. If the framework and metric were validated, ELIB would address a real gap: comparing LLM inference across heterogeneous edge devices with different acceleration stacks. The open-source benchmarking tool, the integration of multiple backends (OpenBLAS, OpenCL, Metal), and the automatic quantization flow are practical contributions, and the paper explicitly promises reproducible documentation. However, the validity of the GPU results on two of the three platforms is explicitly contradicted by the paper's own accuracy data: OpenCL GPU inference on NanoPI and Xiaomi yields perplexity scores near 54-68 versus 4-7 on CPU, indicating that the GPU path is not performing correct LLM inference. Because these invalid GPU measurements are then used to support the paper's main throughput, FLOPS, and MBU conclusions, the empirical foundation of the claimed results is compromised. In addition, the claimed correlation between MBU and throughput is partly tautological given Equations (1)-(2).

major comments (3)
  1. [Section 5.2 (Accuracy/RQ3), Table 6] The OpenCL GPU rows for NanoPI and Xiaomi report perplexity scores of roughly 54-68, while the CPU rows report 4-7 on the same models. The paper itself states that this is 'almost 10 times higher' and attributes it to 'suboptimal parallelization design and data precision issues.' Yet these same GPU rows are included in Table 6 and used in Section 5.2 to support conclusions about GPU FLOPS, throughput, MBU, and the trade-off between quantization and accuracy. A 10x perplexity degradation indicates that the GPU path is producing near-random outputs, so FLOPS/throughput/MBU values for those configurations do not measure LLM inference performance. These rows must be excluded or clearly separated, and all GPU-related heuristics must be re-derived without them. This is load-bearing, not a minor presentation issue.
  2. [Section 4.2, Eqs. (1)-(2), and RQ1 in Section 5.2] MBU is defined as Achieved Memory Bandwidth / Peak Memory Bandwidth, and Achieved Memory Bandwidth is defined as (Total Model Parameter Size + KV Cache Size) / TPOT. Since TPOT is the reciprocal of throughput, for a fixed model, fixed quantization, and fixed batch/sequence configuration, MBU is proportional to throughput by construction: MBU = (Model+KV size) / (Peak Bandwidth) x Throughput. Therefore the statement in RQ1 that 'a higher MBU value consistently correlates with higher FLOPS and throughput' is not an empirical finding but a mathematical consequence of the metric definition. Similarly, the heuristic that increasing batch size increases MBU partly restates the standard effect of batch size on throughput and KV-cache size. The paper should either redefine MBU so that this correlation is non-trivial, or clearly present the throughput correlation as a definitional property rather
  3. [Table 6 and Section 5.2] All benchmark values are reported as single numbers with no error bars, no standard deviations, and no explicit number of repeated runs for throughput/latency/FLOPS. The text mentions '100 inference iterations' only in the accuracy discussion. On mobile and IoT devices, thermal throttling, DVFS, and background processes can cause significant run-to-run variation. Without repeated measurements and dispersion statistics, comparative claims such as 'GPU throughput is 1.18X/1.41X/1.53X higher than CPU' cannot be assessed for statistical significance. This is especially important because the 4-thread vs 8-thread FLOPS result (4 threads outperforming 8 threads) is itself counterintuitive and needs robustness checks.
minor comments (5)
  1. [Throughout] The manuscript contains numerous typos and inconsistencies: 'rencent', 'quantitation' (should be 'quantization'), 'TTLM' versus 'TTLM/TTLM' terminology, 'Xiaomi Redmi Node12' (Table 1) versus 'Redmi Note12 Turbo' in text, and 'Lox' in Table 2 (likely 'Low'). These should be fixed.
  2. [Figures 3-6] The figure captions and axis labels are duplicated and have unusual formatting (e.g., '0.0 0', '20 .00'). The subfigures are not clearly referenced in the text; please label subpanels (a), (b), etc., and refer to them explicitly.
  3. [Section 5.1.1, Table 5] The quantization naming convention is described (qX_Y), but the table uses 'q4 0' with a space instead of 'q4_0'. This makes it easy to misread the quantization types. Use the underscore notation consistently.
  4. [Section 4.2] The FLOPS definition says 'floating-point operations per second,' but quantized GGML inference on CPU/GPU may use integer or mixed-precision kernels. How FLOPS is computed for quantized models (e.g., whether dequantization is counted) is not specified and should be clarified.
  5. [References] The paper cites community leaderboards and frameworks through footnotes rather than formal references; some references are incomplete (e.g., no page numbers for DAWNBench). Please unify the citation style.

Circularity Check

2 steps flagged · score 6.0 of 10

MBU's central validation—that higher MBU correlates with higher throughput—is true by definition, because MBU is a constant multiple of throughput; the RQ1 'finding' reduces to Eqs. 1-2.

  1. self definitional [Section 4.2 (Eqs. 1-2, TPOT definition) and Section 5.2 RQ1]
    "MBU is defined as the ratio of achieved memory bandwidth to peak memory bandwidth (1). Achieved memory bandwidth is calculated as the sum of the total model parameter size and the KV cache size, divided by the total processing time (2). ... Typically, TPOT is the inverse of throughput. ... Therefore, both in theoretical calculations and experimental data, a higher MBU value consistently correlates with higher FLOPS and throughput."

    From Eq. 2, Achieved BW = (Model+KV)/TPOT. Since the paper states TPOT is the inverse of throughput, Achieved BW = (Model+KV) × throughput. Dividing by peak bandwidth gives MBU = (Model+KV) × throughput / peak BW. For a fixed model, KV configuration, and peak bandwidth, MBU is a constant multiple of throughput. Thus the RQ1 claim that 'higher MBU consistently correlates with higher throughput' is an algebraic identity derived from the metric's own definition, not an empirical finding. The validation of MBU therefore reduces to its construction.

  2. self definitional [Section 4.2 Eq. 3 and Section 5.2 RQ1]
    "KV Cache Size = Batch Size × Sequence Length × (dmodel/nheads) × nlayers × nkv heads × Data Byte × 2 (3) ... The first is increasing the batch size. Larger batch sizes allow more data to be processed simultaneously, which increases the achieved memory bandwidth."

    Eq. 3 inserts Batch Size as a multiplicative factor in the KV Cache Size term, which in turn appears in the numerator of the achieved-memory-bandwidth definition, Eq. 2. Therefore the heuristic that increasing batch size increases achieved memory bandwidth (and thus MBU) is a direct consequence of the metric's definition, not an independently observed optimization principle. The only non-definitional part would be how TPOT responds to batch size, which the paper does not separately model or measure in a controlled way.

full rationale

ELIB is largely an engineering/benchmarking paper with no fitted parameters and no load-bearing self-citations; the ELIB tool, the measured benchmark data, and the comparisons across platforms have independent content. The circular component is concentrated in the interpretation of MBU. Eq. 2 divides bytes by TPOT, and the paper explicitly says TPOT is the inverse of throughput, so MBU = (model_size+KV) × throughput / peak_bandwidth. Consequently, the RQ1 conclusion that 'a higher MBU value consistently correlates with higher FLOPS and throughput' restates the definition rather than reporting an empirical correlation. Similarly, the claim that increasing batch size raises MBU follows from Eq. 3's batch-size factor in the KV cache term. These are self-definitional steps, not fitted-input or self-citation circularity. The acknowledged 10x perplexity degradation on NanoPI/Xiaomi OpenCL GPU paths (Section 5.2, Table 6, perplexity 54-68 vs 4-8 on CPU) is a serious threat to the validity of those rows as LLM-inference benchmarks, but it is a correctness/validity concern rather than circularity and does not by itself raise the circularity score. Overall score 6 reflects that one central supporting claim reduces by construction while the rest of the paper's empirical content remains independent.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central metric MBU relies on domain assumptions about memory-bound decoding and datasheet bandwidth. No new physical entities are introduced. The implicit validity of broken GPU inference results is an unstated and contradicted assumption.

assumptions (3)
  • domain assumption LLM decoding is memory-bandwidth-bound, and reading the full model parameters from DRAM once per output token is the dominant cost.
    Invoked in Section 4.2 and RQ1 to justify MBU as the key metric. This is a well-known property of autoregressive decoding, but it is stated as a premise for the metric's relevance.
  • domain assumption Datasheet peak memory bandwidth is the correct normalization denominator for MBU.
    Used in Equation 1. Real achievable peak is usually lower, so this choice inflates MBU and makes cross-platform comparisons dependent on datasheet accuracy.
  • ad hoc to paper GPU OpenCL inference on NanoPI and Xiaomi produces valid LLM outputs despite perplexity scores of 54 to 68.
    The paper reports these GPU results in Table 6 and uses them for analysis, implicitly assuming they represent meaningful inference performance. The accuracy data contradicts this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Inference performance evaluation for LLMs on edge devices with a novel benchmarking framework and metric." pith.science (2026). https://pith.science/paper/ZYUSPJUW

@misc{pith2026250811269,
  author       = {Pith},
  title        = {Pith review of: Inference performance evaluation for LLMs on edge devices with a novel benchmarking framework and metric},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZYUSPJUW}},
  note         = {Machine review of arXiv:2508.11269}
}
read the original abstract

With the significant success achieved by large language models (LLMs) like LLaMA, edge computing-based LLM inference services for mobile and PC are in high demand for data privacy. However, different edge platforms have different hardware characteristics and the large demand for memory capacity and bandwidth makes it very challenging to deploy and benchmark LLMs on edge devices. In this paper, we introduce a benchmarking tool named ELIB (edge LLM inference benchmarking) to evaluate LLM inference performance of different edge platforms, and propose a novel metric named MBU to indicate the percentage of the theoretically efficient use of available memory bandwidth for a specific model running on edge hardware to optimize memory usage. We deploy ELIB on three edge platforms and benchmark using five quantized models to optimize MBU in combination with other metrics such as FLOPS, throughput, latency and accuracy. And we analyze the results to derive the key factors, constraints, unpredictability in optimizing MBU that can guide deploying LLMs on more edge platforms.

Figures

Figures reproduced from arXiv: 2508.11269 by the authors.

Figure 1
Figure 1. The design overview of ELIB indicates that the core component is the benchmarking runtime framework, including model adaptation, [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Model-Graph-Kernel structure of benchmark runtime framework o [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Think Before You Grid-Search: Floor-First Triage for LLM Serving

    cs.PF 2026-07 conditional novelty 6.0 of 10

    LLM serving should triage by five-resource analytical floors and wall ordering, not grid search; on 16×H20, TP16 is capacity-capped at ~70 while EP+DP attention reaches ~644 concurrent 8K requests.

Reference graph

Works this paper leans on

37 extracted references · 24 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, I. Polosukhin, Attention is all you need, in: I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information Processing Systems, V ol. 30, Curran Associates, Inc., 2017

    work page 2017

  2. [2]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, L...

    work page 2020

  3. [3]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, R. Lowe, Training language models to follow instructions with human feedback, in: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. ...

    work page 2022

  4. [4]

    Z. Du, Y . Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, J. Tang, GLM: General language model pretraining with autoregressive blank infilling, arXiv preprint (Mar. 2021). arXiv:2103.10360

    arXiv 2021

  5. [5]

    A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y . Xu, W. Zheng, X. Xia, W. L. Tam, Z. Ma, Y . Xue, J. Zhai, W. Chen, P. Zhang, Y . Dong, J. Tang, GLM-130B: An open bilingual pre-trained model, arXiv preprint (Oct. 2022).arXiv:2210.02414

    arXiv 2022

  6. [6]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, G. Lample, LLaMA: Open and efficient foundation language models, arXiv preprint (Feb. 2023). arXiv:2302.13971

    arXiv 2023

  7. [7]

    Shahriar, K

    S. Shahriar, K. Hayawi, Let’s have a chat! a conversation with ChatGPT: Technology, applications, and limitations, Artificial Intelligence and Applications (Feb. 2023). doi:10.47852/bonviewaia3202939

    work page doi:10.47852/bonviewaia3202939 2023

  8. [8]

    J. Oppenlaender, The creativity of text-to-image generation, in: Proceedings of the 25th International Academic Mindtrek Conference, Academic Mindtrek 2022, ACM, Tampere, Finland, 2022, pp. 192–202. doi:10.1145/3569219.3569352

    work page arXiv 2022

Show all 37 references
  1. [9]

    Zhang, F

    M. Zhang, F. Zhang, N. D. Lane, Y . Shu, X. Zeng, B. Fang, S. Yan, H. Xu, Deep Learning in the Era of Edge Computing: Challenges and Opportunities, Wiley, 2020, Ch. 3, pp. 67–78. doi:10.1002/9781119551713.ch3

    2020 doi

  2. [10]

    M. V . Barbera, S. Kosta, A. Mei, J. Stefa, To offload or not to offload? the bandwidth and energy costs of mobile cloud computing, in: 2013 Proceedings IEEE INFOCOM, IEEE, Turin, Italy, 2013, pp. 1285–1293. doi:10.1109/infcom.2013.6566921

    2013

  3. [11]

    J. Jo, S. Jeong, P. Kang, Benchmarking GPU-accelerated edge devices, in: 2020 IEEE International Conference on Big Data and Smart Computing (BigComp), IEEE, Busan, Korea (South), 2020, pp. 117–120. doi:10.1109/bigcomp48618.2020.00-89

    2020

  4. [12]

    X. Ma, T. Yao, M. Hu, Y . Dong, W. Liu, F. Wang, J. Liu, A survey on deep learning empowered IoT applications, IEEE Access 7 (2019) 181721–181732. doi:10.1109/access.2019.2958962

    2019

  5. [13]

    W. Hu, Y . Gao, K. Ha, J. Wang, B. Amos, Z. Chen, P. Pillai, M. Satyanarayanan, Quantifying the impact of edge computing on mobile applications, in: Proceedings of the 7th ACM SIGOPS Asia-Pacific Workshop on Systems, APSys ’16, ACM, Hong Kong, China, 2016, pp. 1–8. doi:10.1145...

    2016

  6. [14]

    H. Kong, S. Huai, D. Liu, L. Zhang, H. Chen, S. Zhu, S. Li, W. Liu, M. Rastogi, R. Subramaniam, M. Athreya, M. A. Lewis, Edlab: A benchmark for edge deep learning accelerators, IEEE Design & Test 39 (3) (2022) 8–17. doi:10.1109/mdat.2021.3095215

    2022

  7. [15]

    Varghese, N

    B. Varghese, N. Wang, D. Bermbach, C.-H. Hong, E. D. Lara, W. Shi, C. Stewart, A survey on edge performance benchmarking, ACM Computing Surveys 54 (3) (2021) 1–33. doi:10.1145/3444692

    2021 doi

  8. [16]

    Bianco, R

    S. Bianco, R. Cadene, L. Celona, P. Napoletano, Benchmark analysis of representative deep neural network architectures, IEEE Access 6 (2018) 64270–64277. doi:10.1109/access.2018.2877890

    2018

  9. [17]

    Coleman, D

    C. Coleman, D. Narayanan, D. Kang, T. Zhao, J. Zhang, L. Nardi, P. Bailis, K. Olukotun, C. R ´e, M. Zaharia, DAWNBench: An end-to-end deep learning benchmark and competition, in: Advances in Neural Information Processing Systems, 2017

    2017

  10. [18]

    Kuzmin, M

    A. Kuzmin, M. van Baalen, Y . Ren, M. Nagel, J. Peters, T. Blankevoort, FP8 quantization: The power of the exponent, in: S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh (Eds.), Advances in Neural Information Processing Systems, V ol. 35, Curran Associates, Inc....

    2022

  11. [19]

    Micikevicius, D

    P. Micikevicius, D. Stosic, N. Burgess, M. Cornea, P. Dubey, R. Grisenthwaite, S. Ha, A. Heinecke, P. Judd, J. Kamalu, N. Mellempudi, S. Oberman, M. Shoeybi, M. Siu, H. Wu, FP8 formats for deep learning, arXiv preprint (Sep. 2022). arXiv:2209.05433

    2022 arXiv

  12. [20]

    Rodriguez, E

    A. Rodriguez, E. Segal, E. Meiri, E. Fomenko, Y . J. Kim, H. Shen, B. Ziv, Lower numerical precision deep learning inference and training, Intel White Paper (Jan. 2018)

    2018

  13. [21]

    S. S. Ogden, T. Guo, MODI: Mobile deep inference made e fficient by edge computing, in: USENIX Workshop on Hot Topics in Edge Computing (HotEdge 18), USENIX Association, Boston, MA, 2018. URL https://www.usenix.org/conference/hotedge18/presentation/ogden

    2018

  14. [22]

    K. A. N. G. Pilsung, J. O. Jongmin, Benchmarking modern edge devices for ai applications, IEICE Transactions on Information and Systems E104.D (3) (2021) 394–403. doi:10.1587/transinf.2020edp7160

    2021 doi

  15. [23]

    S. P. Baller, A. Jindal, M. Chadha, M. Gerndt, DeepEdgeBench: Benchmarking deep neural networks on edge devices, in: 2021 IEEE International Conference on Cloud Engineering (IC2E), IEEE, 2021. doi:10.1109/ic2e52221.2021.00016

    2021

  16. [24]

    V . J. Reddi, C. Cheng, D. Kanter, P. Mattson, G. Schmuelling, C.-J. Wu, B. Anderson, M. Breughe, M. Charlebois, W. Chou, R. Chukka, C. Coleman, S. Davis, P. Deng, G. Diamos, J. Duke, D. Fick, J. S. Gardner, I. Hubara, S. Idgunji, T. B. Jablin, J. Jiao, T. S. John, P. Kanwar, ...

    2020

  17. [25]

    Banerjee, P

    D. Banerjee, P. Singh, A. Avadhanam, S. Srivastava, Benchmarking LLM powered chatbots: Methods and metrics, arXiv preprint (Aug. 2023). arXiv:2308.04624. 16

    2023 arXiv

  18. [26]

    J. Liu, C. Liu, P. Zhou, Q. Ye, D. Chong, K. Zhou, Y . Xie, Y . Cao, S. Wang, C. You, P. S. Yu, LLMRec: Benchmarking large language models on recommendation task, arXiv preprint (Aug. 2023). arXiv:2308.12241

    2023 arXiv

  19. [27]

    Zhang, F

    T. Zhang, F. Ladhak, E. Durmus, P. Liang, K. McKeown, T. B. Hashimoto, Benchmarking large language models for news summarization, arXiv preprint (Jan. 2023). arXiv:2301.13848

    2023 arXiv

  20. [28]

    J. E. Stone, D. Gohara, G. Shi, OpenCL: A parallel programming standard for heterogeneous computing systems, Computing in Science & Engineering 12 (3) (2010) 66–73. doi:10.1109/mcse.2010.69

    2010 doi

  21. [29]

    Nugteren, CLBlast: A tuned OpenCL BLAS library, in: Proceedings of the International Workshop on OpenCL, IWOCL ’18, ACM, Oxford, UK, 2018

    C. Nugteren, CLBlast: A tuned OpenCL BLAS library, in: Proceedings of the International Workshop on OpenCL, IWOCL ’18, ACM, Oxford, UK, 2018. doi:10.1145/3204919.3204924

    2018

  22. [30]

    Gebraad, A

    L. Gebraad, A. Fichtner, Seamless gpu acceleration for c ++-based physics with the metal shading language on apple’s m series unified chips, Seismological Research Letters (Feb. 2023). doi:10.1785/0220220241

    2023 doi

  23. [31]

    G. Park, B. Park, M. Kim, S. Lee, J. Kim, B. Kwon, S. J. Kwon, B. Kim, Y . Lee, D. Lee, Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models, arXiv preprint (Jun. 2022). arXiv:2206.09557

    2022 arXiv

  24. [32]

    Frantar, S

    E. Frantar, S. Ashkboos, T. Hoefler, D. Alistarh, Gptq: Accurate post-training quantization for generative pre-trained transformers, arXiv preprint (Oct. 2022). arXiv:2210.17323

    2022 arXiv

  25. [33]

    Dettmers, M

    T. Dettmers, M. Lewis, Y . Belkada, L. Zettlemoyer, LLM.int8(): 8-bit matrix multiplication for transformers at scale, arXiv preprint (Aug. 2022). arXiv:2208.07339

    2022 arXiv

  26. [34]

    X. Sun, J. Choi, C.-Y . Chen, N. Wang, S. Venkataramani, V . V . Srinivasan, X. Cui, W. Zhang, K. Gopalakrishnan, Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks, in: H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch ´e-Buc, E. Fox, R. Gar...

    2019

  27. [35]

    Banner, Y

    R. Banner, Y . Nahshan, D. Soudry, Post training 4-bit quantization of convolutional networks for rapid-deployment, in: H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch ´e-Buc, E. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems, V ol. 32, Curran A...

    2019

  28. [36]

    H. Shen, H. Chang, B. Dong, Y . Luo, H. Meng, Efficient llm inference on cpus, arXiv preprint (Nov. 2023). arXiv:2311.00502

    2023 arXiv

  29. [37]

    Huyen, Evaluation metrics for language modeling, The Gradient (2019)

    C. Huyen, Evaluation metrics for language modeling, The Gradient (2019). 17

    2019

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.