Pith. sign in

REVIEW 3 major objections 5 minor 64 references

Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that vLLM inference-engine configuration choices — attention kernel type and prefix caching in particular — measurably change energy use, latency, and sometimes benchmark accuracy, with no configuration optimal across mode

desk verdict Large, careful factorial study of vLLM options; the latency and interaction findings are solid, but energy rankings rest on 1 Hz nvidia-smi sampling and should be treated as provisional, and the accuracy effect is an open question. read the letter →

arxiv 2607.09172 v2 pith:4AIZO2ZZ submitted 2026-07-10 cs.SE cs.AIcs.PF

Nada Zine , Tristan Coignion , Vincenzo Stoico , Clément Quinton , Ivano Malavolta , Romain Rouvoy , Patricia Lago This is my paper
classification cs.SEcs.AIcs.PF
keywords LLMinferencevenergyconsumptionattentionkernelprefixcachingchunkedprefillParetofrontierbenchmarkreproducibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the inference engine's configuration is not a neutral implementation detail: for a fixed model and task, switching attention kernels or toggling prefix caching can shift total energy, request latency, time-to-first-token, and even benchmark accuracy, while chunked prefill barely matters under vLLM's recommended settings. The evidence is a controlled full-factorial experiment — three attention backends, prefix caching on/off, chunked prefill on/off, five open-weight models, five task types, 9,000 runs with 30 repetitions each — analyzed with nonparametric factorial ANOVA, effect sizes, and Pareto fronts. The practical stakes: if the paper is right, benchmark scores and energy/latency claims are only meaningful when the full inference configuration is reported, and efficiency tuning must be done per model and per workload rather than by picking a single best config.

What carries the argument

The argument rests on a full-factorial controlled experiment: three attention kernels × prefix caching on/off × chunked prefill on/off, crossed with five models and five task datasets, each configuration repeated 30 times (9,000 runs in total). Nonparametric factorial ANOVA with multiple-comparison correction identifies which main effects and interactions are significant; Cliff's delta and dominance statistics measure effect sizes; Pareto dominance decides which configurations are jointly energy/latency/accuracy optimal. The proposed mechanism for accuracy shifts is floating-point non-associativity: attention kernels compute the same mathematical attention but can reduce sums in different or

What would settle it

Run the FlashInfer-versus-FlashAttention-3 contrasts with fixed random seeds, comparing per-token logits, while logging power at high frequency (or with a hardware wattmeter); identical logits and a disappearing energy edge would show the paper's headline accuracy and energy effects are measurement artifacts.

Watch

Extended reading notes

Core claim

Under the studied vLLM setup, attention kernel type and prefix caching have statistically significant effects on energy, latency, and time to first token, and they interact: whether prefix caching helps depends on the attention kernel. Across most model-task pairs, FlashInfer tends to use the least energy while FlashAttention-3 tends to respond fastest, so energy-optimal and latency-optimal choices diverge. Chunked prefill has no meaningful effect because most prompts stay below the default token budget that triggers chunking. Model choice sets the overall regime; configuration tuning moves a deployment only locally along the Pareto frontier. The unexpected result is that attention type and

Load-bearing premise

The energy rankings rest on sampling power once per second and interpolating between samples; if short power spikes are missed unevenly across attention kernels, the measured energy differences — and FlashInfer's apparent advantage — could be biased.

Editorial extensions

If this is right

  • Published benchmark scores can shift with attention backend or prefix-cache setting, so accuracy results are reproducible only when the full inference configuration is reported.
  • Energy and latency can favor different backends, so optimizing a single metric can pick the wrong configuration for the other.
  • Prefix caching usually reduces energy and time-to-first-token, but the paper finds counterexamples (e.g., one model-task pair where it increases energy), so it must be validated per workload.
  • Chunked prefill can be left at vLLM's default for short-prompt workloads; it only becomes relevant when prompts exceed the configured chunking threshold.
  • Attention type and prefix caching interact, so evaluating options independently can lead to suboptimal choices; factorial designs are needed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the floating-point explanation for accuracy shifts, if confirmed by fixed-seed logit comparisons, would generalize: any inference-stack change that reorders arithmetic (quantization, batching, kernel selection) could alter outputs, not just energy.
  • Editorial inference: the 1 Hz power-sampling caveat the paper reports means its energy rankings are estimates; re-running the decisive contrasts with high-frequency metering would tell whether FlashInfer's energy advantage is real or an artifact.
  • Editorial inference: the per-task Pareto fronts suggest a practical decision procedure — choose model first, then tune within the frontier — that could be extended to other vLLM options (quantization, batch size, tensor parallelism) in a configuration-sensitivity benchmark.
  • Editorial inference: because results were obtained on A100 GPUs with vLLM 0.10.2 and offline batch inference, transferring conclusions to other hardware, newer versions, or online serving is an open question; the most direct test is to rerun this factorial design on H100 with online endpoints.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper reports a controlled, full-factorial measurement study of three vLLM configuration options (attention kernel type, prefix caching, chunked prefill) across five open-weight LLMs and five task datasets, totaling 9,000 runs and 93,600 measurements. The authors evaluate energy consumption, latency (TTFT and end-to-end), and accuracy, using ART ANOVA with Holm-Bonferroni correction and effect sizes. They find that attention type and prefix caching are the most impactful options, while chunked prefill shows limited effects under the default vLLM threshold; effects are highly model- and task-dependent, and no configuration dominates. They also report that configuration choices can affect benchmark accuracy, and that model choice dominates the Pareto frontier, with configuration tuning providing only local improvements.

Significance. If the results hold, this is a valuable empirical contribution to a practically important and understudied area: it provides a large, openly available dataset, a factorial design that captures interactions, and a clear warning that inference-engine configuration is not output-neutral. The study's strengths include the scale and randomization of the experiment, the use of non-parametric factorial analysis with multiple-comparison correction, the explicit scoping of chunked prefill to the default threshold, and the provision of a replication package. However, the headline energy-ranking claims rest on a measurement method with acknowledged systematic bias, and the accuracy finding is presented as a result while the paper itself stops short of the validation needed to separate it from stochastic noise. These issues are central to the paper's main claims and require additional analysis or careful reframing before publication.

major comments (3)
  1. [§IV and §VII] Energy is computed as E=W×T using nvidia-smi sampled at 1 Hz. Section VII concedes that on A100 GPUs nvidia-smi samples power for only about 25% of runtime and can incur errors up to 65% on spiky workloads. Thirty repetitions reduce random noise (CV=0.043) but do not remove systematic, configuration-dependent bias. Because the three attention kernels have different execution and power profiles, the bias is not necessarily common across configurations; the FlashInfer energy advantage in Table IV, the attention×task energy interactions in Figure 3, and the energy axis of the Pareto front in Table V/Fig. 5 could all be distorted. The local differences being ranked are small (3–7% in several Pareto comparisons), so a 10–20% bias is material. Please validate a subset using high-frequency power measurement (e.g., DCGM or a hardware power meter) and report error bounds, or substantially soften
  2. [§V.A.5 and §VI(a)] The abstract and contribution (iv) state that inference options can affect model accuracy, but the evidence is not conclusive under the paper's own discussion. Decoding uses temperature 0.4 and top-p 0.95 with sampling, and the paper states that disentangling a real effect from statistical noise would require repeated runs with fixed seeds and per-token logit comparison, which is left to future work. The reported accuracy differences (e.g., Qwen-4B on LB, where FlashInfer scored 0.37–0.43 vs. 0.20–0.35 for FlashAttention) could arise from stochastic generation interacting with numerical reordering. Since this is a headline finding with strong implications for benchmark reproducibility, the claim should either be validated with a controlled seed/logit analysis or explicitly reclassified as a preliminary, hypothesis-generating observation throughout the abstract, contributions, and conclus
  3. [§IV and Table I] The paper states that all selected models 'have fewer than 40B active parameters to fit into a single GPU' and reports running on Nvidia A100-SXM4-40GB. Qwen3-32B has 32B parameters; at the default bf16 precision the weights alone require roughly 64GB, so it cannot fit in a 40GB GPU unless quantized, split across GPUs, or offloaded. The paper never states the dtype, quantization format, tensor-parallel size, or any memory-saving mechanism used for this model. If Qwen-32B was quantized or split across multiple GPUs, the energy/latency comparisons involving it are not on equal footing with the other models, and the single-GPU criterion is misleading. Please report the exact loading/serving configuration used for Qwen-32B and justify that its measurements are directly comparable to the other models, or remove this model from the study.
minor comments (5)
  1. [Table IV] The caption says 'Holm and Cliff’s δ' but the table does not define the sign convention for Cliff's delta. A positive value is presumably the first listed option having higher energy, but this should be stated explicitly, especially because some contrasts (e.g., f3−fi for Magistral-24B on LB) have negative signs with large absolute values and the text says FlashInfer 'generally' has an advantage.
  2. [Table V] The symbols ✓ and × are used for prefix caching and chunked prefill but are not defined in the caption or table footnotes. Please add a legend, e.g., ✓=on, ×=off.
  3. [§V.B.1] The sentence 'The largest decreases' is incomplete; it is followed by a new paragraph without a predicate. This appears to be a copy-editing error.
  4. [§V.B.2] The statement that chunked prefill shows no significant task-level interaction (0/15) should explicitly include the caveat that under the default max_num_batched_tokens of 8192, chunking is rarely triggered for the short-prompt tasks. While this is discussed in §VI and §VII, the results section alone can be misread as a general negative result.
  5. [Fig. 1] The x-axis label is rendered as 'T otal Energy (kJ)' in the figure; the spacing should be corrected to 'Total Energy (kJ)'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the paper is an empirical measurement study; self-citations are not load-bearing and the accuracy and energy limitations are validity threats, not circularity.

full rationale

This is a controlled measurement study, not a derivation in which outputs are constructed from inputs. Energy is obtained by sampling power (nvidia-smi/perf at 1 Hz) and computing E=W×T; latency and TTFT are directly logged by vLLM; accuracy is computed with standard benchmark scorers. There is no fitted parameter that is later renamed as a prediction, and no equation in which a claimed result is identical to a defining assumption. The self-citations to prior work by the same authors are not load-bearing: [34] provides the AssistantTraces dataset as one of five task corpora, and [18] is cited for the infeasibility of exhaustive configuration exploration and for related work; neither determines the measured energy, latency, or accuracy outcomes. The paper also explicitly brackets its own limitations: Section VII concedes that nvidia-smi samples only ~25% of runtime on A100s and can carry up to 65% error on spiky workloads, which is a measurement validity threat rather than a circularity; Section VI states that chunked prefill was often not triggered under the default max_num_batched_tokens=8192, making the 'limited effect' claim explicitly conditional; and Section VI reports that the accuracy differences may be statistical noise rather than a predicted effect. None of these steps reduce a claimed result to its inputs by construction. Therefore no significant circularity is present.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

Measurement study; no free parameters or invented entities. Load-bearing premises are measurement validity, representativeness of the configuration space, and the neutrality of system-level optimizations.

assumptions (5)
  • domain assumption nvidia-smi 1 Hz power sampling with interpolation yields energy estimates accurate enough to rank configurations.
    Load-bearing for all energy conclusions; acknowledged in Section VII (Construct Validity) but not mitigated.
  • domain assumption Holding all other vLLM options at defaults is representative of common deployments and does not mask interaction effects.
    Section VII Internal Validity; the factorial design only varies 3 of dozens of options.
  • domain assumption The selected 5 models and 5 tasks are representative of LLM inference workloads.
    External validity; limits generalization.
  • standard math ART ANOVA with Holm-Bonferroni correction controls Type I error for the 25 tests per metric.
    Section III-F.
  • domain assumption Attention kernel and prefix caching should be output-neutral; any observed accuracy differences are presumed to be artifacts or floating-point non-associativity.
    Section VI-a; the accuracy-effect claim rests on this assumption being false, but the default hypothesis is neutrality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations." pith.science (2026). https://pith.science/paper/4AIZO2ZZ

@misc{pith2026260709172,
  author       = {Pith},
  title        = {Pith review of: Attention to Detail: Evaluating Energy, Performance, and Accuracy Trade-offs Across vLLM Configurations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4AIZO2ZZ}},
  note         = {Machine review of arXiv:2607.09172}
}
abstract

Large Language Models are reshaping how software is developed and maintained. They are typically deployed in production using inference engines such as vLLM, which can efficiently serve pre-trained, highly configurable models. While prior work has focused on model architectures and hardware acceleration, the impact of inference engine configuration on energy consumption, performance, and output quality remains poorly understood. In this paper, we present a large-scale controlled study of three selected vLLM configuration options: attention kernel type, prefix caching, and chunked prefill. We evaluate all combinations of these configurations across 5 open-weight LLMs and 5 diverse inference tasks, totaling $9,000$ runs and $93,600$ measures. We analyze energy consumption, latency, and accuracy, and examine both main effects and interaction effects between configuration options and tasks. Our results show that the studied configuration options significantly impact energy and performance, mainly driven by attention type and prefix caching, while chunked prefill has a limited effect under the default vLLM serving configuration and evaluated workloads. These effects are highly model- and workload-dependent, and no configuration is universally optimal. We further show that model choice dominates global trade-offs, while configuration tuning provides local improvements along the Pareto frontier. Unexpectedly, inference options can also affect model accuracy.

Figures

Figures reproduced from arXiv: 2607.09172 by the authors.

Figure 1
Figure 1. Total energy consumption (kJ) by model, task, [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Representative 2-way interaction effects on end-to-end [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Task and attention type interaction on total energy [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Difference in total energy consumption (prefix caching [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Two-dimensional projection of the EvoEval Pareto [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 12 linked inside Pith

  1. [1]

    Gpts are gpts: Labor market impact potential of llms,

    T. Eloundou, S. Manning, P. Mishkin, and D. Rock, “Gpts are gpts: Labor market impact potential of llms,”Science, vol. 384, no. 6702, pp. 1306–1308, 2024

    2024

  2. [2]

    An analysis of large language models: their impact and potential applications,

    G. Bharathi Mohan, R. Prasanna Kumar, P. Vishal Krishh, A. Keerthi- nathan, G. Lavanya, M. K. U. Meghana, S. Sulthana, and S. Doss, “An analysis of large language models: their impact and potential applications,”Knowledge and Information Systems, vol. 66, no. 9, pp. 5047–5070, 2024

    2024

  3. [3]

    The impact of llm chatbots on learning outcomes in advanced driver assistance systems education,

    M. Murtaza, C.-T. Cheng, B. M. Albahlal, M. M. A. Muslam, and M. S. Raza, “The impact of llm chatbots on learning outcomes in advanced driver assistance systems education,”Scientific Reports, vol. 15, no. 1, p. 7260, 2025

    2025

  4. [4]

    Recalibrating global artificial intelligence e-waste estimates,

    A. d. Vries-Gao, “Recalibrating global artificial intelligence e-waste estimates,”Resources, Conservation and Recycling, vol. 229, p. 108872, 2026. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S0921344926000960

    2026

  5. [5]

    Towards greener llms: Bringing energy-efficiency to the forefront of llm infer- ence,

    J. Stojkovic, E. Choukse, C. Zhang, I. Goiri, and J. Torrellas, “Towards greener llms: Bringing energy-efficiency to the forefront of llm infer- ence,”arXiv preprint arXiv:2403.20306, 2024

    arXiv 2024

  6. [6]

    How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference,

    N. Jegham, M. Abdelatti, C. Y . Koh, L. Elmoubarki, and A. Hendawi, “How hungry is ai? benchmarking energy, water, and carbon footprint of llm inference,”arXiv preprint arXiv:2505.09598, 2025

    arXiv 2025

  7. [7]

    Llmco2: Advancing accu- rate carbon footprint prediction for llm inferences,

    Z. Fu, F. Chen, S. Zhou, H. Li, and L. Jiang, “Llmco2: Advancing accu- rate carbon footprint prediction for llm inferences,”ACM SIGENERGY Energy Informatics Review, vol. 5, no. 2, pp. 63–68, 2025

    2025

  8. [8]

    Characterizing power management opportunities for llms in the cloud,

    P. Patel, E. Choukse, C. Zhang, ´I. Goiri, B. Warrier, N. Mahalingam, and R. Bianchini, “Characterizing power management opportunities for llms in the cloud,” inProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, 2024, pp. 207–222

    2024

Show all 64 references
  1. [9]

    Sustainable llm serving: Environmental implica- tions, challenges, and opportunities,

    Y . Ding and T. Shi, “Sustainable llm serving: Environmental implica- tions, challenges, and opportunities,” in2024 IEEE 15th International Green and Sustainable Computing Conference (IGSC). IEEE, 2024, pp. 37–38

    2024

  2. [10]

    A survey on in- ference engines for large language models: Perspectives on optimization and efficiency,

    S. Park, S. Jeon, C. Lee, S. Jeon, B.-S. Kim, and J. Lee, “A survey on in- ference engines for large language models: Perspectives on optimization and efficiency,”preprint arXiv:2505.01658, 2025

    2025

  3. [11]

    Llm inference serving: Survey of recent advances and opportunities,

    B. Li, Y . Jiang, V . Gadepally, and D. Tiwari, “Llm inference serving: Survey of recent advances and opportunities,” in2024 IEEE High Performance Extreme Computing Conference (HPEC). IEEE, 2024, pp. 1–8

    2024

  4. [12]

    Efficient memory management for large language model serving with pagedattention,

    W. Kwon, Z. Li, S. Zhuang, Y . Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, “Efficient memory management for large language model serving with pagedattention,” inProceedings of the 29th symposium on operating systems principles, 2023, pp. 611–626

    2023

  5. [13]

    Huggingface’s trans- formers: State-of-the-art natural language processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowiczet al., “Huggingface’s trans- formers: State-of-the-art natural language processing,”arXiv preprint arXiv:1910.03771, 2019

    1910 arXiv

  6. [14]

    Using ollama,

    F. S. Marcondes, A. Gala, R. Magalh ˜aes, F. Perez de Britto, D. Dur ˜aes, and P. Novais, “Using ollama,” inNatural Language Analytics with Generative Large-Language Models: A Practical Approach with Ollama and Open-Source LLMs. Springer, 2025, pp. 23–35

    2025

  7. [15]

    ggml-org/llama.cpp,

    “ggml-org/llama.cpp,” Mar. 2026. [Online]. Available: https://github. com/ggml-org/llama.cpp

    2026

  8. [16]

    Seesaw: High-throughput llm inference via model re-sharding,

    Q. Su, W. Zhao, X. Li, M. Andoorveedu, C. Jiang, Z. Zhu, K. Song, C. Giannoula, and G. Pekhimenko, “Seesaw: High-throughput llm inference via model re-sharding,”preprint arXiv:2503.06433, 2025

    2025 arXiv

  9. [17]

    Towards green ai: Decoding the energy of llm inference in software development,

    L. Solovyeva and F. Castor, “Towards green ai: Decoding the energy of llm inference in software development,”preprint arXiv:2602.05712, 2026

    2026

  10. [18]

    Pimp My LLM: Leveraging Variability Modeling to Tune Inference Hyperparameters,

    N. Zine, C. Quinton, and R. Rouvoy, “Pimp My LLM: Leveraging Variability Modeling to Tune Inference Hyperparameters,” inEASE’26 - 30th International Conference on Evaluation and Assessment in Software Engineering, Glasgow, United Kingdom, Jun. 2026. [Online]. Available: https:...

    2026

  11. [19]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H ¨ost, M. Ohlsson, B. Regnell, and A. Wessl ´en,Experimentation in Software Engineering, ser. Computer Science. Springer, 2012

    2012

  12. [20]

    Replication package of this study,

    Anonymous, “Replication package of this study,” Aug. 2026. [Online]. Available: https://doi.org/10.5281/zenodo.19253906

    2026 doi

  13. [21]

    Sustainable ai: Environmental implications, challenges and opportunities,

    C.-J. Wu, R. Raghavendra, U. Gupta, B. Acun, N. Ardalani, K. Maeng, G. Chang, F. Aga, J. Huang, C. Baiet al., “Sustainable ai: Environmental implications, challenges and opportunities,”Proceedings of machine learning and systems, vol. 4, pp. 795–813, 2022

    2022

  14. [22]

    Dy- namollm: Designing llm inference clusters for performance and energy efficiency,

    J. Stojkovic, C. Zhang, ´I. Goiri, J. Torrellas, and E. Choukse, “Dy- namollm: Designing llm inference clusters for performance and energy efficiency,” in2025 IEEE International Symposium on High Perfor- mance Computer Architecture (HPCA). IEEE, 2025, pp. 1348–1362

    2025

  15. [23]

    Llm-pilot: Characterize and opti- mize performance of your llm inference services,

    M. Lazuka, A. Anghel, and T. Parnell, “Llm-pilot: Characterize and opti- mize performance of your llm inference services,” inSC24: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2024, pp. 1–18

    2024

  16. [24]

    From words to watts: Benchmarking the energy costs of large language model inference,

    S. Samsi, D. Zhao, J. McDonald, B. Li, A. Michaleas, M. Jones, W. Bergeron, J. Kepner, D. Tiwari, and V . Gadepally, “From words to watts: Benchmarking the energy costs of large language model inference,” inIEEE High Performance Extreme Computing Conference (HPEC). IEEE, 2023, pp. 1–9

    2023

  17. [25]

    Power hungry processing: Watts driving the cost of ai deployment?

    S. Luccioni, Y . Jernite, and E. Strubell, “Power hungry processing: Watts driving the cost of ai deployment?” inProceedings of the 2024 ACM conference on fairness, accountability, and transparency, 2024, pp. 85– 99

    2024

  18. [26]

    Investigating energy efficiency and performance trade-offs in llm inference across tasks and dvfs settings,

    P. J. Maliakel, S. Ilager, and I. Brandic, “Investigating energy efficiency and performance trade-offs in llm inference across tasks and dvfs settings,”arXiv preprint arXiv:2501.08219, 2025

    2025

  19. [27]

    Offline energy-optimal llm serv- ing: Workload-based energy models for llm inference on heterogeneous systems,

    G. Wilkins, S. Keshav, and R. Mortier, “Offline energy-optimal llm serv- ing: Workload-based energy models for llm inference on heterogeneous systems,”ACM SIGENERGY Energy Informatics Review, vol. 4, no. 5, pp. 113–119, 2024

    2024

  20. [28]

    Sustainable llm inference for edge ai: Evaluating quantized llms for energy efficiency, output accuracy, and inference latency,

    E. J. Husom, A. Goknil, M. Astekin, L. K. Shar, A. K ˚asen, S. Sen, B. A. Mithassel, and A. Soylu, “Sustainable llm inference for edge ai: Evaluating quantized llms for energy efficiency, output accuracy, and inference latency,”arXiv preprint arXiv:2504.03360, 2025

    2025 arXiv

  21. [29]

    Understanding efficiency: Quantization, batching, and serving strategies in llm energy use,

    J. Delavande, R. Pierrard, and S. Luccioni, “Understanding efficiency: Quantization, batching, and serving strategies in llm energy use,”arXiv preprint arXiv:2601.22362, 2026

    2026

  22. [30]

    Small prompts, big energy and co 2 impact: Benchmarking ollama llms on cpu and gpu,

    A. Kolovska, M. Gusev, and D. Mileski, “Small prompts, big energy and co 2 impact: Benchmarking ollama llms on cpu and gpu,” in2025 33rd Telecommunications Forum (TELFOR). IEEE, 2025, pp. 1–4

    2025

  23. [31]

    A thorough examination of decoding methods in the era of llms, 2024,

    C. Shi, H. Yang, D. Cai, Z. Zhang, Y . Wang, Y . Yang, and W. Lam, “A thorough examination of decoding methods in the era of llms, 2024,” URL https://arxiv. org/abs/2402.06925

    2024 arXiv

  24. [32]

    Decoding decoded: Understanding hyperparameter effects in open-ended text generation,

    E. G. Arias, M. Li, C. Heumann, and M. Aßenmacher, “Decoding decoded: Understanding hyperparameter effects in open-ended text generation,” inProceedings of the 31st International Conference on Computational Linguistics, 2025, pp. 9992–10 020

    2025

  25. [33]

    Impact of decoding strategies on gpu energy usage in large language model text generation,

    A. Nik, M. A. Riegler, and P. Halvorsen, “Impact of decoding strategies on gpu energy usage in large language model text generation,”Scientific Reports, 2025

    2025

  26. [34]

    Green my llm: Studying the key factors affecting the energy consumption of code assistants,

    T. Coignion, C. Quinton, and R. Rouvoy, “Green my llm: Studying the key factors affecting the energy consumption of code assistants,”arXiv preprint arXiv:2411.11892, 2024

    2024 arXiv

  27. [35]

    The impact of hyperparameters on large language model inference performance: An evaluation of vllm and huggingface pipelines,

    M. Martinez, “The impact of hyperparameters on large language model inference performance: An evaluation of vllm and huggingface pipelines,” inProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, 2025, pp. 1672–1678

    2025

  28. [36]

    Energy considerations of large language model inference and efficiency optimizations,

    J. Fernandez, C. Na, V . Tiwari, Y . Bisk, S. Luccioni, and E. Strubell, “Energy considerations of large language model inference and efficiency optimizations,”preprint arXiv:2504.17674, 2025

    2025 arXiv

  29. [37]

    Benchmarking Energy Efficiency of Large Language Models Using vLLM,

    K. Pronk and Q. Zhao, “Benchmarking Energy Efficiency of Large Language Models Using vLLM,” Sep. 2025

    2025

  30. [38]

    Energy Efficient or Exhaus- tive? Benchmarking Power Consumption of LLM Inference Engines,

    C. Niu, W. Zhang, Y . Zhao, and Y . Chen, “Energy Efficient or Exhaus- tive? Benchmarking Power Consumption of LLM Inference Engines,” SIGENERGY Energy Inform. Rev., vol. 5, no. 2, pp. 56–62, 2025

    2025

  31. [39]

    Shull, J

    F. Shull, J. Singer, and D. I. Sjøberg,Guide to advanced empirical software engineering. Springer, 2007

    2007

  32. [40]

    A. Guldner andet al., “Development and evaluation of a reference measurement model for assessing the resource and energy efficiency of software products and components-green software measurement model (gsmm),”Future Generation Computer Systems, vol. 155, pp. 402–418, 2024

    2024

  33. [41]

    A Beginner’s Guide to Power and Energy Measurement and Estimation for Computing and Machine Learning,

    A. Jagannadharao, N. Beckage, S. Biswas, H. Egan, J. Gafur, T. Metsch, D. Nafus, G. Raffa, and C. Tripp, “A Beginner’s Guide to Power and Energy Measurement and Estimation for Computing and Machine Learning,” Dec. 2024

    2024

  34. [42]

    The tame project: Towards improvement-oriented software environments,

    V . R. Basili and H. D. Rombach, “The tame project: Towards improvement-oriented software environments,”IEEE Transactions on Software Engineering, vol. 14, no. 6, pp. 758–773, 1988

    1988

  35. [43]

    vllm – configuration options,

    “vllm – configuration options,” 2026. [Online]. Available: https: //docs.vllm.ai/en/latest/configuration

    2026

  36. [44]

    Flashattention: Fast and memory-efficient exact attention with io-awareness,

    T. Dao, D. Fu, S. Ermon, A. Rudra, and C. R ´e, “Flashattention: Fast and memory-efficient exact attention with io-awareness,”Advances in neural information processing systems, vol. 35, pp. 16 344–16 359, 2022

    2022

  37. [45]

    Flashattention-2: Faster attention with better parallelism and work partitioning,

    T. Dao, “Flashattention-2: Faster attention with better parallelism and work partitioning,”arXiv preprint arXiv:2307.08691, 2023

    2023 arXiv

  38. [46]

    Flashinfer: Efficient and customizable attention engine for llm inference serving,

    Z. Ye, L. Chen, R. Lai, W. Lin, Y . Zhang, S. Wang, T. Chen, B. Kasikci, V . Grover, A. Krishnamurthyet al., “Flashinfer: Efficient and customizable attention engine for llm inference serving,”arXiv preprint arXiv:2501.01005, 2025

    2025 arXiv

  39. [47]

    On the suitability of hug- ging face hub for empirical studies,

    A. Ait, J. L. C ´anovas Izquierdo, and J. Cabot, “On the suitability of hug- ging face hub for empirical studies,”Empirical Software Engineering, vol. 30, no. 2, p. 57, 2025

    2025

  40. [48]

    Text Generation Models – Hugging Face — huggingface.co,

    “Text Generation Models – Hugging Face — huggingface.co,” https:// huggingface.co/models?pipeline tag=text-generation&sort=downloads, 2025

    2025

  41. [49]

    Top leaderboard ranking= top coding proficiency, always? evoeval: Evolving coding benchmarks via llm,

    C. S. Xia, Y . Deng, and L. Zhang, “Top leaderboard ranking= top coding proficiency, always? evoeval: Evolving coding benchmarks via llm,” arXiv preprint arXiv:2403.19114, 2024

    2024 arXiv

  42. [50]

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks,

    Y . Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y . Donget al., “Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics ...

    2025

  43. [51]

    Natural questions: a benchmark for question answering research,

    T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Leeet al., “Natural questions: a benchmark for question answering research,”Transactions of the Association for Computational Linguistics, vol. 7, pp. 453–466, 2019

    2019

  44. [52]

    Lighteval: A lightweight framework for llm evaluation,

    N. Habib, C. Fourrier, H. Kydl ´ıˇcek, T. Wolf, and L. Tunstall, “Lighteval: A lightweight framework for llm evaluation,” 2023. [Online]. Available: https://github.com/huggingface/lighteval

    2023

  45. [53]

    Taming the long-tail: Efficient reasoning rl training with adaptive drafter,

    Q. Hu, S. Yang, J. Guo, X. Yao, Y . Lin, Y . Gu, H. Cai, C. Gan, A. Klimovic, and S. Han, “Taming the long-tail: Efficient reasoning rl training with adaptive drafter,”preprint arXiv:2511.16665, 2025

    2025

  46. [54]

    Wildchat: 1m chatgpt interaction logs in the wild,

    W. Zhao, X. Ren, J. Hessel, C. Cardie, Y . Choi, and Y . Deng, “Wildchat: 1m chatgpt interaction logs in the wild,”preprint arXiv:2405.01470, 2024

    2024 arXiv

  47. [55]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    2021 arXiv

  48. [56]

    Pareto multi objective optimization,

    P. Ngatchou, A. Zarei, and A. El-Sharkawi, “Pareto multi objective optimization,” inProceedings of the 13th International Conference on, Intelligent Systems Application to Power Systems, 2005, pp. 84–91

    2005

  49. [57]

    The aligned rank transform for nonparametric factorial analyses using only anova procedures,

    J. O. Wobbrock, L. Findlater, D. Gergle, and J. J. Higgins, “The aligned rank transform for nonparametric factorial analyses using only anova procedures,” inProceedings of the SIGCHI conference on human factors in computing systems, 2011, pp. 143–146

    2011

  50. [58]

    A simple sequentially rejective multiple test procedure,

    S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian journal of statistics, pp. 65–70, 1979

    1979

  51. [59]

    Dominance statistics: Ordinal analyses to answer ordinal questions

    N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinal questions.”Psychological bulletin, vol. 114, no. 3, p. 494, 1993

    1993

  52. [60]

    A critique and improvement of the cl common language effect size statistics of mcgraw and wong,

    A. Vargha and H. D. Delaney, “A critique and improvement of the cl common language effect size statistics of mcgraw and wong,”Journal of Educational and Behavioral Statistics, vol. 25, no. 2, pp. 101–132, 2000

    2000

  53. [61]

    The Curious Case of Neural Text Degeneration,

    A. Holtzman, J. Buys, L. Du, M. Forbes, and Y . Choi, “The Curious Case of Neural Text Degeneration,” Feb. 2020

    2020

  54. [62]

    vllm – optimization and tuning,

    “vllm – optimization and tuning,” 2026. [Online]. Available: https: //docs.vllm.ai/en/latest/configuration/optimization.html

    2026

  55. [63]

    Guidelines for Empirical Studies in Software Engineering involving Large Language Models,

    S. Baltes, F. Angermeir, C. Arora, M. M. Bar ´on, C. Chen, L. B ¨ohme, F. Calefato, N. Ernst, D. Falessi, B. Fitzgerald, D. Fucci, M. Kalinowski, S. Lambiase, D. Russo, M. Lungu, L. Prechelt, P. Ralph, R. van Tonder, C. Treude, and S. Wagner, “Guidelines for Empirical Studies ...

    2025

  56. [64]

    Part-time Power Measurements: Nvidia-smi’s Lack of Attention,

    Z. Yang, K. Adamek, and W. Armour, “Part-time Power Measurements: Nvidia-smi’s Lack of Attention,” inSC24: International Conference for High Performance Computing, Networking, Storage and Analysis, Nov. 2024, pp. 1–17

    2024

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.