Language Models in Software Development Tasks: An Experimental Analysis of Energy and Accuracy

Boris Belchev; Fernando Castor; Negar Alizadeh; Nishant Saurabh; Patricia Kelbert

REVIEW 4 major objections 5 minor 4 cited by

Language Models in Software Development Tasks: An Experimental Analysis of Energy and Accuracy

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper measures inference energy and accuracy for 18 language-model families across four software development tasks and two GPUs.

desk verdict Useful measurement matrix on LLM energy/accuracy trade-offs for coding tasks; main caveat is an unstated idle-power subtraction that needs checking before the numbers are trusted. read the letter →

arxiv 2412.00329 v2 pith:AGX6555O submitted 2024-11-30 cs.SE

Negar Alizadeh , Boris Belchev , Nishant Saurabh , Patricia Kelbert , Fernando Castor This is my paper · ORCID

classification cs.SE

keywords energyefficiencyLLMinferencemodelquantizationGGUFHumanEvalPackParetofrontsoftwaredevelopmenttaskspass@1

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether spending more energy on a locally deployed language model buys better results on everyday software development tasks. Across 18 model families, four tasks (code generation, bug fixing, docstring generation, test generation), full-precision and quantized variants, and two GPUs, it measures both inference energy and pass@1 accuracy. The central finding is that the energy-accuracy trade-off is not strict: larger models usually draw far more power while performing about the same as, or worse than, smaller models, and quantized versions of large models often beat full-precision medium models on both energy and accuracy. The paper concludes that no single model is best for all tasks, so model selection should be task-specific and informed by measured trade-offs. If right, this gives developers a practical argument for choosing smaller or quantized local models instead of assuming bigger is better.

What carries the argument

The quantitative backbone is a controlled measurement setup: GPU power sampled at 10 Hz with pyNVML, Intel RAPL for CPU energy, fixed hyperparameters (temperature 0.1, top-p 0.95), and identical prompts across models on the Python subset of HumanEvalPack. Accuracy is pass@1 against the benchmark's test assertions, docstrings judged by GPT-4o-mini, and the energy-accuracy tension is visualized with Pareto fronts. The architecture-efficiency link is quantified with Spearman correlations (Bonferroni-corrected) between model characteristics and metrics like tokens per joule.

What would settle it

Re-run the benchmark on the same GPUs using a wall-plug power meter or a sampling rate above 10 Hz, and verify whether subtracting the reported idle-power baselines changes the per-model Wh values; if the Pareto fronts shift materially, the conclusion that bigger models do not buy accuracy would need qualification.

Watch

Extended reading notes

Core claim

The paper claims that in software-development inference, higher energy expenditure does not reliably translate into higher accuracy, and in several cases a quantized small model dominates a full-precision larger one on both objectives simultaneously. It shows this with Pareto fronts per task: the frontier models are often not the largest, and general-purpose models like gemma:2b appear near the ideal point for docstring and test generation. It also reports that tokens per joule is nearly constant for a given model across tasks, so a model's efficiency can be estimated from its architecture plus expected output length.

Load-bearing premise

All comparisons rest on the measured watt-hours, which depend on the 10 Hz GPU power sampling and on how the reported idle powers (9.92 W for the RTX 3070, 46.82 W for the A100) were treated relative to the active measurements.

Editorial extensions

If this is right

Model selection for local LLM deployment should be driven by the target task and by measured energy-accuracy trade-offs, not by parameter count alone.
Quantized large models are often the best compromise when a single model must serve several tasks, since they can beat full-precision medium models on both energy and accuracy.
The tokens-per-joule metric allows estimating a model's inference energy for a new task from its architecture and expected output length, before running a full benchmark.
Code-specific models dominate only in code generation; fine-tuning them for docstring and bug-fixing tasks is a promising direction for improving both accuracy and efficiency.
Energy benchmarking of inference should include CPU energy, which accounted for roughly 16% of total energy on the tested laptop GPU setup.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The finding implies that energy-aware model selection could be offered as a service by model registries, presenting Pareto fronts per task so users can pick a model from measured data rather than from reputation or size.
The strong correlation between architecture size and tokens per joule suggests that energy estimates could be derived from model-card metadata alone, before running any benchmark, which would make energy-aware selection far cheaper to apply at scale.
A testable extension is to run the same 18 models on a CPU-only machine, since the paper's measurements show CPU energy is non-negligible and quantized large models might behave differently without a GPU.
The paper's null result on accuracy versus parameter count is specific to HumanEvalPack; extending to more complex, multi-file programming tasks could change the Pareto fronts and should be checked before generalizing the advice.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look: this is the broadest comparison I know of that puts energy and accuracy side by side for LLMs in coding tasks. Eighteen model families, three precision formats, four tasks, two GPUs—that measurement matrix is a legitimate new contribution. The headline finding, that spending more energy doesn't reliably buy more accuracy and that quantized large models can dominate full-precision medium ones on both axes, is supported by the displayed Pareto fronts and Table II. The authors also did the right things: fixed hyperparameters, fixed seed, isolated GPU, three runs, public replication package. The tokens/J efficiency metric in Section V-C is a useful normalization.

Now the soft spots. The load-bearing issue is in Section IV-B. They report idle powers for both GPUs (9.92W and 46.82W) but never explicitly state that these baselines were subtracted from the active Wh values. If idle power was not subtracted, every energy number carries an additive component proportional to runtime. Larger and full-precision models run longer, so that component would inflate their energy disproportionately and could create or exaggerate the paper's central pattern. On the A100, 46.82W over 30 minutes is about 23Wh, the same order as many reported task totals. The text says idle power was measured 'to measure the energy usage of the hardware and OS when idle,' which suggests subtraction was intended, but the methodology never says so. The replication package should settle this, but it was not inspected here. That's a concrete reproducibility gap, not a fundamental flaw.

Minor concerns: three-run averages come without variance; docstring accuracy depends on GPT-4o-mini as judge without validation; the model sample is a convenience sample capped at 20B parameters, using only Python and HumanEvalPack. The authors acknowledge the last two in Threats to Validity, which is honest. The central descriptive claim likely survives the minor issues; the idle-power subtraction is the one thing that could move the Pareto fronts.

Bottom line: this deserves a serious referee. A revision that states the idle-power subtraction explicitly, reports per-run data or error bars, and validates the judge would make it solid. I'd cite it as a benchmark in green-software work. Send it to review.

Referee Report

4 major / 5 minor

Summary. This manuscript reports an experimental measurement study of inference energy use and task accuracy for 18 open-LLM families, in full-precision and GGUF-quantized variants, on four HumanEvalPack Python tasks (code generation, bug fixing, docstring generation, and test generation). Experiments were run on two infrastructures, an NVIDIA A100 and an RTX 3070, with GPU power sampled at 10 Hz via pyNVML and CPU energy via RAPL; each configuration was run three times and averaged. The central descriptive finding is that accuracy and energy do not always trade off: larger models often consume substantially more energy without commensurate accuracy gains, quantized versions of large models can dominate full-precision medium models on both axes, and the best model varies by task.

Significance. If the measurements are sound, this is a useful empirical contribution to green-AI and model-selection research for software engineering. Its strengths include the breadth of models (18 families, 54 variants), fixed hyperparameters, test-execution-based evaluation for three of the four tasks, a replication package, and falsifiable Pareto-front and correlation claims. The paper also makes a practical point that task-specific model selection matters. However, because the central claims rest on the energy axis and on the docstring-accuracy axis, the unclear idle-power accounting, missing uncertainty reporting, and unvalidated LLM-as-judge step are material. The findings are plausible in direction but need measurement-reporting revisions before they can be accepted.

major comments (4)

[Section IV-B] The methodology reports idle power values (9.92 W for the RTX 3070, 46.82 W for the A100, and 1.92 W for the CPU) but never explicitly states whether these baselines were subtracted from the Wh values used in Figures 2-4 and Table II. Since total energy is computed as time times average power, including idle draw adds a component proportional to runtime. Because longer-running full-precision and larger models would receive a larger additive bias, this can change relative energy rankings and Pareto-front membership; for a 30-minute A100 run the idle contribution alone is about 23.4 Wh, which is the same order as many reported task energies. Please state the exact computation, include the subtraction in the replication code, and, if idle power was not subtracted, re-run the analysis. The omission should also be acknowledged in Section VI.
[Section IV-B / Section V-B] No variance or uncertainty information accompanies the three-run averages. Many comparisons in Section V-B are close, such as the roughly 2% energy differences between some q4 and q8 variants or the 4.27% versus 3.05% accuracy improvements cited for phi3 and starcoder2, and without confidence intervals it is not possible to determine whether the reported orderings and Pareto-front memberships are stable. Please report per-run spread, standard deviations, or a sensitivity analysis for at least the Pareto-front comparisons. This is load-bearing because the central 'dominates' claims depend on relative energy and accuracy values.
[Section IV-D] Docstring accuracy is evaluated by having GPT-4o-mini generate code from the generated docstring and then applying pass@1, but the manuscript does not validate this judge against ground truth, a manual sample, or an alternative judge. Because docstring generation is one of the four tasks used in the task-level conclusions, a systematic judge bias could affect which models lie on the Pareto front in Figure 4(c). Please add a validation subset, an agreement measure, or a sensitivity comparison with another judge, and report the results.
[Figure 4] The caption states that data points exceeding the y-axis limit are not displayed. A Pareto-front plot is only correct if all non-dominated points are shown; hiding points outside the axis range can omit points that belong on the frontier. Please either include all points using an inset or broken axis, or explicitly report that all omitted points are dominated by displayed points and identify them.

minor comments (5)

[Table II] Table II is difficult to parse as rendered: the header groups for Accuracy, C1, C2, and Efficiency do not align unambiguously with the numeric fields. Please reformat the table or split it into separate tables so each task column is clearly identified.
[Section V-A] The text states that CPU energy accounts for approximately 16% of total energy on the RTX 3070, but it does not specify whether the CPU idle baseline of 1.92 W was subtracted. This should be clarified in the methodology alongside the GPU idle-power treatment.
[Figure 4] The Pareto-front figures do not label their axes. Adding explicit axis labels for energy (Wh) and accuracy (%) would make the plots self-contained.
[Section V-C] The claim that the 4-bit variant had the lowest energy usage in 212 out of 216 analyzed cases needs a clear definition of the denominator; please state that this corresponds to 54 model variants across four tasks and list the exceptions.
[Section VI] The threats-to-validity section is candid about prompt templates, incomplete Ollama responses, code extraction, and dataset coverage, but it does not discuss the idle-power accounting or the uncertainty of the energy measurements; these should be added given their importance to the central claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's claims are direct empirical measurements and descriptive analyses, with no fitted-input predictions or load-bearing self-citations.

full rationale

This is an empirical measurement study. Energy is measured with pyNVML and pyRAPL ('The energy usage is then determined as the product of time and average power'), accuracy is measured with HumanEvalPack pass@1, and the reported Pareto fronts, comparisons, and correlations are descriptive summaries of those measurements. No parameter is fitted to a subset of the data and then used to predict a closely related quantity; no 'prediction' reduces by construction to an input. The self-citations to Alizadeh and Castor [41] and Castor [43] are methodological references for power-sampling tools and rates, and they do not determine any of the paper's conclusions about model energy or accuracy. The central claims about larger models having larger energy footprints without proportional accuracy gains are read directly from Table II and Figure 4, not derived from a definition or from a cited uniqueness theorem. The legitimate concern raised in the skeptical analysis, namely that idle power values (9.92W for RTX 3070, 46.82W for A100) are reported but the text does not explicitly state whether they were subtracted from the active measurement Wh values, is a measurement-transparency or reproducibility issue that could bias the energy axis; it is not a circularity, because it concerns a possible bias in measurement rather than an equivalence between the paper's inputs and its outputs. The threats-to-validity section itself acknowledges limitations in prompt uniformity, dataset scope, and generalizability, but none of these constitute a circular step. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims are measurements, not derivations. The main assumptions are the representativeness of the benchmark, the validity of the accuracy metric, the unbiasedness of the external judge, and the accuracy of the power measurement method.

assumptions (4)

domain assumption HumanEvalPack Python subset is a representative proxy for typical software development tasks.
The study's external validity rests on this. The authors themselves note in Section VI that the benchmark includes only basic functions and only Python.
domain assumption pass@1 at temperature 0.1 with top-p 0.95 is a valid accuracy measure for the four tasks.
The paper adopts the Codex pass@1 metric and follows prior HumanEval practice; accuracy claims depend on this metric being a meaningful proxy.
domain assumption GPT-4o-mini provides unbiased docstring evaluations.
Docstring accuracy is scored by using GPT-4o-mini to generate code from the docstring; if the judge has systematic bias, the docstring accuracy scores are biased.
domain assumption The 10 Hz GPU power sampling and idle-power subtraction (if performed) produce accurate energy measurements.
All energy results depend on the measurement method in Section IV-B; the paper does not explicitly state whether the reported idle powers were subtracted.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Language Models in Software Development Tasks: An Experimental Analysis of Energy and Accuracy." pith.science (2026). https://pith.science/paper/AGX6555O

@misc{pith2026241200329,
  author       = {Pith},
  title        = {Pith review of: Language Models in Software Development Tasks: An Experimental Analysis of Energy and Accuracy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AGX6555O}},
  note         = {Machine review of arXiv:2412.00329}
}

read the original abstract

The use of generative AI-based coding assistants like ChatGPT and Github Copilot is a reality in contemporary software development. Many of these tools are provided as remote APIs. Using third-party APIs raises data privacy and security concerns for client companies, which motivates the use of locally-deployed language models. In this study, we explore the trade-off between model accuracy and energy consumption, aiming to provide valuable insights to help developers make informed decisions when selecting a language model. We investigate the performance of 18 families of LLMs in typical software development tasks on two real-world infrastructures, a commodity GPU and a powerful AI-specific GPU. Given that deploying LLMs locally requires powerful infrastructure which might not be affordable for everyone, we consider both full-precision and quantized models. Our findings reveal that employing a big LLM with a higher energy budget does not always translate to significantly improved accuracy. Additionally, quantized versions of large models generally offer better efficiency and accuracy compared to full-precision versions of medium-sized ones. Apart from that, not a single model is suitable for all types of software development tasks.

Figures

Figures reproduced from arXiv: 2412.00329 by the authors.

**Figure 2.** Energy (Wh) consumed by the A100 GPU when performing each task. Model names are presented according to the [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Energy (Wh) consumed by the RTX 3070 GPU and the Intel Core i7 CPU when performing each task. The lower part [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Relation between accuracy and total energy for each task on GPU A100. Black dashed line indicates the Pareto frontier. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Spearman’s correlation matrix for all models across all [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Smaller Models, Unexpected Costs: Trade-offs in LLM Quantization for Automated Program Repair
cs.SE 2026-06 unverdicted novelty 6.0 of 10

Empirical evaluation of 13 quantization configurations on 6 LLMs for APR shows reduced memory (up to 85%) but increased inference time/energy, different repaired problem sets with little overlap, and 48% of configs st...
Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks
cs.SE 2025-06 conditional novelty 6.0 of 10

Across ten software engineering tasks and four LLMs, no prompting technique wins consistently; ES-KNN is best on many tasks, some techniques underperform the baseline, and USC is best for code QA and code generation.
Aggregating empirical evidence from data strategy studies: a case on model quantization
cs.SE 2025-05 conditional novelty 5.0 of 10

A synthesis of six studies shows model quantization costs a little accuracy but reliably cuts storage, latency, and energy use, and demonstrates a structured synthesis method on data-driven software engineering studies.
Greening AI-enabled Systems with Software Engineering: A Research Agenda for Environmentally Sustainable AI Practices
cs.SE 2025-06 accept novelty 4.0 of 10

A 29-participant workshop synthesized a research agenda for reducing AI's environmental footprint through software engineering, covering measurement, benchmarking, architecture, empirical methods, and education.

Reference graph

Works this paper leans on

56 extracted references · 36 canonical work pages · cited by 4 Pith papers

[1]

Generative AI for software practitioners,

C. Ebert and P. Louridas, “Generative AI for software practitioners,” IEEE Softw. , vol. 40, no. 4, pp. 30–38, 2023. [Online]. Available: https://doi.org/10.1109/MS.2023.3265877

arXiv 2023
[2]

AI at Work Is Here—Now Comes the Hard Part,

Microsoft Corporation, “AI at Work Is Here—Now Comes the Hard Part,” 2024, accessed: 2024-11-05. [Online]. Avail- able: https://www.microsoft.com/en-us/worklab/work-trend-index/ai-at- work-is-here-now-comes-the-hard-part

work page 2024
[3]

Developers get by with a little help from ai: Stack overflow knows code - assistant pulse survey results,

Stack Overflow, “Developers get by with a little help from ai: Stack overflow knows code - assistant pulse survey results,” 2024, accessed: 2024-11-08. [Online]. Available: https: //stackoverflow.blog/2024/05/29/developers-get-by-with-a-little-help- from-ai-stack-overflow-knows-code-assistant-pulse-survey-results/

work page 2024
[4]

Research: Quantifying github copilot’s impact in the enterprise with accenture,

GitHub, “Research: Quantifying github copilot’s impact in the enterprise with accenture,” 2024, accessed: 2024-11-08. [Online]. Available: https://github.blog/news-insights/research/research- quantifying-github-copilots-impact-in-the-enterprise-with-accenture/

work page 2024
[5]

Energy and policy consid- erations for deep learning in NLP,

E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy consid- erations for deep learning in NLP,” CoRR, vol. abs/1906.02243, 2019

arXiv 1906
[6]

Quantifying the carbon emissions of machine learning,

A. Lacoste, A. Luccioni, V . Schmidt, and T. Dandres, “Quantifying the carbon emissions of machine learning,” CoRR, vol. abs/1910.09700, 2019

arXiv 1910
[7]

Estimating the carbon footprint of bloom, a 176b parameter language model,

A. S. Luccioni, S. Viguier, and A. Ligozat, “Estimating the carbon footprint of bloom, a 176b parameter language model,” CoRR, vol. abs/2211.02001, 2022

arXiv 2022
[8]

The carbon footprint of machine learning training will plateau, then shrink,

D. A. Patterson, J. Gonzalez, U. H ¨olzle, Q. V . Le, C. Liang, L. Munguia, D. Rothchild, D. R. So, M. Texier, and J. Dean, “The carbon footprint of machine learning training will plateau, then shrink,” Computer, vol. 55, no. 7, pp. 18–28, 2022

work page 2022

Show all 56 references

[9]

A survey of large language models for code: Evolution, benchmarking, and future trends,

Z. Zheng, K. Ning, Y . Wang, J. Zhang, D. Zheng, M. Ye, and J. Chen, “A survey of large language models for code: Evolution, benchmarking, and future trends,” CoRR, vol. abs/2311.10372, 2023

2023 arXiv
[10]

A survey on large language models for code generation,

J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,” CoRR, vol. abs/2406.00515, 2024

2024 arXiv
[11]

Replication package,

Anonymous, “Replication package,” November 2024, accessed: 2024- 11-09. [Online]. Available: https://doi .org/10.5281/zenodo.14064267

2024 doi
[12]

Attention is all you need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long...

2017
[13]

History, development, and principles of large language models: an introductory survey,

Z. Wang, Z. Chu, T. V . Doan, S. Ni, M. Yang, and W. Zhang, “History, development, and principles of large language models: an introductory survey,” AI and Ethics , pp. 1–17, 2024

2024
[14]

A comprehensive evaluation of quantization strategies for large language models,

R. Jin, J. Du, W. Huang, W. Liu, J. Luan, B. Wang, and D. Xiong, “A comprehensive evaluation of quantization strategies for large language models,” in Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024...

2024
[15]

A survey of low-bit large language models: Basics, systems, and algorithms,

R. Gong, Y . Ding, Z. Wang, C. Lv, X. Zheng, J. Du, H. Qin, J. Guo, M. Magno, and X. Liu, “A survey of low-bit large language models: Basics, systems, and algorithms,” CoRR, vol. abs/2409.16694, 2024

2024
[16]

Benchmarking emerging deep learning quantization methods for energy efficiency,

S. Rajput and T. Sharma, “Benchmarking emerging deep learning quantization methods for energy efficiency,” in 21st IEEE International Conference on Software Architecture, ICSA 2024 - Companion, Hyder- abad, India, June 4-8, 2024 . IEEE, 2024, pp. 238–242

2024
[17]

A systematic literature review on the use of deep learning in software engineering research,

C. Watson, N. Cooper, D. Nader-Palacio, K. Moran, and D. Poshyvanyk, “A systematic literature review on the use of deep learning in software engineering research,” ACM Trans. Softw. Eng. Methodol., vol. 31, no. 2, pp. 32:1–32:58, 2022

2022
[18]

Deepbugs: A learning approach to name-based bug detection,

M. Pradel and K. Sen, “Deepbugs: A learning approach to name-based bug detection,” Proceedings of the ACM on Programming Languages , vol. 2, no. OOPSLA, pp. 1–25, 2018

2018
[19]

Evaluating the effectiveness of deep learning models for foundational program analysis tasks,

Q. Chen, C. Yu, R. Liu, C. Zhang, Y . Wang, K. Wang, T. Su, and L. Wang, “Evaluating the effectiveness of deep learning models for foundational program analysis tasks,” Proceedings of the ACM on Programming Languages, vol. 8, no. OOPSLA1, pp. 500–528, 2024

2024
[20]

Octopack: Instruction tuning code large language models,

N. Muennighoff, Q. Liu, A. Zebaze, Q. Zheng, B. Hui, T. Y . Zhuo, S. Singh, X. Tang, L. von Werra, and S. Longpre, “Octopack: Instruction tuning code large language models,” CoRR, vol. abs/2308.07124, 2023

2023 arXiv
[21]

Evaluating large language models trained on code,

M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards et al., “Evaluating large language models trained on code,” CoRR, vol. abs/2107.03374, 2021

2021 arXiv
[22]

Code llama: Open foundation models for code,

B. Rozi `ere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. Canton-Ferrer, A. Grattafiori, W. Xiong, A. D ´efossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scial...

2023 arXiv
[23]

Starcoder: may the source be with you!

R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., “Starcoder: may the source be with you!” Trans. Mach. Learn. Res. , vol. 2023, 2023

2023
[24]

Pydex: Repairing bugs in introductory python assign- ments using llms,

J. Zhang, J. P. Cambronero, S. Gulwani, V . Le, R. Piskac, G. Soares, and G. Verbruggen, “Pydex: Repairing bugs in introductory python assign- ments using llms,” Proc. ACM Program. Lang. , vol. 8, no. OOPSLA1, pp. 1100–1124, 2024

2024
[25]

Carbontracker: Tracking and predicting the carbon footprint of training deep learning models,

L. F. W. Anthony, B. Kanding, and R. Selvan, “Carbontracker: Tracking and predicting the carbon footprint of training deep learning models,” CoRR, vol. abs/2007.03051, 2020

2007 arXiv
[26]

Evaluating the energy efficiency of deep convolutional neural networks on cpus and gpus,

D. Li, X. Chen, M. Becchi, and Z. Zong, “Evaluating the energy efficiency of deep convolutional neural networks on cpus and gpus,” in 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computi...

2016
[27]

Uncov- ering energy-efficient practices in deep learning training: Preliminary steps towards green AI,

T. Yarally, L. Cruz, D. Feitosa, J. Sallou, and A. van Deursen, “Uncov- ering energy-efficient practices in deep learning training: Preliminary steps towards green AI,” in 2nd IEEE/ACM International Conference on AI Engineering - Software Engineering for AI, CAIN 2023, Melbour...

2023
[28]

The computational limits of deep learning,

N. C. Thompson, K. H. Greenewald, K. Lee, and G. F. Manso, “The computational limits of deep learning,” CoRR, vol. abs/2007.05558, 2020

2007 arXiv
[29]

Great power, great responsibility: Recommendations for reducing en- ergy for training language models,

J. McDonald, B. Li, N. C. Frey, D. Tiwari, V . Gadepally, and S. Samsi, “Great power, great responsibility: Recommendations for reducing en- ergy for training language models,” in Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States...

2022
[30]

Compute and energy consumption trends in deep learning inference,

R. Desislavov, F. Mart ´ınez-Plumed, and J. Hern ´andez-Orallo, “Compute and energy consumption trends in deep learning inference,” CoRR, vol. abs/2109.05472, 2021

2021 arXiv
[31]

From words to watts: Benchmarking the energy costs of large language model inference,

S. Samsi, D. Zhao, J. McDonald, B. Li, A. Michaleas, M. Jones, W. Bergeron, J. Kepner, D. Tiwari, and V . Gadepally, “From words to watts: Benchmarking the energy costs of large language model inference,” in IEEE High Performance Extreme Computing Conference, HPEC 2023, Boston...

2023
[32]

Power hungry processing: Watts driving the cost of AI deployment?

S. Luccioni, Y . Jernite, and E. Strubell, “Power hungry processing: Watts driving the cost of AI deployment?” in The 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT 2024, Rio de Janeiro, Brazil, June 3-6, 2024 . ACM, 2024, pp. 85–99

2024
[33]

So you want your private LLM at home? A survey and benchmark of methods for efficient gpts,

L. Tuggener, P. Sager, Y . Taoudi-Benchekroun, B. F. Grewe, and T. Stadelmann, “So you want your private LLM at home? A survey and benchmark of methods for efficient gpts,” in11th IEEE Swiss Conference on Data Science, SDS 2024, Zurich, Switzerland, May 30-31, 2024 . IEEE, 202...

2024
[34]

Hugging Face: The AI Community Building the Future,

Hugging Face, “Hugging Face: The AI Community Building the Future,” 2024, accessed: 2024-11-05. [Online]. Available: https://huggingface.co/

2024
[35]

Ollama Model Library,

Ollama, “Ollama Model Library,” 2024, accessed: 2024-11-05. [Online]. Available: https://ollama.com/library

2024
[36]

GGUF Format Documentation,

G. Gerganov, “GGUF Format Documentation,” 2023, accessed: 2024- 11-05. [Online]. Available: https://github .com/ggerganov/ggml/blob/ master/docs/gguf.md

2023
[37]

llama.cpp: A C++ Implementation of LLaMA Model,

Georgi Gerganov, “llama.cpp: A C++ Implementation of LLaMA Model,” 2023, accessed: 2024-11-05. [Online]. Available: https: //github.com/ggerganov/llama.cpp

2023
[38]

EvalPlus Leaderboard,

EvalPlus, “EvalPlus Leaderboard,” 2023, accessed: 2024-11-05. [Online]. Available: https://evalplus.github.io/leaderboard.html

2023
[39]

Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” in Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems ...

2023
[40]

How to use local llms,

LangChain, “How to use local llms,” 2024, accessed: 2024-07-

2024
[41]

Available: https://python .langchain.com/v0.2/docs/how to/local llms/#inference

[Online]. Available: https://python .langchain.com/v0.2/docs/how to/local llms/#inference
[42]

Green AI: a preliminary empirical study on energy consumption in DL models across different runtime infrastruc- tures,

N. Alizadeh and F. Castor, “Green AI: a preliminary empirical study on energy consumption in DL models across different runtime infrastruc- tures,” in Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI, CAIN 2024, Lisbon, P...

2024
[43]

Measuring and improving the energy efficiency of large language models inference,

M. F. Argerich and M. Pati ˜no-Mart´ınez, “Measuring and improving the energy efficiency of large language models inference,” IEEE Access , vol. 12, pp. 80 194–80 207, 2024

2024
[44]

Estimating the energy footprint of software systems: a primer,

F. Castor, “Estimating the energy footprint of software systems: a primer,” CoRR, vol. abs/2407.11611, 2024

2024 arXiv
[45]

NVIDIA Management Library (NVML),

NVIDIA Corporation, “NVIDIA Management Library (NVML),” 2024, https://developer.nvidia.com/management-library-nvml. Last ac- cessed August 20th, 2024

2024
[46]

pynvml: Python bindings for NVML,

PyPI Contributors, “pynvml: Python bindings for NVML,” 2024, accessed: 2024-11-06. [Online]. Available: https://pypi .org/project/ pynvml/

2024
[47]

RAPL in action: Experiences in using RAPL for power measurements,

K. N. Khan, M. Hirki, T. Niemi, J. K. Nurminen, and Z. Ou, “RAPL in action: Experiences in using RAPL for power measurements,” ACM Trans. Model. Perform. Evaluation Comput. Syst., vol. 3, no. 2, pp. 9:1– 9:26, 2018

2018
[48]

pyRAPL: Python library for measuring energy consumption with RAPL,

PyPI Contributors, “pyRAPL: Python library for measuring energy consumption with RAPL,” 2024, accessed: 2024-11-06. [Online]. Available: https://pypi.org/project/pyRAPL/

2024
[49]

Running Average Power Limit (RAPL) Energy Reporting,

Intel Corporation, “Running Average Power Limit (RAPL) Energy Reporting,” 2024, accessed: 2024-11-05. [Online]. Available: https://www.intel.com/content/www/us/en/developer/articles/technical/ software-security-guidance/advisory-guidance/running-average-power- limit-energy-rep...

2024
[50]

NVIDIA System Management Interface (nvidia- smi),

NVIDIA Corporation, “NVIDIA System Management Interface (nvidia- smi),” 2024, accessed: 2024-11-06. [Online]. Available: https:// developer.nvidia.com/system-management-interface

2024
[51]

Starcoder 2 and the stack v2: The next generation,

A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Wei et al. , “Starcoder 2 and the stack v2: The next generation,” CoRR, vol. abs/2402.19173, 2024

2024 arXiv
[52]

Mercury: An efficiency benchmark for LLM code synthesis,

M. Du, A. T. Luu, B. Ji, and S. Ng, “Mercury: An efficiency benchmark for LLM code synthesis,” CoRR, vol. abs/2402.07844, 2024

2024 arXiv
[53]

Magicoder: Em- powering code generation with oss-instruct,

Y . Wei, Z. Wang, J. Liu, Y . Ding, and L. Zhang, “Magicoder: Em- powering code generation with oss-instruct,” in Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024

2024
[54]

Alpaca: A strong, replicable instruction- following model,

R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Alpaca: A strong, replicable instruction- following model,” Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html , vol. 3, no. 6,...

2023
[55]

Large language models as test case generators: Performance evaluation and enhancement,

K. Li and Y . Yuan, “Large language models as test case generators: Performance evaluation and enhancement,” CoRR, vol. abs/2404.13340, 2024

2024 arXiv
[56]

Coverage.py: The code coverage tool for Python

N. Batchelder, “Coverage.py: The code coverage tool for Python.” [Online]. Available: https://github .com/nedbat/coveragepy

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Generative AI for software practitioners,

C. Ebert and P. Louridas, “Generative AI for software practitioners,” IEEE Softw. , vol. 40, no. 4, pp. 30–38, 2023. [Online]. Available: https://doi.org/10.1109/MS.2023.3265877

arXiv 2023

[2] [2]

AI at Work Is Here—Now Comes the Hard Part,

Microsoft Corporation, “AI at Work Is Here—Now Comes the Hard Part,” 2024, accessed: 2024-11-05. [Online]. Avail- able: https://www.microsoft.com/en-us/worklab/work-trend-index/ai-at- work-is-here-now-comes-the-hard-part

work page 2024

[3] [3]

Developers get by with a little help from ai: Stack overflow knows code - assistant pulse survey results,

Stack Overflow, “Developers get by with a little help from ai: Stack overflow knows code - assistant pulse survey results,” 2024, accessed: 2024-11-08. [Online]. Available: https: //stackoverflow.blog/2024/05/29/developers-get-by-with-a-little-help- from-ai-stack-overflow-knows-code-assistant-pulse-survey-results/

work page 2024

[4] [4]

Research: Quantifying github copilot’s impact in the enterprise with accenture,

GitHub, “Research: Quantifying github copilot’s impact in the enterprise with accenture,” 2024, accessed: 2024-11-08. [Online]. Available: https://github.blog/news-insights/research/research- quantifying-github-copilots-impact-in-the-enterprise-with-accenture/

work page 2024

[5] [5]

Energy and policy consid- erations for deep learning in NLP,

E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy consid- erations for deep learning in NLP,” CoRR, vol. abs/1906.02243, 2019

arXiv 1906

[6] [6]

Quantifying the carbon emissions of machine learning,

A. Lacoste, A. Luccioni, V . Schmidt, and T. Dandres, “Quantifying the carbon emissions of machine learning,” CoRR, vol. abs/1910.09700, 2019

arXiv 1910

[7] [7]

Estimating the carbon footprint of bloom, a 176b parameter language model,

A. S. Luccioni, S. Viguier, and A. Ligozat, “Estimating the carbon footprint of bloom, a 176b parameter language model,” CoRR, vol. abs/2211.02001, 2022

arXiv 2022

[8] [8]

The carbon footprint of machine learning training will plateau, then shrink,

D. A. Patterson, J. Gonzalez, U. H ¨olzle, Q. V . Le, C. Liang, L. Munguia, D. Rothchild, D. R. So, M. Texier, and J. Dean, “The carbon footprint of machine learning training will plateau, then shrink,” Computer, vol. 55, no. 7, pp. 18–28, 2022

work page 2022