Pith. sign in

REVIEW 4 major objections 5 minor 41 references

AI-Powered, But Power-Hungry? Energy Efficiency of LLM-Generated Code

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-generated code is energy-competitive with efficient human code in Python but 34–134% more energy-intensive in C++.

desk verdict Legitimate extension of LLM code energy benchmarking to Java/C++, but the C++ result needs stronger statistical grounding before it should be cited as fact. read the letter →

arxiv 2502.02412 v1 pith:R7BKBOAM submitted 2025-02-04 cs.SE

Lola Solovyeva , Sophie Weidmann , Fernando Castor This is my paper · ORCID
classification cs.SE
keywords energyefficiencyLLM-generatedcodegenerationLeetbenchmarksoftwareconsumptionGitHubCopilotGPT-4oOpenAIo1-mini
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to establish whether code written by three frontier LLMs—GitHub Copilot, GPT-4o, and OpenAI o1-mini—can match the energy efficiency of efficient human-written code, and whether the answer depends on language and machine. It benchmarks 53 LeetCode 'hard' problems in Python, Java, and C++ on a Mac and a PC, running each LLM solution and each top-voted human solution on the same tests and reporting energy and runtime relative to the human baseline. The central result is language-dependent: LLM output is roughly energy-competitive in Python (about 91–104% of baseline), somewhat worse in Java (111–148%), and substantially worse in C++ (134–234%). The authors conclude that LLM-generated C++ should not be assumed efficient enough for energy-critical software without review, while in Python it can sometimes beat a good human solution.

What carries the argument

The load-bearing instrument is a controlled comparative benchmark: 53 LeetCode hard problems, each with a top-voted human solution in Python, Java, and C++ serving as the efficiency baseline, and 477 LLM-generated candidate solutions from three models prompted with the same problem statement and type signature. The same test workloads—LeetCode examples plus a few maximal and minimal edge cases, about seven tests per problem—are run on both human and generated code, repeated 10 times (13 for Java with the first 3 discarded for JVM warm-up), with energy sampled at 100 Hz through powermetrics on macOS and perf with Intel RAPL on Ubuntu. Reporting energy and runtime as percentages of the human baseline is what turns raw measurements into the paper's language-level ratios.

What would settle it

Re-run the same 53 problems with test inputs 10 to 100 times larger than LeetCode's examples, keeping the same models, prompts, and human baselines, and recompute the C++ energy ratios. If the 134–234% of baseline shrinks toward parity, the paper's C++ penalty is an artifact of light workloads; if it widens or stays, the conclusion is confirmed.

Watch

Extended reading notes

Core claim

The paper claims that, on 53 hard LeetCode tasks, the energy cost of a correct LLM-generated solution relative to a top-voted human solution is governed mainly by programming language. Pass@1 accuracy is highest in Python (up to 66% for o1-mini), lower in Java (up to 64%), and lowest in C++ (up to 51%). Energy follows the same ordering: generated Python runs at 91–104% of baseline energy, generated Java at 111–148%, and generated C++ at 134–234%, with the largest C++ penalty coming from GitHub Copilot on Ubuntu. Across models, o1-mini improves correctness, especially on search, sorting, bit manipulation, math, graph, and game-theory tasks, but it consumes more energy than GPT-4o and GitHub Copilot; Copilot is the most efficient model for Python and Java, and GPT-4o for C++. The paper also claims the relative efficiency of generated code is stable across the two platforms (Spearman correlations around 0.8–0.9), so the language-level pattern does not appear to be an artifact of one machine, whereas human solutions correlate less strongly across platforms for C++.

Load-bearing premise

The load-bearing assumption, stated in Section III-C, is that LeetCode's example inputs plus a few extra edge cases—about seven tests per problem—effectively stand in for real-world usage, so the measured energy ratios carry over to actual software.

Editorial extensions

If this is right

  • In Python, teams can treat LLM-generated solutions as approximately energy-neutral relative to efficient human code; the measured gap is at most a few percent and sometimes negative.
  • In C++, LLM-generated solutions should be expected to burn 34–134% more energy than a good human solution, and to pass at lower rates, so energy-critical C++ code needs human review before deployment.
  • OpenAI o1-mini is the most accurate of the three models on these hard problems, but its accuracy comes with higher energy consumption than GPT-4o and GitHub Copilot.
  • Energy-efficiency rankings of generated code persist across macOS and Ubuntu, so a solution that is efficient or inefficient on one platform is likely to be the same on the other.
  • Sorting, graph, greedy, math, and recursion problems are the weak spots: generated solutions for these categories combine lower pass@1 accuracy with higher-than-baseline energy consumption.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: because the tests are mostly small LeetCode examples, the measured C++ penalty may understate what happens on large production-sized inputs, where algorithmic overhead in generated code tends to matter more.
  • Beyond the paper: the paper compiles all C++ with the default -O0 flag; re-running with -O2 or -O3 would show how much of the 34–134% penalty survives normal production build settings, since compiler optimization can change energy profiles substantially.
  • Beyond the paper: the strong cross-platform correlation suggests the same benchmark could be run on cloud instances as a cheap pre-deployment screen, flagging LLM-generated functions whose energy footprint is far from baseline before they reach production.
  • Beyond the paper: the o1-mini accuracy-versus-energy tradeoff raises a concrete cost question the paper does not answer: whether one-time generation energy plus repeated execution energy favors the more accurate reasoning model or the cheaper older models at a given usage volume.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper compares the energy consumption and runtime of LLM-generated code from GitHub Copilot, GPT-4o, and OpenAI o1-mini against human-written LeetCode solutions for 53 hard problems in Python, Java, and C++, on macOS and Ubuntu. The authors report pass@1 accuracy, energy and time relative to the baseline, per-category results, and cross-platform correlations. The central finding is that generated Python code is roughly energy-competitive with the baseline, Java is moderately worse (111-148% of baseline energy), and C++ is substantially worse (134-234%), while C++ also has much lower pass@1 rates. The paper concludes that LLMs perform best in Python, that o1-mini improves correctness but often consumes more energy, and that generated solutions are machine-agnostic across the two studied platforms.

Significance. If the central results are robust, this would be a valuable contribution as one of the first multi-language studies of LLM-generated code energy efficiency, using a non-trivial benchmark and two platforms. Strengths include the public replication package, the use of hard LeetCode problems, and comparison against efficient human-written baselines. The study is observational and does not reduce to a fitted prediction, so circularity is not a concern. However, the main C++ energy claim currently rests on aggregate point estimates without confidence intervals or effect sizes, and the workloads are light, so the significance of the empirical claims is presently qualified by statistical and external-validity gaps.

major comments (4)
  1. [Section IV-A, Table II] The paper's central claim that C++ solutions are 'significantly more energy-intensive' rests on aggregate mean ratios (134-234%) computed from only 17-27 correct C++ solutions per model (pass@1 32-51%, Table III), yet the manuscript reports no per-problem energy ratios, confidence intervals, or effect sizes. The p-value threshold reported in Section IV only establishes that some difference from baseline exists, not the magnitude or stability of the ratio; with one generated solution per problem, a small number of outlier solutions could move the mean from the o1/macOS figure of 139% to the Copilot/Ubuntu figure of 232%. Please report the distribution of per-problem ratios (e.g., boxplots or confidence intervals), the promised Hedges' g values from Section III-F, and a sensitivity analysis demonstrating that the result is not driven by a few problems.
  2. [Section III-C] Section III-C assumes that roughly seven test cases per problem, mostly LeetCode example inputs, 'effectively simulate the average real-world usage of these problems.' No evidence is provided for this assumption, and the energy ratios (especially the C++ penalty) are measured on light workloads. Since a correct-but-asymptotically-poor solution can pass all seven tests while being unrepresentative of energy behavior on larger inputs, the external validity of the language-level conclusions is threatened. Please validate the workload choice, for example by including input-size sweeps or comparing against heavier test suites, and quantify how sensitive the Table II ratios are to workload size.
  3. [Section III-E] All C++ code is compiled with the default -O0 flag. Energy comparisons at -O0 are not representative of typical C++ deployment, where optimization is normally enabled, and may either mask or exaggerate the gap between generated and human-written solutions. The authors should either report results with a realistic optimization level (e.g., -O2) or explicitly justify why -O0 is appropriate for the research question.
  4. [Section IV-A, Table III] Energy ratios are computed only for solutions that pass the test suite; for C++ the pass@1 accuracy is 32-51%, so the ratios in Table II are based on a non-random subset of problems. This selection can confound model comparisons with problem difficulty (for example, if a model only passes the easiest C++ problems), yet no analysis addresses this missingness. Please report whether the conclusions are robust on the subset of problems solved by all models, or otherwise model the selection effect.
minor comments (5)
  1. [Section II] The related-work discussion refers to reference [7] as 'Varziotis et al.' in one place and 'Vartziotis et al.' in another; use a consistent spelling.
  2. [Section III-F] Hedges' g is listed as part of the planned analysis, but no effect-size estimates appear in the results section; either report them or remove the promise.
  3. [Section IV-A] The statement that 'the relationship between energy consumption and execution time remains linear across all programming languages, systems, and machines' is asserted without showing the supporting data or correlation coefficients.
  4. [Section IV-C] The claim that o1-mini's accuracy improvements stem from its chain-of-thought mechanism is speculative and not tested; the text should present this as a hypothesis rather than an explanation.
  5. [Table III, Section IV-B] Several categories contain very small numbers of problems (Game Theory 1, Simulation 1, Two Pointers 2, Divide and Conquer 3), yet Figure 1 and Section IV-B draw categorical conclusions from them; add explicit caveats or confidence information for these groups.

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the study measures LLM-generated code energy against an external LeetCode human baseline with no fitted parameters or self-referential predictions.

full rationale

The paper's central claims, including the per-language energy ratios in Table II, pass@1 accuracy, and cross-platform correlations, are observational measurements of independently generated code executed against an external human-written baseline drawn from LeetCode. No parameter is fitted to the outcome, and no prediction is derived from a model calibrated on the same data; the baseline is not defined in terms of LLM output, and the LLM output is not defined in terms of the baseline. The only self-citation, reference [38] by co-author Castor, is used to justify discarding the first three Java runs for JVM warm-up; that is a standard measurement practice and does not supply any load-bearing premise for the energy-efficiency findings. The weak workload assumption in Section III-C is a generalizability threat, not a circularity, and the absence of confidence intervals for the C++ ratios is a robustness concern, not evidence that the result reduces to its inputs. The derivation chain is therefore self-contained with respect to circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The study contains no fitted parameters or invented entities; its claims rest on measurement design assumptions. The central comparisons depend on baseline validity, workload representativeness, CPU-only energy measurement, and the -O0 compiler setting.

assumptions (5)
  • domain assumption LeetCode upvoted human solutions are efficient and serve as a valid baseline.
    Section III-A states the 159 human solutions were chosen by user upvotes and their authors minimized time and space complexity, but no independent energy-efficiency validation is reported.
  • domain assumption LeetCode example inputs plus custom edge cases simulate average real-world usage.
    Section III-C explicitly says the mostly example-based tests "effectively simulate the average real-world usage," with about 7 test cases per problem; this generalization is untested.
  • domain assumption CPU energy consumption is an adequate proxy for code energy efficiency.
    Section III-B defines the dependent variable as energy consumption "focusing on CPU energy"; total system energy, DRAM, and GPU contributions are not reported.
  • domain assumption Compiling all code with -O0 provides a fair basis for comparing LLM and human solutions.
    Section III-E sets default compiler optimization flag -O0 for all executions; this may disadvantage C++ code, where optimization is normally expected, and affects the energy ratios.
  • domain assumption The selected LeetCode "hard" problems were not directly seen during LLM training.
    Section I states the analyzed LLMs "were not directly trained on them," an unverifiable claim about proprietary training data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AI-Powered, But Power-Hungry? Energy Efficiency of LLM-Generated Code." pith.science (2026). https://pith.science/paper/R7BKBOAM

@misc{pith2026250202412,
  author       = {Pith},
  title        = {Pith review of: AI-Powered, But Power-Hungry? Energy Efficiency of LLM-Generated Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R7BKBOAM}},
  note         = {Machine review of arXiv:2502.02412}
}
read the original abstract

Large language models (LLMs) are used in software development to assist in various tasks, e.g., code generation and code completion, but empirical evaluations of the quality of the results produced by these models focus on correctness and ignore other relevant aspects, such as their performance and energy efficiency. Studying the performance of LLM-produced programs is essential to understand how well LLMs can support the construction of performance- and energy-critical software, such as operating systems, servers, and mobile applications. This paper presents the first study analyzing the energy efficiency and performance of LLM-generated code for three programming languages Python, Java, and C++, on two platforms, a Mac and a PC, leveraging three frontier LLMs, Github Copilot, GPT-4o, and the recently-released OpenAI o1-mini, and targeting ``hard'' programming problems from LeetCode. Our results show that the models are much more successful in generating Python and Java than C++ code.

Figures

Figures reproduced from arXiv: 2502.02412 by the authors.

Figure 1
Figure 1. Each subplot illustrates the average energy consumption required to complete programming problems within each category (x-axis) for a specific [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the Spearman correlation between the energy consumption results for solutions generated by each model across two platforms. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Example solutions for the First Missing Positive problem. The left solution was generated by OpenAI’s o1-mini model, while the right solution represents the highest-rated human implementation from LeetCode. The difference between the newer OpenAI o1-mini version and the older GPT-4o version can also be observed by [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 24 canonical work pages

  1. [1]

    Impacts of software and its engineering on the carbon footprint of ict,

    E. Kern, M. Dick, S. Naumann, and T. Hiller, “Impacts of software and its engineering on the carbon footprint of ict,” Environmental Impact Assessment Review , vol. 52, pp. 53–61, 2015, information technology and renewable energy - Modelling, simulation, decision support and environmental assessment. [Online]. Available: https://www.sciencedirect.com/scie...

    work page 2015

  2. [2]

    Is software “green

    E. Capra, C. Francalanci, and S. A. Slaughter, “Is software “green”? application development environments and energy efficiency in open source applications,” Information and Software Technology , vol. 54, no. 1, pp. 60–71, 2012. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S0950584911001777

    work page 2012

  3. [3]

    An empirical study of practitioners’ perspectives on green software engineering,

    I. Manotas, C. Bird, R. Zhang, D. Shepherd, C. Jaspan, C. Sadowski, L. Pollock, and J. Clause, “An empirical study of practitioners’ perspectives on green software engineering,” in Proceedings of the 38th International Conference on Software Engineering , ser. ICSE ’16. New York, NY , USA: Association for Computing Machinery, 2016, p. 237–248. [Online]. A...

    arXiv 2016

  4. [4]

    A comprehensive review of green computing: Past, present, and future research,

    S. G. Paul, A. Saha, M. S. Arefin, T. Bhuiyan, A. A. Biswas, A. W. Reza, N. M. Alotaibi, S. A. Alyami, and M. A. Moni, “A comprehensive review of green computing: Past, present, and future research,” IEEE Access, vol. 11, pp. 87 445–87 494, 2023

    work page 2023

  5. [5]

    Future data center energy- conservation and emission-reduction technologies in the context of smart and low-carbon city construction,

    H. Zhu, D. Zhang, H. H. Goh, S. Wang, T. Ahmad, D. Mao, T. Liu, H. Zhao, and T. Wu, “Future data center energy- conservation and emission-reduction technologies in the context of smart and low-carbon city construction,” Sustainable Cities and Society, vol. 89, p. 104322, 2023. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S221067...

    work page 2023

  6. [6]

    Ict sector electricity consumption and greenhouse gas emissions – 2020 outcome,

    J. Malmodin, N. L ¨ovehagen, P. Bergmark, and D. Lund ´en, “Ict sector electricity consumption and greenhouse gas emissions – 2020 outcome,” Telecommunications Policy, vol. 48, no. 3, p. 102701, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0308596123002124

    work page 2020

  7. [7]

    Learn to code sustainably: An empirical study on llm-based green code generation,

    T. Vartziotis, I. Dellatolas, G. Dasoulas, M. Schmidt, F. Schneider, T. Hoffmann, S. Kotsopoulos, and M. Keckeisen, “Learn to code sustainably: An empirical study on llm-based green code generation,” arXiv preprint , vol. arXiv:2403.03344v1, 2024. [Online]. Available: https://arxiv.org/abs/2403.03344v1

    arXiv 2024

  8. [8]

    Green artificial intelligence initiatives: Potentials and challenges,

    Y . I. Alzoubi and A. Mishra, “Green artificial intelligence initiatives: Potentials and challenges,” Journal of Cleaner Production , vol. 468, p. 143090, 2024. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S0959652624025393

    work page 2024

Show all 41 references
  1. [9]

    Ai-powered green cloud and data center,

    J. Yang, W. Xiao, C. Jiang, M. S. Hossain, G. Muhammad, and S. U. Amin, “Ai-powered green cloud and data center,” IEEE Access, vol. 7, pp. 4195–4203, 2019

    2019

  2. [10]

    Trends in ai inference energy consumption: Beyond the performance-vs- parameter laws of deep learning,

    R. Desislavov, F. Mart ´ınez-Plumed, and J. Hern ´andez-Orallo, “Trends in ai inference energy consumption: Beyond the performance-vs- parameter laws of deep learning,” Sustainable Computing: Informatics and Systems , vol. 38, p. 100857, 2023. [Online]. Available: https://www....

    2023

  3. [11]

    Estimating the carbon footprint of bloom, a 176b parameter language model,

    A. S. Luccioni, S. Viguier, and A.-L. Ligozat, “Estimating the carbon footprint of bloom, a 176b parameter language model,” Journal of Machine Learning Research, vol. 24, no. 253, pp. 1–15, 2023. [Online]. Available: http://jmlr.org/papers/v24/23-0069.html

    2023

  4. [12]

    Reducing carbon footprint in ai: A framework for sustainable training of large language models,

    S. Iftikhar and S. Davy, “Reducing carbon footprint in ai: A framework for sustainable training of large language models,” in Proceedings of the Future Technologies Conference (FTC) 2024, Volume 1 , K. Arai, Ed. Cham: Springer Nature Switzerland, 2024, pp. 325–336

    2024

  5. [13]

    A review of green artificial intelligence: Towards a more sustainable future,

    V . Bol ´on-Canedo, L. Mor ´an-Fern´andez, B. Cancela, and A. Alonso- Betanzos, “A review of green artificial intelligence: Towards a more sustainable future,” Neurocomputing, vol. 599, p. 128096, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0...

    2024

  6. [14]

    Github repository: Energyefficiencyllmcode,

    “Github repository: Energyefficiencyllmcode,” 2024, accessed: Dec. 6, 2024. [Online]. Available: https://github.com/energyefficienctcode/ EnergyEfficiencyLLMCode

    2024

  7. [15]

    A review on code generation with llms: Application and evaluation,

    J. Wang and Y . Chen, “A review on code generation with llms: Application and evaluation,” in 2023 IEEE International Conference on Medical Artificial Intelligence (MedAI) , 2023, pp. 284–289

    2023

  8. [16]

    Evaluating the quality of llm-generated explanations for logical errors in cs1 student programs,

    R. Balse, V . Kumar, P. Prasad, and J. M. Warriem, “Evaluating the quality of llm-generated explanations for logical errors in cs1 student programs,” in Proceedings of the 16th Annual ACM India Compute Conference , ser. COMPUTE ’23. New York, NY , USA: Association for Computin...

    2023

  9. [17]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Pond ´e, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F...

    2021 arXiv

  10. [18]

    On sample-efficient code generation,

    H. Han, Y . J. Kim, B. Kim, Y . Lee, K. Lee, K. Lee, M. Lee, K. Bae, and S.-w. Hwang, “On sample-efficient code generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, M. Wang and I. Zitouni, Eds. Singapore: Assoc...

    2023

  11. [19]

    Enhancing software development efficiency through ai- powered code generation,

    N. Sherje, “Enhancing software development efficiency through ai- powered code generation,” Research Journal of Computer Systems and Engineering, vol. 5, no. 1, p. 01–12, Jul. 2024. [Online]. Available: https://technicaljournals.org/RJCSE/index.php/journal/article/view/90

    2024

  12. [20]

    Assessing the quality of github copilot’s code generation,

    B. Yetistiren, I. Ozsoy, and E. Tuzun, “Assessing the quality of github copilot’s code generation,” in Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering, ser. PROMISE 2022. New York, NY , USA: Association for Compu...

    2022

  13. [21]

    A performance study of llm-generated code on leetcode,

    T. Coignion, C. Quinton, and R. Rouvoy, “A performance study of llm-generated code on leetcode,” in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, ser. EASE ’24. New York, NY , USA: Association for Computing Machinery, 20...

    2024

  14. [22]

    Ranking programming languages by energy efficiency,

    R. Pereira, M. Couto, F. Ribeiro, R. Rua, J. Cunha, J. P. Fernandes, and J. Saraiva, “Ranking programming languages by energy efficiency,” Science of Computer Programming , vol. 205, p. 102609, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0167...

    2021

  15. [23]

    Mercury: A code efficiency benchmark for code large language models,

    M. Du, A. T. Luu, B. Ji, Q. Liu, and S.-K. Ng, “Mercury: A code efficiency benchmark for code large language models,” arXiv preprint , vol. arXiv:2402.07844v4, 2024. [Online]. Available: https://arxiv.org/abs/2402.07844v4

    2024 arXiv

  16. [24]

    Effibench: Benchmarking the efficiency of automatically generated code,

    D. Huang, Y . Qing, W. Shang, H. Cui, and J. M. Zhang, “Effibench: Benchmarking the efficiency of automatically generated code,” arXiv preprint , vol. arXiv:2402.02037v4, 2024. [Online]. Available: https://arxiv.org/abs/2402.02037v4

    2024 arXiv

  17. [25]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ...

  18. [26]

    Measuring coding challenge competence with apps,

    D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. X. Song, and J. Steinhardt, “Measuring coding challenge competence with apps,” ArXiv, vol. abs/2105.09938,

    arXiv

  19. [27]

    ReCode: Robustness evaluation of code generation models,

    S. Wang, Z. Li, H. Qian, C. Yang, Z. Wang, M. Shang, V . Kumar, S. Tan, B. Ray, P. Bhatia, R. Nallapati, M. K. Ramanathan, D. Roth, and B. Xiang, “ReCode: Robustness evaluation of code generation models,” in Proceedings of the 61st Annual Meeting of the Association for Computa...

    2023

  20. [28]

    Effi-code: Unleashing code efficiency in language models,

    D. Huang, G. Zeng, J. Dai, M. Luo, H. Weng, Y . Qing, H. Cui, Z. Guo, and J. M. Zhang, “Effi-code: Unleashing code efficiency in language models,” ArXiv, vol. abs/2410.10209, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:273345361

    2024 arXiv

  21. [29]

    Evaluating language models for efficient code generation,

    J. Liu, S. Xie, J. Wang, Y . Wei, Y . Ding, and L. Zhang, “Evaluating language models for efficient code generation,” 2024. [Online]. Available: https://arxiv.org/abs/2408.06450

    2024 arXiv

  22. [30]

    Evaluating large language models in class-level code generation,

    X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y . Chen, J. Feng, C. Sha, X. Peng, and Y . Lou, “Evaluating large language models in class-level code generation,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , ser. ICSE ’24. New York, NY , USA...

    2024

  23. [31]

    Piloting copilot and codex: Hot temperature, cold prompts, or black magic?

    J.-B. D ¨oderlein, M. Acher, D. E. Khelladi, and B. Combemale, “Piloting copilot and codex: Hot temperature, cold prompts, or black magic?” ArXiv, vol. abs/2210.14699, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:253117147

    2022 arXiv

  24. [32]

    An empirical evaluation of github copilot’s code suggestions,

    N. Nguyen and S. Nadi, “An empirical evaluation of github copilot’s code suggestions,” in 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR) , 2022, pp. 1–5

    2022

  25. [33]

    Generation probabilities are not enough: Uncertainty highlighting in ai code completions,

    H. Vasconcelos, G. Bansal, A. Fourney, Q. V . Liao, and J. W. Vaughan, “Generation probabilities are not enough: Uncertainty highlighting in ai code completions,” ACM Trans. Comput.-Hum. Interact. , Oct. 2024, just Accepted. [Online]. Available: https://doi.org/10.1145/3702320

    2024 doi

  26. [34]

    van Solingen (Revision), V

    R. van Solingen (Revision), V . Basili (Original article, 1994 ed.), G. Caldiera (Original article, 1994 ed.), and H. D. Rombach (Original article, 1994 ed.), Goal Question Metric (GQM) Approach . John Wiley & Sons, Ltd, 2002. [Online]. Available: https://onlinelibrary. wiley....

    1994 doi

  27. [35]

    On evaluating the efficiency of source code generated by llms,

    C. Niu, T. Zhang, C. Li, B. Luo, and V . Ng, “On evaluating the efficiency of source code generated by llms,” in Proceedings of the AI Foundation Models and Software Engineering (FORGE ’24). Lisbon, Portugal: ACM, 2024, p. 5. [Online]. Available: https://doi.org/10.1145/365010...

    2024

  28. [36]

    Intel® performance counter monitor - a better way to measure cpu

    R. D. Thomas Willhalm, “Intel® performance counter monitor - a better way to measure cpu.” [Online]. Available: https://www.intel.com/content/www/us/en/developer/articles/ tool/performance-counter-monitor.html

  29. [37]

    RAPL in action: Experiences in using RAPL for power measurements,

    K. N. Khan, M. Hirki, T. Niemi, J. K. Nurminen, and Z. Ou, “RAPL in action: Experiences in using RAPL for power measurements,” ACM Trans. Model. Perform. Evaluation Comput. Syst., vol. 3, no. 2, pp. 9:1– 9:26, 2018

    2018

  30. [38]

    Understanding energy behaviors of thread management constructs,

    G. Pinto, F. Castor, and Y . D. Liu, “Understanding energy behaviors of thread management constructs,” in Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages & Applications , ser. OOPSLA ’14. New York, NY , USA: Association...

    2014

  31. [39]

    Energy efficiency analysis of compiler optimizations on the spec cpu 2017 benchmark suite,

    N. Schmitt, J. Bucek, K.-D. Lange, and S. Kounev, “Energy efficiency analysis of compiler optimizations on the spec cpu 2017 benchmark suite,” in Companion of the ACM/SPEC International Conference on Performance Engineering , ser. ICPE ’20. New York, NY , USA: Association for ...

    2017

  32. [2021]

    Available: https://api.semanticscholar.org/CorpusID: 234790100

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 234790100

  33. [2024]

    Available: https://doi.org/10.1145/3597503.3623316

    [Online]. Available: https://doi.org/10.1145/3597503.3623316

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.