Pith. sign in

REVIEW 4 major objections 7 minor 2 cited by

Is Quantization a Deal-breaker? Empirical Insights from Large Code Models

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read 4-bit quantization keeps code quality intact in large code models.

desk verdict Useful first look at how 4-bit AWQ quantization affects code quality, but the headline numbers are over-aggregated and the conclusion outruns the statistics. read the letter →

arxiv 2507.09665 v1 pith:IPJJP4ZQ submitted 2025-07-13 cs.SE

Saima Afrin , Bowen Xu , Antonio Mastropaolo This is my paper
classification cs.SE
keywords quantizationlargecodemodelsgenerationqualityAWQstaticanalysismaintainabilityempiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that quantizing large code models down to 4-bit precision with Activation-aware Weight Quantization does not silently degrade the code they produce. The authors compare full-precision and AWQ-quantized versions of CodeLlama and DeepSeek-Coder on Java and Python benchmarks, measuring not only whether tests pass but also maintainability, reliability, complexity, security hotspots, and code smells with static analysis tools. Across most metrics the quantized versions look statistically indistinguishable from the full-precision ones, and on one Python benchmark the aggregate issue count is about 6 percent lower for quantized models. The conclusion, stated in the paper's own words, is that quantization is not a deal-breaker: it preserves functional correctness and most qualitative code attributes, with readability showing the only consistent signs of mild degradation.

What carries the argument

The load-bearing component is Activation-aware Weight Quantization (AWQ), a post-training quantization method that decides which weights to keep at high precision by looking at activation magnitudes rather than weights alone, then applies per-channel scaling to cut the model to 4-bit with minimal quantization error. AWQ's selective preservation of salient weights is what the paper credits for compressed models behaving like their full-precision selves. The supporting machinery is the paired evaluation: each generation task is run with full-precision and quantized models, the outputs are scored by SonarCloud, Pylint, Flake8, and PMD, and the per-task metric distributions are compared with a Wilcoxon signed-rank test under Holm correction, with Cliff's delta measuring effect sizes.

What would settle it

Rerunning the same comparison while normalizing issue counts by lines of code (or by function) would settle whether the quality preservation is real; if 4-bit models show more issues per line on a larger set of tasks, the claim that quantization preserves code quality would be contradicted by the paper's own aggregate evidence.

Watch

Extended reading notes

Core claim

The paper's central discovery is that 4-bit AWQ quantization preserves the quality profile of code generated by large code models, not just their pass rates. Across two model families, two benchmarks, and two languages, quantized models produced code whose reliability, maintainability, cyclomatic and cognitive complexity, security-hotspot counts, and code-smell counts were generally comparable to full-precision output; statistical tests found significant differences only for Pylint Warning and PMD Code Style metrics, with effect sizes ranging from negligible to small. Quantized models even showed slightly lower total SonarCloud issue counts than full-precision models on McEval-Python (2,892 versus 3,084) and a marginal reduction on Java. A manual review of 50 sampled predictions per language found 82 percent rated acceptable or good, while readability was the weakest dimension, with lower inter-rater agreement and more poor ratings. The paper interprets these results as contradicting its own starting hypothesis that information loss from quantization would degrade code quality.

Load-bearing premise

The headline comparison assumes that different kinds of static-analysis findings, such as bugs, code smells, complexity scores, and security hotspots, can be added into one total-issues number and compared fairly even though quantized models sometimes generate longer code than full-precision models.

Editorial extensions

If this is right

  • Teams can deploy 4-bit AWQ-quantized code models for code generation with reduced memory and energy use without an anticipated drop in generated-code quality metrics.
  • Pass@1 differences between full-precision and quantized models are small enough that functional correctness is not the deciding factor in choosing a compressed model.
  • Readability is the one quality dimension that quantized models tend to hurt, so adoption should pair quantization with identifier and naming review or post-processing.
  • Larger models exhibit fewer quality issues whether quantized or not, and quantization's negative effects on maintainability mostly show up in the smaller 7B and 13B variants.
  • For most quality metrics, the absence of statistically significant differences means the practical effect of 4-bit quantization is likely small across the configurations tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A likely explanation for the apparent 6 percent improvement on McEval-Python is that quantized DeepSeek models generated shorter code on that benchmark, and fewer lines naturally yield fewer flagged issues; normalizing by lines of code could turn the improvement into a wash.
  • Because AWQ explicitly protects activation-salient weights, its quality preservation may be specific to AWQ; comparing against weight-only methods such as GPTQ on the same quality metrics would show whether the result generalizes across quantization techniques.
  • The readability drop may worsen as tasks become more open-ended or as model output is used directly in production codebases, since identifier choice and structure matter more in larger programs; a developer study on real tasks would test this.
  • The same methodology could be applied to other LCM tasks such as bug fixing, test generation, or summarization, where compression might degrade different quality attributes than code generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper empirically investigates whether 4-bit Activation-aware Weight Quantization (AWQ) degrades not only functional correctness but also qualitative attributes of code generated by large code models. The study uses CodeLlama (7B, 13B, 34B) and DeepSeek-Coder (1.3B, 6.7B, 33B) on MultiPL-E/HumanEval and McEval benchmarks for Python and Java, compares full-precision and AWQ-4bit models, and evaluates the generated code with SonarCloud, Pylint, Flake8, and PMD. It reports pass@1 and static quality metrics, applies Wilcoxon signed-rank tests with Holm correction and Cliff's delta, and includes a manual readability/consistency assessment. The paper concludes that quantization is not a deal-breaker because it preserves functional correctness and largely maintains qualitative code attributes such as maintainability, reliability, and structural simplicity.

Significance. If the central claim holds, the result is practically valuable: it would indicate that 4-bit quantized code models can be deployed with substantial memory and energy savings without a meaningful loss in code quality. The study addresses a real gap, as prior quantization work for code models focused almost exclusively on functional correctness. Strengths include a publicly available replication package, the use of external standard benchmarks and static analysis tools, paired statistical testing with multiple-comparison correction, and a manual annotation component with reported inter-rater agreement. I found no circularity in the central evaluation: the comparisons rely on external benchmarks and external static-analysis outputs. The main weakness is that the headline aggregate comparisons are computed by summing incommensurable metrics without normalization, while the per-metric statistical results are used as evidence of equivalence without equivalence testing; both issues undermine the strength of the conclusion as currently stated.

major comments (4)
  1. [Section V, Tables I and II] The central evidence for the claim that quality is preserved is the raw aggregate comparison (e.g., 2,892 vs. 3,084 issues on McEval-Python). These totals are obtained by summing Security Hotspots, Reliability bugs, Maintainability code smells, Cyclomatic Complexity, and Cognitive Complexity. Cyclomatic and cognitive complexity are continuous scores on a different scale from count-based bug and smell metrics, so the sum has no clear semantic interpretation. The totals are also not normalized by lines of code, although the tables show quantized models producing longer code in several configurations (e.g., CodeLlama 13B on MultiPL-E-Python increases from 1,130 to 1,180 LoC in Table I). The reported 6% reduction is therefore not a valid basis for the conclusion; the authors should either remove the aggregate totals or replace them with a per-LoC or otherwise justified analysis.
  2. [Section V and Section VI] The paper interprets non-significant Wilcoxon tests as evidence that quantized models preserve code quality. With 42 to 161 paired tasks per configuration and no confidence intervals, equivalence bounds, or Bayes factors, failure to reject the null hypothesis does not establish comparability. The manuscript should report per-metric effect sizes with confidence intervals and, for the claim of no meaningful degradation, a pre-specified equivalence test (e.g., TOST) or a clear bound on acceptable difference. Without such bounds, the conclusion that quantization is not a deal-breaker is stronger than the statistical evidence supports.
  3. [Section V, Tables I and II] The favorable aggregate totals mask consistent per-metric degradations that the paper itself notes. For example, PMD Performance issues on MultiPL-E-Java more than double for CodeLlama 34B (14 to 31) and DeepSeek-Coder 33B (13 to 30), and Maintainability code smells increase for several quantized CodeLlama configurations (e.g., CodeLlama 13B on MultiPL-E-Python: 29 to 32). The conclusion that qualitative aspects are preserved needs to address these directional effects explicitly rather than averaging them into a single total, or the claim should be narrowed to specific metrics.
  4. [Section IV-G] The manual readability and consistency evaluation samples only quantized outputs (50 predictions per language from CodeLlama 34B) and reports that 82% of them are acceptable or good. Because full-precision outputs are not rated in the same manual study, these ratings cannot be used to attribute any quality difference to quantization. In addition, the sample covers only one model family and one size, so it cannot support the cross-family conclusion. The authors should rate paired full-precision outputs as well, or explicitly restrict the claim to the absolute readability of quantized outputs.
minor comments (7)
  1. [Section IV-E] The cyclomatic complexity formula is written as M = E + 2Q - N; the standard formula is M = E - N + 2P (or E - N + 2 for a single connected component). Please correct the formula and define the terms.
  2. [Section IV-E] The cognitive complexity formula is rendered as C = Cbase + sum_{i=1}^n nc with an undefined summation index; provide the actual definition or a more complete citation.
  3. [Tables I and II] The color-coding description in Section V mentions green, red, and yellow for positive, negative, and neutral effects, but the threshold for neutrality is not defined; please state the rule used to assign colors.
  4. [Section IV-G] The sentence 'We compute the total number of issues as the sum of SonarCloud metrics' appears in the manual-assessment subsection and appears to belong to the quantitative analysis rather than to the manual readability study; please move or clarify it.
  5. [Section IV-D] No versions or configuration profiles are reported for SonarCloud, Pylint, Flake8, or PMD. Static-analysis counts depend heavily on rule sets and tool versions, so the paper should report the versions and enabled rules, at least in the replication package.
  6. [Section IV-A] The number of generated samples per task for the main pass@1 results is not stated; ten predictions per instance are mentioned only for the variability check in Section VI. Please clarify the sampling protocol for the reported pass@1 values.
  7. [Section VII] The conclusion generalizes from AWQ 4-bit quantization to 'quantization' generally; the title, abstract, and conclusion should consistently qualify that only 4-bit AWQ was evaluated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular construction: the quality comparisons are direct measurements on external benchmarks, and the few self-citations are not load-bearing.

full rationale

The paper contains no derived formal result: 'quality' is operationalized through SonarCloud, PMD, Pylint, and Flake8 outputs, and the full-precision versus AWQ-quantized comparisons are direct measurements on external benchmarks (MultiPL-E and McEval). The aggregate 'total issues' values are computed from the same tables, so the reported 6% reduction is a restatement of measured data rather than a fitted output, a prediction forced by construction, or an equation that reduces to its own inputs. Whether summing heterogeneous metrics is meaningful is a measurement-validity concern, not circularity. The paper's self-citations (e.g., prior work by the authors on code summarization and the replication package) are contextual and do not supply a uniqueness theorem, ansatz, or fitted parameter on which the central conclusion depends. The manual readability assessment rates only quantized outputs, but that is an evidentiary limitation (lack of a full-precision baseline for the manual check), not a definitional tautology. No step in the paper requires accepting the conclusion in order to derive it.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The study makes no fitted derivations; its conclusions rest on measurement assumptions: the validity of static analysis metrics as proxies for code quality, the faithfulness of pre-quantized checkpoints, the additivity of heterogeneous metrics, and the sufficiency of single-sample per task. No new entities are posited.

assumptions (4)
  • domain assumption Static analysis tools (SonarCloud, Pylint, Flake8, PMD) yield valid, comparable measurements of code quality attributes such as maintainability, reliability, and complexity.
    The entire evaluation treats tool outputs as ground truth for code quality; the paper acknowledges this as a construct validity risk in Section VI but does not validate the metrics against human judgment beyond the small manual sample.
  • domain assumption Pre-quantized AWQ 4-bit checkpoints from Hugging Face (TheBloke) faithfully represent AWQ quantization of the corresponding base models, and the 16-bit checkpoints are the correct full-precision baselines.
    Section IV-C states the study used pre-quantized AWQ models 'ensuring adherence to the platform's recommended quantization guidelines'. If the checkpoints were not produced from the same base weights or with equivalent calibration, the comparison is compromised.
  • ad hoc to paper Heterogeneous metrics can be summed into a single 'total number of quality issues'.
    Section V derives total issues by summing Security-Hotspots, Reliability issues, Maintainability Issues, CyC, and CoC. This assumes these dimensions are commensurable and additive, which is not justified.
  • domain assumption A single generation per task per model configuration is sufficient to estimate quality metrics and pass@1 for all models.
    Section VI checks variability only for the smallest models (7B and 1.3B) on McEval; the main tables rely on one output per task, so sampling noise could mask or create differences.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is Quantization a Deal-breaker? Empirical Insights from Large Code Models." pith.science (2026). https://pith.science/paper/IPJJP4ZQ

@misc{pith2026250709665,
  author       = {Pith},
  title        = {Pith review of: Is Quantization a Deal-breaker? Empirical Insights from Large Code Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IPJJP4ZQ}},
  note         = {Machine review of arXiv:2507.09665}
}
read the original abstract

The growing scale of large language models (LLMs) not only demands extensive computational resources but also raises environmental concerns due to their increasing carbon footprint. Model quantization emerges as an effective approach that can reduce the resource demands of LLMs by decreasing parameter precision without substantially affecting performance (e.g., 16 bit to 4 bit). While recent studies have established quantization as a promising approach for optimizing large code models (LCMs), a specialized subset of LLMs tailored for automated software engineering, their findings offer only limited insights into its practical implications. Specifically, current investigations focus only on the functional correctness of the code generated by quantized models, neglecting how quantization impacts critical aspects of code quality such as reliability, maintainability, and security. To bridge this gap, our study investigates the effects of quantization on the qualitative aspects of automatically generated code. We apply Activation-aware Weight Quantization (AWQ) to two widely used code models, CodeLlama and DeepSeekCoder, to generate Java and Python code. Using state-of-the-art static analysis tools, we evaluate software quality metrics and static features including cyclomatic complexity, cognitive complexity, and lines of code. Our findings reveal that quantization is a robust technique that not only preserves functional correctness, but also retains key qualitative code attributes sought after by developers, such as maintainability and structural simplicity.

Figures

Figures reproduced from arXiv: 2507.09665 by the authors.

Figure 1
Figure 1. Example of four distinct implementations of two different given [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Prompt language affects LLM code generation, but English is not consistently best: Chinese prompts improve Python correctness on CoderEval, while quality and lexicon effects vary by model and programming language.

  2. Parameter-Efficient Multi-Task Fine-Tuning in Code-Related Tasks

    cs.SE 2026-01 conditional novelty 5.0 of 10

    Multi-task QLoRA on Qwen2.5-Coder matches or beats single-task QLoRA and full fine-tuning for code generation and Python summarization, but lags in Java-to-C# translation.

Reference graph

Works this paper leans on

80 extracted references · 40 canonical work pages · cited by 2 Pith papers

  1. [1]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology , 2023

    work page 2023

  2. [2]

    A systematic literature review on the use of deep learning in software engineering research,

    C. Watson, N. Cooper, D. N. Palacio, K. Moran, and D. Poshyvanyk, “A systematic literature review on the use of deep learning in software engineering research,” ACM Transactions on Software Engineering and Methodology, TOSEM, vol. 31, no. 2, pp. 1–58, 2022

    work page 2022

  3. [3]

    “Github,” https://github.com/, accessed: 2022-11-10

    work page 2022

  4. [4]

    GitHub Copilot – Your AI pair programmer,

    “GitHub Copilot – Your AI pair programmer,” https://github.com/ features/copilot/, accessed: 2024-03-10

    work page 2024

  5. [5]

    Starcoder: may the source be with you!

    R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al., “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161 , 2023

    arXiv 2023

  6. [6]

    Sustainable ai: Environmental implications, challenges and opportunities,

    C.-J. Wu, R. Raghavendra, U. Gupta, B. Acun, N. Ardalani, K. Maeng, G. Chang, F. Aga, J. Huang, C. Baiet al., “Sustainable ai: Environmental implications, challenges and opportunities,” Proceedings of Machine Learning and Systems , vol. 4, pp. 795–813, 2022

    2022

  7. [7]

    The growing energy footprint of artificial intelligence,

    A. de Vries, “The growing energy footprint of artificial intelligence,” Joule, vol. 7, no. 10, pp. 2191–2194, 2023

    2023

  8. [8]

    Reducing the carbon impact of generative ai inference (today and in 2035),

    A. A. Chien, L. Lin, H. Nguyen, V . Rao, T. Sharma, and R. Wijayawar- dana, “Reducing the carbon impact of generative ai inference (today and in 2035),” in Proceedings of the 2nd Workshop on Sustainable Computer Systems, 2023, pp. 1–7

    work page 2023

Show all 80 references
  1. [9]

    Llm-qat: Data-free quantization aware training for large language models,

    Z. Liu, B. Oguz, C. Zhao, E. Chang, P. Stock, Y . Mehdad, Y . Shi, R. Kr- ishnamoorthi, and V . Chandra, “Llm-qat: Data-free quantization aware training for large language models,” arXiv preprint arXiv:2305.17888 , 2023

    2023 arXiv

  2. [10]

    8-bit optimizers via block-wise quantization,

    T. Dettmers, M. Lewis, S. Shleifer, and L. Zettlemoyer, “8-bit optimizers via block-wise quantization,” arXiv preprint arXiv:2110.02861 , 2021

    2021 arXiv

  3. [11]

    A survey of quantization methods for efficient neural network infer- ence,

    A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer, “A survey of quantization methods for efficient neural network infer- ence,” in Low-Power Computer Vision. Chapman and Hall/CRC, 2022, pp. 291–326

    2022

  4. [12]

    Towards greener yet powerful code generation via quantization: An empirical study,

    X. Wei, S. K. Gonugondla, S. Wang, W. Ahmad, B. Ray, H. Qian, X. Li, V . Kumar, Z. Wang, Y . Tianet al., “Towards greener yet powerful code generation via quantization: An empirical study,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Sympo...

    2023

  5. [14]

    Gptq: Accurate post-training quantization for generative pre-trained transformers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  6. [15]

    Edgeqat: Entropy and distribution guided quantization-aware training for the acceleration of lightweight llms on the edge,

    X. Shen, Z. Kong, C. Yang, Z. Han, L. Lu, P. Dong, C. Lyu, C.-h. Li, X. Guo, Z. Shu et al. , “Edgeqat: Entropy and distribution guided quantization-aware training for the acceleration of lightweight llms on the edge,” arXiv preprint arXiv:2402.10787 , 2024

    2024 arXiv

  7. [16]

    Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quanti- zation for on-device llm compression and acceleration,” Proceedings of Machine Learning and Systems , vol. 6, pp. 87–100, 2024

    2024

  8. [17]

    Resource-efficient & effective code summarization,

    S. Afrin, J. Call, K.-N. Nguyen, O. Chaparro, and A. Mastropaolo, “Resource-efficient & effective code summarization,” arXiv preprint arXiv:2502.03617, 2025

    2025 arXiv

  9. [18]

    A user-centered security eval- uation of copilot,

    O. Asare, M. Nagappan, and N. Asokan, “A user-centered security eval- uation of copilot,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024, pp. 1–11

    2024

  10. [19]

    Is github’s copilot as bad as humans at introducing vulnerabilities in code?

    ——, “Is github’s copilot as bad as humans at introducing vulnerabilities in code?” Empirical Software Engineering, vol. 28, no. 6, p. 129, 2023

    2023

  11. [20]

    How secure is code generated by chatgpt?

    R. Khoury, A. R. Avila, J. Brunelle, and B. M. Camara, “How secure is code generated by chatgpt?” in 2023 IEEE international conference on systems, man, and cybernetics (SMC) . IEEE, 2023, pp. 2445–2451

    2023

  12. [21]

    Quality assessment of chatgpt generated code and their use by developers,

    M. L. Siddiq, L. Roney, J. Zhang, and J. C. D. S. Santos, “Quality assessment of chatgpt generated code and their use by developers,” in Proceedings of the 21st International Conference on Mining Software Repositories, 2024, pp. 152–156

    2024

  13. [22]

    Replication package,

    “Replication package,” https://github.com/saimaafrin/ quantized-model-code-quality.git, 2025

    2025

  14. [23]

    Codegen: An open large language model for code with multi-turn program synthesis,

    E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” in The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,

    2023

  15. [24]

    Competition- level code generation with alphacode,

    Y . Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago et al. , “Competition- level code generation with alphacode,” Science, vol. 378, no. 6624, pp. 1092–1097, 2022

    2022

  16. [25]

    A systematic evaluation of large language models of code,

    F. F. Xu, U. Alon, G. Neubig, and V . J. Hellendoorn, “A systematic evaluation of large language models of code,” in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, 2022, pp. 1–10

    2022

  17. [28]

    The claude 3 model family: Opus, sonnet, haiku,

    Anthropic, “The claude 3 model family: Opus, sonnet, haiku,” https://www-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model Card Claude 3.pdf, n.d., accessed: 2025-02-26

    2025

  18. [29]

    Gemini: A family of highly capable multimodal models,

    G. T. et al., “Gemini: A family of highly capable multimodal models,”

  19. [30]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  20. [31]

    Carbon emissions and large neural network training,

    D. Patterson, J. Gonzalez, Q. Le, C. Liang, L.-M. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean, “Carbon emissions and large neural network training,” arXiv preprint arXiv:2104.10350 , 2021

    2021 arXiv

  21. [32]

    Exploring the carbon footprint of hugging face’s ml models: A repository mining study,

    J. Casta ˜no, S. Mart´ınez-Fern´andez, X. Franch, and J. Bogner, “Exploring the carbon footprint of hugging face’s ml models: A repository mining study,” in 2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) . IEEE, 2023, pp. 1–12

    2023

  22. [33]

    Energy and policy con- siderations for modern deep learning research,

    E. Strubell, A. Ganesh, and A. McCallum, “Energy and policy con- siderations for modern deep learning research,” in Proceedings of the AAAI conference on artificial intelligence , vol. 34, no. 09, 2020, pp. 13 693–13 696

    2020

  23. [34]

    Efficient and green large language models for software engineering: Vision and the road ahead,

    J. Shi, Z. Yang, and D. Lo, “Efficient and green large language models for software engineering: Vision and the road ahead,” ACM Transactions on Software Engineering and Methodology , 2024

    2024

  24. [35]

    Learned step size quantization,

    S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha, “Learned step size quantization,” arXiv preprint arXiv:1902.08153, 2019

    1902 arXiv

  25. [36]

    Zeroq: A novel zero shot quantization framework,

    Y . Cai, Z. Yao, Z. Dong, A. Gholami, M. W. Mahoney, and K. Keutzer, “Zeroq: A novel zero shot quantization framework,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 13 169–13 178

    2020

  26. [37]

    PB-LLM: partially binarized large language models,

    Z. Yuan, Y . Shang, and Z. Dong, “PB-LLM: partially binarized large language models,” in The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. [Online]. Available: https: //openreview.net/forum?id=B...

    2024

  27. [38]

    Spqr: A sparse- quantized representation for near-lossless llm weight compression,

    T. Dettmers, R. Svirschevski, V . Egiazarian, D. Kuznedelev, E. Frantar, S. Ashkboos, A. Borzunov, T. Hoefler, and D. Alistarh, “Spqr: A sparse- quantized representation for near-lossless llm weight compression,” arXiv preprint arXiv:2306.03078 , 2023

    2023 arXiv

  28. [39]

    Smoothquant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “Smoothquant: Accurate and efficient post-training quantization for large language models,” in International Conference on Machine Learning . PMLR, 2023, pp. 38 087–38 099

    2023

  29. [40]

    Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale,

    T. Dettmers, M. Lewis, Y . Belkada, and L. Zettlemoyer, “Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale,”Advances in Neural Information Processing Systems , vol. 35, pp. 30 318–30 332, 2022

    2022

  30. [41]

    Do users write more insecure code with ai assistants?

    N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do users write more insecure code with ai assistants?” in Proceedings of the 2023 ACM SIGSAC conference on computer and communications security , 2023, pp. 2785–2799

    2023

  31. [42]

    Lost at c: A user study on the security implications of large language model code assistants,

    G. Sandoval, H. Pearce, T. Nys, R. Karri, S. Garg, and B. Dolan-Gavitt, “Lost at c: A user study on the security implications of large language model code assistants,” in 32nd USENIX Security Symposium (USENIX Security 23), 2023, pp. 2205–2222

    2023

  32. [43]

    On the robustness of code generation techniques: An empirical study on github copilot,

    A. Mastropaolo, L. Pascarella, E. Guglielmi, M. Ciniselli, S. Scalabrino, R. Oliveto, and G. Bavota, “On the robustness of code generation techniques: An empirical study on github copilot,” in 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Mel- bour...

    2023

  33. [44]

    Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt,

    B. Yetis ¸tiren, I. ¨Ozsoy, M. Ayerdem, and E. T ¨uz¨un, “Evaluating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt,” 2023. [Online]. Available: https://arxiv.org/abs/2304.10778

    2023 arXiv

  34. [45]

    Amazon CodeWhisperer,

    “Amazon CodeWhisperer,” https://aws.amazon.com/codewhisperer/, ac- cessed: 2024-03-10

    2024

  35. [46]

    Sonarcloud,

    SonarSource, “Sonarcloud,” https://docs.sonarsource.com/ sonarqube-cloud/, accessed: 2025-03-03

    2025

  36. [47]

    Codellama,

    “Codellama,” https://github.com/meta-llama/codellama/tree/main

  37. [48]

    Reflectioncoder: Learning from reflection sequence for enhanced one-off code genera- tion,

    H. Ren, M. Zhan, Z. Wu, A. Zhou, J. Pan, and H. Li, “Reflectioncoder: Learning from reflection sequence for enhanced one-off code genera- tion,” arXiv preprint arXiv:2405.17057 , 2024

    2024 arXiv

  38. [49]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Li et al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024

    2024 arXiv

  39. [50]

    Structured chain-of-thought prompting for code generation,

    J. Li, G. Li, Y . Li, and Z. Jin, “Structured chain-of-thought prompting for code generation,” ACM Transactions on Software Engineering and Methodology, 2023

    2023

  40. [51]

    A performance study of llm- generated code on leetcode,

    T. Coignion, C. Quinton, and R. Rouvoy, “A performance study of llm- generated code on leetcode,” in Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering , 2024, pp. 79–89

    2024

  41. [52]

    Evaluating instruction-tuned large language models on code comprehension and generation,

    Z. Yuan, J. Liu, Q. Zi, M. Liu, X. Peng, and Y . Lou, “Evaluating instruction-tuned large language models on code comprehension and generation,” arXiv preprint arXiv:2308.01240 , 2023

    2023 arXiv

  42. [53]

    Instructcoder: Instruction tuning large language models for code editing,

    K. Li, Q. Hu, X. Zhao, H. Chen, Y . Xie, T. Liu, Q. Xie, and J. He, “Instructcoder: Instruction tuning large language models for code editing,” arXiv preprint arXiv:2310.20329 , 2023

    2023 arXiv

  43. [54]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  44. [55]

    Language models are few-shot learners,

    T. B. Brown, “Language models are few-shot learners,” arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  45. [57]

    Knowledge transfer from high- resource to low-resource programming languages for code llms,

    F. Cassano, J. Gouwar, F. Lucchetti, C. Schlesinger, C. J. Anderson, M. Greenberg, A. Jangda, and A. Guha, “Knowledge transfer from high- resource to low-resource programming languages for code llms,” arXiv preprint arXiv:2308.09895, 2023

    2023 arXiv

  46. [58]

    Multipl-e: a scalable and polyglot approach to benchmarking neural code generation,

    F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y . Zi, C. J. Anderson, M. Q. Feldman et al., “Multipl-e: a scalable and polyglot approach to benchmarking neural code generation,” IEEE Transactions on Software Engineering , 2023

    2023

  47. [59]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021

    2021 arXiv

  48. [60]

    Mceval: Massively multilingual code evaluation,

    L. Chai, S. Liu, J. Yang, Y . Yin, K. Jin, J. Liu, T. Sun, G. Zhang, C. Ren, H. Guo et al., “Mceval: Massively multilingual code evaluation,” arXiv preprint arXiv:2406.07436, 2024

    2024 arXiv

  49. [61]

    Generate and pray: Using sallms to evaluate the security of llm generated code,

    M. L. Siddiq and J. C. Santos, “Generate and pray: Using sallms to evaluate the security of llm generated code,” arXiv preprint arXiv:2311.00889, 2023

    2023 arXiv

  50. [62]

    Pylint - code analysis for python,

    PylintTeam, “Pylint - code analysis for python,” https://www.pylint.org/, accessed: 2025-03-03

    2025

  51. [63]

    Checkstyle,

    “Checkstyle,” http://checkstyle.sourceforge.net/, accessed: 2022-11-10

    2022

  52. [64]

    Pmd - source code analyzer,

    P. D. Team, “Pmd - source code analyzer,” 2025, static code analysis tool for Java and other languages. [Online]. Available: https://pmd.github.io

    2025

  53. [65]

    Refining chatgpt-generated code: Characterizing and mitigating code quality issues,

    Y . Liu, T. Le-Cong, R. Widyasari, C. Tantithamthavorn, L. Li, X.- B. D. Le, and D. Lo, “Refining chatgpt-generated code: Characterizing and mitigating code quality issues,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 5, pp. 1–26, 2024

    2024

  54. [66]

    Security and quality in llm-generated code: A multi-language, multi-model analysis,

    M. Kharma, S. Choi, M. AlKhanafseh, and D. Mohaisen, “Security and quality in llm-generated code: A multi-language, multi-model analysis,” arXiv preprint arXiv:2502.01853 , 2025

    2025

  55. [67]

    Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,

    J. Liu, C. S. Xia, Y . Wang, and L. Zhang, “Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation,” Advances in Neural Information Processing Systems , vol. 36, pp. 21 558–21 572, 2023

    2023

  56. [68]

    Flake8: Your tool for style guide enforcement. 2021,

    T. Ziad ´e and I. Cordasco, “Flake8: Your tool for style guide enforcement. 2021,” URL: http://flake8. pycqa. org (besucht am 27. 05. 2019)

    2021

  57. [69]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter,...

    2021 arXiv

  58. [70]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez et al. , “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023

    2023 arXiv

  59. [71]

    Wizardcoder: Empowering code large language models with evol-instruct,

    Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang, “Wizardcoder: Empowering code large language models with evol-instruct,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net...

    2024

  60. [72]

    A review on code generation with llms: Application and evaluation,

    J. Wang and Y . Chen, “A review on code generation with llms: Application and evaluation,” in 2023 IEEE International Conference on Medical Artificial Intelligence (MedAI) . IEEE, 2023, pp. 284–289

    2023

  61. [73]

    Llm- based test-driven interactive code generation: User study and empirical evaluation,

    S. Fakhoury, A. Naik, G. Sakkas, S. Chakraborty, and S. K. Lahiri, “Llm- based test-driven interactive code generation: User study and empirical evaluation,” IEEE Transactions on Software Engineering , 2024

    2024

  62. [74]

    An empirical validation of cognitive complexity as a measure of source code understandability,

    M. Mu ˜noz Bar ´on, M. Wyrich, and S. Wagner, “An empirical validation of cognitive complexity as a measure of source code understandability,” in Proceedings of the 14th ACM/IEEE international symposium on empirical software engineering and measurement (ESEM) , 2020, pp. 1–12

    2020

  63. [75]

    Individual comparisons by ranking methods,

    F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945

    1945

  64. [76]

    A simple sequentially rejective multiple test procedure,

    S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian journal of statistics , pp. 65–70, 1979

    1979

  65. [77]

    R. J. Grissom and J. J. Kim, Effect sizes for research: A broad practical approach, 2nd ed. Lawrence Earlbaum Associates, 2005

    2005

  66. [78]

    A coefficient of agreement for nominal scales,

    J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement , vol. 20, no. 1, pp. 37–46, 1960

    1960

  67. [79]

    Likert scale: Explored and explained,

    A. Joshi, S. Kale, S. Chandel, and D. K. Pal, “Likert scale: Explored and explained,” British journal of applied science & technology , vol. 7, no. 4, p. 396, 2015

    2015

  68. [80]

    Enhancing trust in llm-generated code summaries with calibrated confidence scores,

    Y . Virk, P. Devanbu, and T. Ahmed, “Enhancing trust in llm-generated code summaries with calibrated confidence scores,” arXiv preprint arXiv:2404.19318, 2024

    2024 arXiv

  69. [81]

    On the effectiveness of large language models in statement-level code summarization,

    J. Zhu, Y . Miao, T. Xu, J. Zhu, and X. Sun, “On the effectiveness of large language models in statement-level code summarization,” in 2024 IEEE 24th International Conference on Software Quality, Reliability and Security. IEEE, 2024, pp. 216–227

    2024

  70. [82]

    Synthe- sizing text-to-sql data from weak and strong llms,

    J. Yang, B. Hui, M. Yang, J. Yang, J. Lin, and C. Zhou, “Synthe- sizing text-to-sql data from weak and strong llms,” arXiv preprint arXiv:2408.03256, 2024

    2024 arXiv

  71. [2023]

    OpenReview.net, 2023

    2023

  72. [2024]

    Available: https://arxiv.org/abs/2312.11805

    [Online]. Available: https://arxiv.org/abs/2312.11805

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.