Pith. sign in

REVIEW 4 major objections 5 minor 51 references

On Iterative Evaluation and Enhancement of Code Quality Using GPT-4o

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM that scores code on ten quality dimensions and only accepts rewrites that score higher can evaluate and improve code, yielding a mean relative improvement of 52.6% on 42 Python/JavaScript examples.

desk verdict A cleanly written LLM code-quality loop whose headline improvement number is mostly a self-scoring artifact; the framework and qualitative results are worth a look, but the 52.6% claim doesn't survive contact with its own acceptance rule. read the letter →

arxiv 2502.07399 v1 pith:EAOD4OP6 submitted 2025-02-11 cs.SE cs.AI

Rundong Liu , Andre Frade , Amal Vaidya , Maxime Labonne , Marcus Kaiser , Bismayan Chakrabarti , Jonathan Budd , Sean Moran This is my paper · ORCID
classification cs.SEcs.AI
keywords codequalityevaluationlargelanguagemodelsGPT-4oiterativeimprovementstaticanalysisvalidationPythonJavaScriptsoftwaremaintainability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that an LLM can be turned into a dependable code-quality referee and improver if the judgment is structured rather than left open-ended. CodeQUEST scores code on ten named quality dimensions, each probed by five forced-choice statements, and then runs an optimizer that rewrites the code using the qualitative feedback and accepts only rewrites whose overall score does not drop. On 42 hand-curated Python and JavaScript examples, this loop improved 41 of them, with a mean relative percentage improvement of 52.6% (median 57.1%) over at most five iterations. The claim matters because it offers a concrete, configurable recipe for automating a large share of code review and refactoring, with the score acting as a built-in quality floor.

What carries the argument

The load-bearing mechanism is the Evaluator-Optimizer loop with its acceptance rule. The Evaluator decomposes code quality into ten named dimensions—Readability, Maintainability, Testability, Efficiency, Robustness, Security, Documentation, Modularity, Scalability, Portability—and for each dimension asks five questions answerable only by 'True', 'False', or 'Not Applicable'; mapping those to $+1$, $-1$, and $0$ and averaging yields a code-level score in $[-5,5]$. The Optimizer feeds the qualitative dimension feedback back to GPT-4o, rewrites the code, rejects rewrites that fail compilation or supplied tests, and keeps a rewrite only if its overall CodeQUEST score does not drop relative to the last accepted version. That acceptance rule is what makes improvement monotone by construction; the external proxy metrics validate the scores but do not participate in the loop.

What would settle it

Have expert programmers blindly rate the original and final versions of the 138 improved Python outputs; if experts do not prefer the final versions, or if a control loop that replaces CodeQUEST's score with a random score produces a similar 52.6% 'improvement', then the reported gain is an artifact of the acceptance rule rather than real quality improvement.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that LLM-based code-quality assessment is prompt-sensitive in a fixable way: a dimension-decomposed, forced-choice evaluation produces more comprehensive qualitative feedback and better-calibrated scores than a simple chain-of-thought rating, and those scores can drive a monotonic improvement loop. The framework's Evaluator produces a code-level score as the average of ten dimension scores, each in $[-5,5]$; the Optimizer rewrites code from the dimension feedback, validates that the rewrite compiles and passes available tests, and rejects any version whose overall score falls. Across the curated dataset, 41 of 42 examples improved, the mean relative percentage improvement was $52.6\% \pm 17.9\%$, and the per-iteration change in CodeQUEST scores correlated with per-iteration changes in Pylint, Radon, and Bandit proxies more strongly than the baseline did. The paper also reports that CodeQUEST caught hardcoded API keys in SecurityEval examples that Bandit reported as clean, indicating the LLM evaluator can complement static analyzers.

Load-bearing premise

The load-bearing premise is that the GPT-4o-generated numerical score genuinely tracks code quality, so a higher score after rewriting means better code and not merely a version the model prefers.

Editorial extensions

If this is right

  • Most of the quality gain arrives in the first iteration cycle (about 2 code-quality units on average), with later iterations adding 0.52, 0.27, and then below 0.1 units, so a one- or two-iteration budget captures most of the benefit at lower API cost.
  • Because the Optimizer rejects any rewrite whose overall CodeQUEST score falls, the final delivered version is guaranteed by construction to score at least as high as the original on the framework's own scale.
  • The framework's per-iteration score changes correlate with changes in Pylint, Radon Maintainability Index, and Bandit proxies (Pearson $r = 0.53$ for CodeQUEST versus $0.27$ for the baseline), suggesting the evaluator tracks externally measurable quality shifts better than a generic chain-of-thought prompt does.
  • On the SecurityEval examples, CodeQUEST flagged hardcoded API keys that Bandit missed, so an LLM-based evaluator can complement rather than replace static analysis security tools.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next experiment, not run here, would swap the Evaluator's score for a random score while keeping the same acceptance rule; if the reported improvement persists, the acceptance rule—not genuine quality tracking—would be doing the work.
  • The same judge-and-accept loop could be repurposed for other quality targets, such as performance budgets or accessibility, by replacing the ten dimensions; the paper's generality claim would then be testable without new machinery.
  • The paper's own threat-to-validity section concedes hallucination, stochasticity, small dataset size, and dependence on GPT-4o's training coverage; those caveats mean the 52.6% figure is better read as evidence for the recipe than as a guarantee for arbitrary codebases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript introduces CodeQUEST, a GPT-4o-based framework for evaluating and iteratively improving code quality across ten dimensions (e.g., readability, maintainability, efficiency, security). The Evaluator produces per-dimension quantitative scores and qualitative feedback; the Optimizer uses that feedback to rewrite code, validates compilation, and accepts only rewrites whose CodeQUEST score increases relative to the previous accepted version. On a hand-curated dataset of 42 examples (28 Python, 14 JavaScript), the authors report that 41 of 42 examples improved under their framework, with a mean relative percentage improvement (RPI) of 52.6%. They also compare CodeQUEST's scores against proxy metrics (Pylint, Radon, Bandit), reporting correlations between per-iteration changes in the CodeQUEST score and the proxy score, and they provide qualitative examples, including a security-focused case study.

Significance. If the claims were supported, CodeQUEST would be a practically useful contribution: an open-source, configurable LLM pipeline for automated code-quality assessment and iterative improvement. The paper's strengths include the detailed description of the ten-dimension evaluation rubric, the explicit optimization loop with code validation, the release of code, and the SecurityEval examples showing that CodeQUEST detects hard-coded secrets that Bandit misses. However, the empirical evidence as presented does not establish the central claims. The improvement result is to a large degree a consequence of the acceptance criterion, and the external validation is based on non-independent observations with modest correlations and no human evaluation or functional testing for most examples. The potential of the approach is real, but the current manuscript does not convincingly demonstrate that CodeQUEST reliably improves actual code quality or that its aggregate score is a valid measure of code quality.

major comments (4)
  1. [Section 2.2.3 and Section 4.2] The reported improvement (41/42 examples, mean RPI 52.6%) is a direct consequence of the Optimizer's acceptance rule, which only accepts a new version if its CodeQUEST score, produced by the same GPT-4o evaluator that generated the feedback, is higher than the previous accepted version. The paper acknowledges this in Section 4.2 ("by design, ensures a monotonic improvement"), but this monotonicity is a property of the selection rule, not evidence about actual code quality. To support the headline claim, the authors would need to compare initial and final code versions against external quality measures (e.g., proxy scores, test outcomes, human judges) rather than against the same model's score.
  2. [Section 4.3, Table 3] The proxy validation is too weak to break the circularity around the improvement claim. The correlations are computed over 138 incremental observations that are not independent, since they come from only 28 Python examples across up to five iterations each. The strongest relationship is Pearson rp=0.53, while the Spearman rank correlation is only rs=0.23, and the analysis excludes two invalid attempts (footnote 6). No confidence intervals or clustered/mixed-effects analysis are provided. This does not establish that the CodeQUEST aggregate score is a valid measure of code quality, and it offers no direct evidence for the JavaScript portion of the dataset. A more informative test would be to report whether the proxy scores themselves improve from the original to the final version.
  3. [Section 2.2.2 and Section 4.2] Semantic preservation is verified only for the 8 MBPP examples, which have test cases; for the other 34 examples, only compilation (or syntactic validity) is checked. Given that the Optimizer is instructed to make "meaningful changes" across all ten quality dimensions, the accepted rewrites may improve the GPT-4o score while changing or breaking intended behavior. The paper's own Section 5 acknowledges the hallucination risk, but no mitigation is reported for the non-MBPP examples. The authors should either run functional tests or provide a manual semantic check for all 42 examples before claiming that code quality improved.
  4. [Section 4.1] The claim that CodeQUEST "can effectively and robustly evaluate code quality" is supported mainly by qualitative comparison on a single example (Table 2) and a figure showing score distributions. No quantitative accuracy measure against ground-truth quality labels is provided. The proxy correlations in Section 4.3 are the only quantitative evidence, and they are modest. The authors should consider a more direct validation, such as agreement with human quality ratings on a larger sample.
minor comments (5)
  1. [Section 2.1, footnote 2] The description of the optional self-consistency mechanism is placed in a footnote that interrupts the discussion of the prompt; move it to the main text or an appendix for readability.
  2. [Table 1] The column heading "T est cases" contains an erroneous space; additionally, the use of "True" and "False" in that column is unconventional and should be replaced with "Yes"/"No" or with a count of test cases.
  3. [Section 4.3, Table 3] Report 95% confidence intervals for the Pearson and Spearman correlations and state whether the p-values are one- or two-sided.
  4. [Appendices A and D] The code listings contain numerous OCR-like artifacts (e.g., "__ in it_ _", "C o n s t r u c t s", "m a x _ h e i g h t"); these should be cleaned to match the actual source code.
  5. [Section 3.2] The baseline prompt asks for a score on the same −5 to 5 scale as a single dimension, while CodeQUEST's overall score is an average over ten dimensions; the comparison of absolute scores between baseline and CodeQUEST may be misleading and should be clarified.

Circularity Check

1 steps flagged · score 7.0 of 10

The 52.6% mean RPI is produced by a self-scoring acceptance rule; the reported code-quality improvement is defined and measured by the same GPT-4o-based Evaluator that gates acceptance, with only weak external proxy validation.

  1. self definitional [Section 2.2.3 (Evaluator Assessment) and Section 4.2 (Eq. 1)]
    "if the overall quality score of the new code version drops relatively to the one of the previous version indicates that no overall improvement was achieved. ... Otherwise, if the overall quality score of the new code increases relative to the last successful version, the iteration is deemed successful. ... We also verified that while our setup, by design, ensures a monotonic improvement of overall code quality"

    The acceptance rule defines 'improvement' as an increase in the CodeQUEST/GPT-4o score; rewritten code is kept only when that score rises. Section 4.2 then reports RPI = 100(s_f - s_i)/(s_max - s_i) using those same CodeQUEST scores, yielding 52.6% mean RPI. Positive RPI is therefore guaranteed by the selection criterion (up to the s_max ceiling), not by any externally measured change in code quality. The loop is closed: the model scores, the model rewrites to raise that score, and the same score is used as the outcome measure.

full rationale

The paper's headline quantitative result—41 of 42 examples improved, mean RPI 52.6%—is computed from CodeQUEST's own GPT-4o quality scores. Because Section 2.2.3 makes an increase in that same score the necessary and sufficient condition for accepting a rewritten version, the reported improvement is enforced by the algorithm rather than measured against an independent standard. The paper even states 'by design, ensures a monotonic improvement of overall code quality.' The external proxy validation in Section 4.3 does not break this: it covers only 28 Python examples, uses 138 non-independent delta observations, and achieves only Pearson rp=0.53 and Spearman rs=0.23 between CodeQUEST and proxy deltas, with no human evaluation and semantic preservation checked only for the 8 MBPP examples. The SecurityEval examples in Appendix E show that CodeQUEST can flag hardcoded API keys, but they do not validate the 50-point aggregate score used to compute RPI. No self-citation chain is load-bearing here; the circularity is internal, between the optimizer's acceptance rule and the evaluation metric. Score 7 reflects that the central improvement claim reduces by construction, while the weak external proxy evidence prevents a higher score.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the assumption that GPT-4o's self-reported score is a valid quality measure, plus several domain assumptions about proxies and validation. No external benchmark or formal verification supports these assumptions.

free parameters (4)
  • Answer score mapping = +1 / -1 / 0
    Maps True/False/Not Applicable answers to numeric scores; the sum over five statements gives the dimension score (Section 2.1). This scale is chosen by the authors, not calibrated to external quality.
  • Maximum optimization iterations = 5
    Experiment hyperparameter that bounds the improvement loop (Section 4.2); more iterations would likely change the reported improvement.
  • Quality dimensions and statements = 10 dimensions, 5 statements each
    Hand-crafted rubric with no independent evidence that these statements fully cover code quality (Appendix B).
  • Code-level score aggregation = arithmetic mean of dimension scores
    Choice of aggregation function affects all reported scores and the acceptance decisions (Section 2.1).
assumptions (5)
  • domain assumption GPT-4o's True/False/Not Applicable answers accurately reflect code quality.
    The Evaluator's entire scoring pipeline depends on this (Section 2.1).
  • ad hoc to paper A higher CodeQUEST score after rewriting means better code.
    The optimizer's acceptance rule in Section 2.2.3 assumes score increases are genuine improvements.
  • domain assumption Compilation is a sufficient validity check for code correctness.
    Section 2.2.2 treats compilation as the minimal requirement; for most examples no tests check semantics.
  • domain assumption Pylint, Radon, and Bandit are valid proxies for the code quality dimensions.
    Section 3.3 uses these tools to validate LLM scores without establishing that they measure the ten dimensions.
  • domain assumption Hand-curated and manually degraded examples are representative of real code quality problems.
    Section 3.1 describes manual degradation to create improvement opportunities, which may inflate measured gains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Iterative Evaluation and Enhancement of Code Quality Using GPT-4o." pith.science (2026). https://pith.science/paper/EAOD4OP6

@misc{pith2026250207399,
  author       = {Pith},
  title        = {Pith review of: On Iterative Evaluation and Enhancement of Code Quality Using GPT-4o},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EAOD4OP6}},
  note         = {Machine review of arXiv:2502.07399}
}
read the original abstract

This paper introduces CodeQUEST, a novel framework leveraging Large Language Models (LLMs) to iteratively evaluate and enhance code quality across multiple dimensions, including readability, maintainability, efficiency, and security. The framework is divided into two main components: an Evaluator that assesses code quality across ten dimensions, providing both quantitative scores and qualitative summaries, and an Optimizer that iteratively improves the code based on the Evaluator's feedback. Our study demonstrates that CodeQUEST can effectively and robustly evaluate code quality, with its assessments aligning closely with established code quality metrics. Through a series of experiments using a curated dataset of Python and JavaScript examples, CodeQUEST demonstrated significant improvements in code quality, achieving a mean relative percentage improvement of 52.6%. The framework's evaluations were validated against a set of proxy metrics comprising of Pylint Score, Radon Maintainability Index, and Bandit output logs, showing a meaningful correlation. This highlights the potential of LLMs in automating code quality evaluation and improvement processes, presenting a significant advancement toward enhancing software development practices. The code implementation of the framework is available at: https://github.com/jpmorganchase/CodeQuest.

Figures

Figures reproduced from arXiv: 2502.07399 by the authors.

Figure 1
Figure 1. Schematic representation of CodeQUEST. different results [15] or “hallucinations” which can be challenging to detect and control [18] [9]. Nonetheless, it is clear that LLMs hold immense potential to revolutionize code quality evaluation and improvement. In this paper, we introduce CodeQUEST (Code Quality Understanding and Enhancement System Toolkit), an LLM-powered framework for assessing and improving code quality… view at source ↗
Figure 2
Figure 2. Comparison of the Baseline and CodeQUEST code quality scores. [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Quality score evolution of a Python code example (mbpp/927.py) [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of code quality score improvements per iteration. (We [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Left: Scatter plots and linear regression line for [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 45 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Alt- man, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Advancing GenAI Assisted Programming--A Comparative Study on Prompt Efficiency and Code Quality Between GPT-4 and GLM-4

    Jie Li Angus Yang, Zehan Li. Advancing genai assisted programming–a comparative study on prompt efficiency and code quality between gpt-4 and glm-4. ArXiv, abs/2402.12782, 2024

    work page Pith review arXiv 2024

  3. [3]

    Enhancing decision-making of large language models via actor-critic

    Anonymous. Enhancing decision-making of large language models via actor-critic. In Submitted to The Thirteenth International Conference on Learning Representations, 2024. under review

    work page 2024

  4. [4]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. 16

    arXiv 2021

  5. [5]

    https://github.com/auth0/node-jsonwebtoken

    auth0. https://github.com/auth0/node-jsonwebtoken

    work page

  6. [6]

    Cdk examples https://github.com/aws-samples/aws-cdk-examples

    A WS. Cdk examples https://github.com/aws-samples/aws-cdk-examples

    work page

  7. [7]

    https://github.com/pycqa/bandit

    Bandit. https://github.com/pycqa/bandit

    work page

  8. [8]

    Is gpt-4 a reliable rater? evaluating consistency in gpt-4’s text ratings

    Veronika Hackl, Alexandra Elena M¨ uller, Michael Granitzer, and Maxim- ilian Sailer. Is gpt-4 a reliable rater? evaluating consistency in gpt-4’s text ratings. Frontiers in Education, 8, December 2023

    work page 2023

Show all 51 references
  1. [9]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Hao- tian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023

    2023

  2. [10]

    Re- view of recent systems for automatic assessment of programming assign- ments

    Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Sepp¨ al¨ a. Re- view of recent systems for automatic assessment of programming assign- ments. In Proceedings of the 10th Koli Calling International Conference on Computing Education Research , Koli Calling ’10, page 8...

    2010

  3. [11]

    Science-js https://github.com/jasondavies/science.js/

    Davies Jason. Science-js https://github.com/jasondavies/science.js/

  4. [12]

    A latent space theory for emergent abilities in large language models, 2023

    Hui Jiang. A latent space theory for emergent abilities in large language models, 2023

    2023

  5. [13]

    Large language models are zero-shot reasoners, 2023

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023

    2023

  6. [14]

    Actor-critic algorithms

    Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In S. Solla, T. Leen, and K. M¨ uller, editors,Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999

    1999

  7. [15]

    Gpt understands, too

    Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. AI Open, 2023

    2023

  8. [16]

    Le, and David Lo

    Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamtha- vorn, Li Li, Xuan-Bach D. Le, and David Lo. Refining chatgpt-generated code: Characterizing and mitigating code quality issues. ACM Transactions on Software Engineering and Methodology, Volume 33, Issue 5, Art...

    2024

  9. [17]

    Code quality assessment using trans- formers, 2023

    Mosleh Mahamud and Isak Samsten. Code quality assessment using trans- formers, 2023

    2023

  10. [18]

    On faithfulness and factuality in abstractive summarization

    Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational ...

    1906

  11. [19]

    Radon mi https://radon.readthedocs.io/en/latest/intro.html

    Lacchia Michele. Radon mi https://radon.readthedocs.io/en/latest/intro.html

  12. [20]

    Ndukwe, Sherlock A

    Ifeanyi G. Ndukwe, Sherlock A. Licorish, Amjed Tahir, and Stephen G. MacDonell. How have views on software quality differed over time? research and practice viewpoints. Journal of Systems and Software , 195:111524, 2023

    2023

  13. [21]

    Llm is like a box of chocolates: the non-determinism of chatgpt in code generation

    Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. Llm is like a box of chocolates: the non-determinism of chatgpt in code generation. arXiv preprint arXiv:2308.02828 , 2023

    2023 arXiv

  14. [22]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Pas- sos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit- learn: Machine learning in Python. Journal of Machine Learn...

    2011

  15. [23]

    Poldrack, Thomas Lu, and G

    Russel A. Poldrack, Thomas Lu, and G. Beguˇ s. Ai-assisted coding: Exper- iments with gpt-4. ArXiv, abs/2304.13187, 2023

    2023 arXiv

  16. [24]

    https://github.com/fportantier/vulpy

    Fabian Martinez Portantier. https://github.com/fportantier/vulpy

  17. [25]

    Assessing code review quality with chatgpt: A survey of automated reviewer assignment methods and ex- perimental outcomes

    Jaros law Protasiewicz Przemys law Zydro´ n. Assessing code review quality with chatgpt: A survey of automated reviewer assignment methods and ex- perimental outcomes. Digital Interaction and Machine Intelligence. MIDI

  18. [26]

    https://github.com/snoopysecurity/vulnerable-code- snippets

    Sam Sanoop. https://github.com/snoopysecurity/vulnerable-code- snippets

  19. [27]

    Qscored: A large dataset of code smells and quality metrics

    Tushar Sharma and Marouane Kessentini. Qscored: A large dataset of code smells and quality metrics. In 2021 IEEE/ACM 18th international conference on mining software repositories (MSR) , pages 590–594. IEEE, 2021

    2021

  20. [28]

    Reflexion: language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems ...

    2023

  21. [29]

    Mohammed Latif Siddiq and Joanna C. S. Santos. Securityeval dataset: Mining vulnerability examples to evaluate machine learning-based code generation techniques. In Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Secur...

    2022

  22. [30]

    Calibration and correctness of language models for code, 2024

    Claudio Spiess, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, and Toufique Ahmed. Calibration and correctness of language models for code, 2024. 18

    2024

  23. [31]

    Pylint: https://github.com/pylint-dev/pylint

    Sylvain Th´ enault. Pylint: https://github.com/pylint-dev/pylint

  24. [32]

    Why can large language models generate correct chain-of- thoughts?, 2024

    Rasul Tutunov, Antoine Grosnit, Juliusz Ziomek, Jun Wang, and Haitham Bou-Ammar. Why can large language models generate correct chain-of- thoughts?, 2024

    2024

  25. [33]

    Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, War- ren Weckesser, Jonathan Bright, St´ efan J

    Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, War- ren Weckesser, Jonathan Bright, St´ efan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelso...

    2020

  26. [34]

    Chi, Sha- ran Narang, Aakanksha Chowdhery, and Denny Zhou

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sha- ran Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations , 2023

    2023

  27. [35]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in N...

    2022

  28. [36]

    Testing, checking, linting, pages 51–101

    Matthew Wilkes. Testing, checking, linting, pages 51–101. Apress, Berkeley, CA, 2020

    2020

  29. [37]

    Controlling large language model-based agents for large-scale decision- making: An actor-critic approach, 2024

    Bin Zhang, Hangyu Mao, Jingqing Ruan, Ying Wen, Yang Li, Shao Zhang, Zhiwei Xu, Dapeng Li, Ziyue Li, Rui Zhao, Lijuan Li, and Guoliang Fan. Controlling large language model-based agents for large-scale decision- making: An actor-critic approach, 2024

    2024

  30. [38]

    degraded in quality

    Yuze Zhao, Zhenya Huang, Yixiao Ma, Rui Li, Kai Zhang, Hao Jiang, Qi Liu, Linbo Zhu, and Yu Su. RePair: Automated program repair with process-based feedback. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar, editors, Findings of the Association for Computational Linguistics:...

    2024

  31. [40]

    - The code consistently follows a single specific code style guide

    Readability - Both, variable and function names are descriptive and meaningful. - The code consistently follows a single specific code style guide. - There are comments that clearly explain complex or non-obvious parts of the code provided, without assuming prior knowledge. - ...

  32. [41]

    - The code provided strictly adheres to the DRY (Do not Repeat Your- self) principle, avoiding unnecessary repetition

    Maintainability - The code provided is organized in a logical and understandable man- ner, allowing for easy comprehension. - The code provided strictly adheres to the DRY (Do not Repeat Your- self) principle, avoiding unnecessary repetition. - Code features can be added or mo...

  33. [42]

    - The code provided produces consistent and predictable outputs for specific inputs

    Testability - The structure of the code provided facilitates easy mocking of depen- dencies. - The code provided produces consistent and predictable outputs for specific inputs. - The code provided is free of global states and variables. - The code provided is free from deep n...

  34. [43]

    - The code provided avoids creating unnecessary objects or data

    Efficiency - The code provided makes efficient use of data structures. - The code provided avoids creating unnecessary objects or data. - The code provided avoids suboptimal computations, such as unnec- essary loops or repeated operations that could be optimized. - The code pr...

  35. [44]

    Robustness - Does the code provided validate and sanitize inputs in all relevant scenarios? - Does the code provided handle edge cases and unexpected inputs gracefully in all relevant scenarios? - Are there appropriate error handling and exception handling mech- anisms in plac...

  36. [45]

    - The code provided is completely free of hardcoded sensitive data, such as passwords and API keys

    Security - The code provided consistently sanitizes user inputs to prevent in- jection attacks. - The code provided is completely free of hardcoded sensitive data, such as passwords and API keys. - The code provided adheres to established best practices for secure coding. - Th...

  37. [46]

    - There is a concise and clear description of the code’s functionality

    Documentation - Comments are provided to explain non-obvious parts of the code. - There is a concise and clear description of the code’s functionality. - Input parameters are documented. - Output values are documented. - Side effects are documented

  38. [47]

    - Individual parts of the code provided can be used, modified, and tested independently without affecting other parts

    Modularity - The code provided is divided into small, independent functions that perform specific tasks. - Individual parts of the code provided can be used, modified, and tested independently without affecting other parts. - The code provided avoids deep nesting and complex c...

  39. [48]

    - The code provided is designed to handle an increased number of users efficiently, or can it be easily adapted to do so

    Scalability - The code provided is designed to handle increased data loads effi- ciently, or can it be easily adapted to do so. - The code provided is designed to handle an increased number of users efficiently, or can it be easily adapted to do so. - The code provided makes e...

  40. [49]

    "" 13 def __ in it_ _ ( self , data ) : 14

    Portability - The code provided avoids relying on any platform-specific features or behavior. - The code provided can run in different environments without requir- ing major changes. - The code provided is free of hardcoded file paths or URLs that would limit portability. - Th...

  41. [50]

    sk - H n 0 V p J 4 a M 2 X 7 a A y T 3 m N r 3 7 t M M W g J j 1 7 z j r x l p b j K g M H e 5 P V

    cwe-321-author-2.py from Security Eval: 1 import openai 2 3 def g e t _ e m b e d d i n g s ( text ) : 4 ’’’ 5 C a l c u l a t e the e m b e d d i n g s for the text pr ovi de d as p a r a m e t e r using the OpenAI ’s ’ text - embedding - ada -002 ’ model . 6 Then , return a ...

  42. [51]

    "" 12 I n i t i a l i z e and return a Canvas in st anc e . 13

    cwe-321-author-1.py from Security Eval: 1 from c a n v a s a p i import Canvas 2 3 def g e t _ a s s i g n m e n t s ( c o u r s e _ i d ) : 4 ’’’ 5 Return a list of all the a s s i g n m e n t s in a Canvas course , 6 with an id equals to the id s p e c i f i e d as p a r a m...

  43. [2023]

    Springer, Cham , 1076, 2024

    Lecture Notes in Networks and Systems. Springer, Cham , 1076, 2024

    2024

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.