REVIEW 7 cited by
An Empirical Study of the Non-determinism of ChatGPT in Code Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
There has been a recent explosion of research on Large Language Models (LLMs) for software engineering tasks, in particular code generation. However, results from LLMs can be highly unstable; nondeterministically returning very different codes for the same prompt. Non-determinism is a potential menace to scientific conclusion validity. When non-determinism is high, scientific conclusions simply cannot be relied upon unless researchers change their behaviour to control for it in their empirical analyses. This paper conducts an empirical study to demonstrate that non-determinism is, indeed, high, thereby underlining the need for this behavioural change. We choose to study ChatGPT because it is already highly prevalent in the code generation research literature. We report results from a study of 829 code generation problems from three code generation benchmarks (i.e., CodeContests, APPS, and HumanEval). Our results reveal high degrees of non-determinism: the ratio of coding tasks with zero equal test output across different requests is 75.76%, 51.00%, and 47.56% for CodeContests, APPS, and HumanEval, respectively. In addition, we find that setting the temperature to 0 does not guarantee determinism in code generation, although it indeed brings less non-determinism than the default configuration (temperature=1). These results confirm that there is, currently, a significant threat to scientific conclusion validity. In order to put LLM-based research on firmer scientific foundations, researchers need to take into account non-determinism in drawing their conclusions.
Forward citations
Cited by 7 Pith papers
-
Synthetic Heuristic Evaluation: A Comparison between AI- and Human-Powered Usability Evaluation
An LLM prompted to conduct heuristic evaluation reported more usability issues on two apps than five human experts, but the ground truth included the LLM's own findings.
-
Fault Localisation and Repair for DL Systems: An Empirical Study with LLMs
LLMs, especially GPT-4, outperform existing fault localisation and repair tools for deep learning models in accuracy, speed, and stability.
-
Knowledge-Enhanced Program Repair for Data Science Code
DSrepair combines a knowledge graph of data science APIs with AST-level bug localization to repair LLM-generated code, fixing more DS-1000 tasks than five baseline repair methods.
-
Theoretical Benefit and Limitation of Diffusion Language Model
Masked diffusion language models have a metric-dependent efficiency tradeoff: near-optimal perplexity in constant steps, but sequence-level correctness needs linearly many steps in the worst case.
-
Where Does the Noise Come From? A Variance-Components Decomposition of Non-Determinism in LLM Brand Answers
For AI brand answers, query language is the biggest source of answer noise (26.5%) while brand identity is tiny (1.5%), so repeating the same prompt is the least efficient way to spend a query budget.
-
Leveraging Large Language Models for Command Injection Vulnerability Analysis in Python: An Empirical Study on Popular Open-Source Projects
GPT-4 detected command injection in popular Python projects with 75.5% accuracy and F1 74.5%, missing 15 real cases and beating Bandit on precision.
-
On Iterative Evaluation and Enhancement of Code Quality Using GPT-4o
CodeQUEST, a GPT-4o-based evaluator-optimizer loop, reports a 52.6% mean relative improvement in code quality, but the improvement is measured by the same model that enforces monotonic score increases.
Discussion (0). Continue with ORCID to comment.