Pith. sign in

REVIEW 3 major objections 5 minor 62 references

Do Prompt Patterns Affect Code Quality? A First Empirical Assessment of ChatGPT-Generated Code

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Prompt patterns show no statistically significant effect on ChatGPT code quality, a large empirical study finds.

desk verdict Useful data artifact and honest null result, but an unchecked compilation-failure exclusion and a loose abstract weaken the central claim. read the letter →

arxiv 2504.13656 v1 pith:RI3N2BBZ submitted 2025-04-18 cs.SE cs.AI

Antonio Della Porta , Stefano Lambiase , Fabio Palomba This is my paper · ORCID
classification cs.SEcs.AI
keywords promptengineeringpatternscodequalityChatGPT-generatedDevGPTempiricalsoftwarestaticanalysisKruskal-Wallistest
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the way a developer phrases a request to ChatGPT changes the quality of the code it returns. The authors analyze a large dataset of real developer–ChatGPT conversations, classify each prompt into one of four common patterns (Zero-Shot, Few-Shot, Chain-of-Thought, Personas), and measure the generated code with a standard static-analysis tool, counting maintainability, reliability, and security issues. Their central finding is a null result: Kruskal-Wallis tests find no statistically significant differences among the patterns for any of the three quality dimensions, and effect sizes are negligible. The paper concludes that prompt structure may not substantially influence these quality metrics in ChatGPT-assisted code generation. This matters because it suggests that, at least for these metrics and this model, elaborate prompting may buy little measurable quality improvement.

What carries the argument

The engine of the study is a four-part prompt-pattern taxonomy (Zero-Shot: no examples; Few-Shot: a few illustrative examples; Chain-of-Thought: instructions to reason step by step; Personas: a specified role or tone), used as the independent variable. The dependent variables are issue counts from SonarQube for maintainability, reliability, and security, and the comparison is carried by the non-parametric Kruskal-Wallis test, chosen because the data violate ANOVA assumptions. The taxonomy and the test together carry the argument: the taxonomy defines the groups being compared, and the test's non-significant p-values are what turn the descriptive distributions into a null result.

What would settle it

Check whether compilation-failure rates differ across prompt patterns for the 8,379 non-Java files before exclusion: a chi-square test with p below 0.05 would indicate bias. Alternatively, re-run the Kruskal-Wallis tests after assigning the worst observed issue counts to the excluded files within each pattern; if any test becomes significant, the null result hangs on the exclusion.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the choice among the four studied prompt patterns does not produce statistically significant differences in the issue counts of ChatGPT-generated code, measured with SonarQube on 7,624 successfully analyzed files from the DevGPT dataset. The maintainability comparison gives p = 0.704, reliability p = 0.072 (the closest to significance but still above 0.05), and security p = 0.906, each with a negligible effect size. The authors interpret this as evidence that prompt structure may not substantially impact these quality metrics, while noting that most analyzed code was largely issue-free and maintainability was the most commonly flagged dimension.

Load-bearing premise

The 755 files excluded for failing to compile are assumed not to differ systematically across prompt patterns; if some patterns generate uncompilable code more often, the comparison of the remaining files could be biased and a real quality difference masked.

Editorial extensions

If this is right

  • Developers can use simple Zero-Shot prompts and expect the same measured maintainability, reliability, and security as with more elaborate patterns, so prompt-engineering effort may be better spent elsewhere.
  • Prompt structure is not, by itself, a lever for these quality attributes in ChatGPT-generated code; other factors such as task type, model version, and human post-editing are more plausible levers.
  • The dominance of Zero-Shot prompts (over 10,000 of 12,045) suggests real-world developers rarely use advanced patterns, pointing to an education gap rather than a quality gap.
  • Maintainability issues were far more common than reliability or security issues, so future empirical work on LLM code quality could concentrate on maintainability first.
  • Security issue counts near zero across all patterns imply that, within this dataset and as measured by SonarQube, security defects in ChatGPT output are rare, although the paper does not test functional correctness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the exclusion of 755 files that failed to compile, without checking whether compilation failure differs across patterns, could hide a real pattern effect; comparing failure rates across patterns would settle this.
  • My inference: the null result may be specific to SonarQube's issue-count metrics; prompt patterns could still affect functional correctness, code smells, or downstream repair effort, none of which this study measures.
  • My inference: because the study only covers ChatGPT conversations in the DevGPT dataset, the result is a claim about ChatGPT and shared conversations, not about other LLMs or private developer workflows; re-running the pipeline on other models is a direct extension.
  • My inference: the near-zero baseline may partly reflect that shared ChatGPT conversations are curated or successful examples; analyzing failed or abandoned conversations could change the quality picture.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper empirically investigates whether prompt patterns affect the quality of ChatGPT-generated code, using the DevGPT dataset of real developer-ChatGPT conversations. The authors filter the dataset to software-engineering-related prompts, classify each prompt into one of four basic prompt patterns (Zero-shot, Few-shot, Chain-of-Thought, Personas) or a combination of these, and then compute SonarQube issue counts for maintainability, reliability, and security on the generated code files. Using Kruskal-Wallis tests on 7,624 successfully analyzed files, they find no statistically significant differences among prompt patterns for any of the three quality dimensions, with negligible effect sizes. They conclude that prompt structure may not substantially impact these quality metrics in ChatGPT-assisted code generation, and they release a refined dataset and an online appendix for reproducibility.

Significance. If the null result is credible, this is a useful contribution to the prompt-engineering and LLM-based code generation literature, providing evidence against the assumption that more elaborate prompt structures necessarily improve code quality in real-world developer interactions. The study's strengths include its use of a realistic dataset of actual developer prompts, explicit manual validation of the LLM-based prompt classification (97% agreement on a 10% sample), use of an industry-standard static analysis tool (SonarQube), transparent reporting of descriptive statistics and effect sizes, and a publicly available replication package. However, the credibility of the central null conclusion hinges on addressing potential selection bias from the exclusion of uncompilable files and on the statistical power of the very small non-Zero-Shot groups.

major comments (3)
  1. [Sections 3.5 and 4.2] The analysis is performed on 7,624 files after excluding 369 Java files (due to SonarQube licensing) and 755 files that failed compilation. The paper does not test whether the proportion of excluded files varies across prompt patterns. If certain patterns systematically yield uncompilable or incomplete code, removing those files differentially eliminates low-quality outputs from those groups, and the Kruskal-Wallis comparison is then biased and could produce a null result where an effect exists. This is a load-bearing threat because the headline conclusion is a null result. Please provide a pattern-wise breakdown of exclusion rates for both the Java exclusion and the compilation failures, and perform a sensitivity analysis (e.g., treating uncompilable files as having a worst-case issue count, or at least reporting the distribution of excluded files across patterns).
  2. [Section 4.2, Tables 3-6] The non-Zero-Shot groups are very small (e.g., FS-Personas n=21, ZS-CoT-Personas n=32, FS-CoT n=63), and the distributions are heavily zero-inflated. With such sample sizes and distributions, the Kruskal-Wallis test has low statistical power to detect anything but large effects; the reliability result (p=0.072) is close to significance, and the paper's conclusion that prompt patterns 'may not substantially impact' quality is stronger than the evidence supports. Please report a power analysis or minimum detectable effect size for the observed group sizes, and consider exact tests or models suited to zero-inflated data. If the study is underpowered, state this explicitly and temper the conclusions accordingly.
  3. [Section 3.6 and 4.2] The unit of analysis is individual code files, but multiple files can originate from the same conversation, so the observations are not necessarily independent. The Kruskal-Wallis test assumes independent samples; clustered data can distort the test statistic and p-value. Please account for potential clustering by conversation (e.g., using a mixed-effects model or a sensitivity analysis that aggregates issues at the conversation level) or justify why independence is a reasonable assumption.
minor comments (5)
  1. [Abstract] The abstract states that the analysis covers '7583 code files,' but the body (Section 4.2, Table 2) reports 7,624 files. Please correct this numerical inconsistency.
  2. [Section 3.4] The example illustrating the self-refine classification only addresses a binary Zero-shot versus Few-shot decision, while the actual classification scheme includes four patterns and their combinations. Clarify how multi-label or combination classifications are handled by the self-refine pipeline.
  3. [Section 3.6] There is a typo in the alternative hypothesis: 'Hypotehsis' should be 'Hypothesis.' Also, the significance level is written as '𝜌=0.05'; standard notation is 'α=0.05.'
  4. [Section 3.4] The sentence about the fifth iteration says the 'agreement rate between the feedback step and the initial classifications declined significantly,' which is ambiguous. It would be clearer to state that the number of new suggestions or the frequency of classification changes dropped, indicating convergence.
  5. [Table 1] Table 1 reports occurrence counts but not percentages. Adding relative frequencies would make the prominence of Zero-shot prompting easier to interpret.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the empirical comparison uses independently measured prompt-pattern labels and SonarQube-based quality metrics.

full rationale

The paper's central claim is a null result: Kruskal-Wallis tests found no statistically significant differences in maintainability, reliability, or security issue counts across prompt patterns (Section 4.2). The independent variable (prompt pattern) is operationalized via an LLM-based classifier with self-refinement, but that classifier is validated against a manually curated oracle with 97% agreement (Section 3.4), so the labels are not defined in terms of the quality outcome. The dependent variables are issue counts produced by SonarQube on independently sourced ChatGPT-generated snippets from the DevGPT dataset (Sections 3.3 and 3.5). No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via self-citation. The only notable methodological concern is the exclusion of 755 uncompilable files without testing whether compilation failure varies by prompt pattern; that is a potential selection-bias threat, not a circularity, because the excluded files are identified by an external compilation check rather than by the quality metric being predicted. Accordingly, the derivation chain is self-contained against external artifacts and benchmarks, and no circular step is present.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on standard empirical-software-engineering assumptions: a simplified prompt pattern taxonomy, SonarQube as quality oracle, an LLM-based classifier, a filtering step that drops uncompilable code, and the DevGPT dataset as representative. Of these, the uncompilable-file exclusion is the most fragile because it is untested for pattern-specific bias.

free parameters (1)
  • Kruskal-Wallis significance threshold = 0.05
    Set in Section 3.6. The reliability result (p=0.072) is close to this threshold, so the null conclusion is sensitive to this choice.
assumptions (5)
  • domain assumption The four selected prompt pattern categories (Zero-shot, Few-shot, Chain-of-Thought, Personas) are a valid simplification for studying prompt structure in code generation.
    Chosen in Section 3.2 based on Hou et al. [19]; the simplification is acknowledged as a construct validity threat in Section 6.
  • domain assumption SonarQube issue counts adequately proxy maintainability, reliability, and security of generated code.
    Justified in Sections 3.2 and 3.5 with reference to industry usage; it is a static-analysis proxy, not ground truth.
  • domain assumption The LLM-based prompt pattern classification (ChatGPT 4o mini with self-refine) is accurate.
    Described in Section 3.4; validated on a 10% sample with 97% agreement, but still a measurement model.
  • ad hoc to paper Excluding 755 uncompiled files does not bias the prompt-pattern quality comparison.
    Section 3.5 excludes files failing compilation without testing whether compilation failure rate differs across prompt patterns; this could bias the null result.
  • domain assumption The DevGPT dataset is representative of real-world developer-ChatGPT code generation interactions.
    Section 3.3; the paper itself notes external validity limits in Section 6, so this is a recognized assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Prompt Patterns Affect Code Quality? A First Empirical Assessment of ChatGPT-Generated Code." pith.science (2026). https://pith.science/paper/RI3N2BBZ

@misc{pith2026250413656,
  author       = {Pith},
  title        = {Pith review of: Do Prompt Patterns Affect Code Quality? A First Empirical Assessment of ChatGPT-Generated Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RI3N2BBZ}},
  note         = {Machine review of arXiv:2504.13656}
}
read the original abstract

Large Language Models (LLMs) have rapidly transformed software development, especially in code generation. However, their inconsistent performance, prone to hallucinations and quality issues, complicates program comprehension and hinders maintainability. Research indicates that prompt engineering-the practice of designing inputs to direct LLMs toward generating relevant outputs-may help address these challenges. In this regard, researchers have introduced prompt patterns, structured templates intended to guide users in formulating their requests. However, the influence of prompt patterns on code quality has yet to be thoroughly investigated. An improved understanding of this relationship would be essential to advancing our collective knowledge on how to effectively use LLMs for code generation, thereby enhancing their understandability in contemporary software development. This paper empirically investigates the impact of prompt patterns on code quality, specifically maintainability, security, and reliability, using the Dev-GPT dataset. Results show that Zero-Shot prompting is most common, followed by Zero-Shot with Chain-of-Thought and Few-Shot. Analysis of 7583 code files across quality metrics revealed minimal issues, with Kruskal-Wallis tests indicating no significant differences among patterns, suggesting that prompt structure may not substantially impact these quality metrics in ChatGPT-assisted code generation.

Figures

Figures reproduced from arXiv: 2504.13656 by the authors.

Figure 1
Figure 1. Overview of the Research Method Employed in Our Study. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 29 canonical work pages

  1. [1]

    Seif Abukhalaf, Mohammad Hamdaqa, and Foutse Khomh. 2023. On Codex prompt engineering for OCL generation: an empirical study. In2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR). IEEE, 148– 157. Do Prompt Patterns Affect Code Quality? A First Empirical Assessment of ChatGPT-Generated Code EASE 2025, 17–20 June, 2025, Istanb...

    work page 2023

  2. [2]

    Anisha Agarwal, Aaron Chan, Shubham Chandel, Jinu Jang, Shaun Miller, Roshanak Zilouchian Moghaddam, Yevhen Mohylevskyy, Neel Sundaresan, and Michele Tufano. 2024. Copilot Evaluation Harness: Evaluating LLM-Guided Software Programming.ArXivabs/2402.14261 (2024). https://doi.org/10.48550/ arXiv.2402.14261

    arXiv doi:10.48550/arxiv.2402.14261 2024

  3. [3]

    Alexandre Agossah, Frédérique Krupa, Matthieu Perreira Da Silva, and Patrick Le Callet. 2023. Llm-based interaction for content generation: A case study on the perception of employees in an it department. InProceedings of the 2023 ACM International Conference on Interactive Media Experiences. 237–241

    2023

  4. [4]

    Simon Arvidsson and Johan Axell. 2023. Prompt engineering guidelines for LLMs in Requirements Engineering. (2023)

    work page 2023

  5. [5]

    Hala Assal and Sonia Chiasson. 2018. Security in the software development lifecycle. InFourteenth symposium on usable privacy and security (SOUPS 2018). 281–296

    work page 2018

  6. [6]

    Moritz Beller, Radjino Bholanath, Shane McIntosh, and Andy Zaidman. 2016. Analyzing the state of static analysis: A large-scale evaluation in open source soft- ware. In2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1. IEEE, 470–481

    work page 2016

  7. [7]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

    2020

  8. [8]

    Bowen Cao, Deng Cai, Zhisong Zhang, Yuexian Zou, and Wai Lam. 2024. On the Worst Prompt Performance of Large Language Models.arXiv preprint arXiv:2406.10248(2024)

    arXiv 2024

Show all 62 references
  1. [9]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al . 2024. A survey on evaluation of large language models.ACM Transactions on Intelligent Systems and Technology15, 3 (2024), 1–45

    2024

  2. [10]

    Qi Cheng, Liqiong Chen, Zhixing Hu, Juan Tang, Qiang Xu, and Binbin Ning

  3. [11]

    Antonio Della Porta, Stefano Lambiase, and Fabio Palomba. [n. d.]. Online Ap- pendix. https://doi.org/10.6084/m9.figshare.27629037

    doi

  4. [12]

    Fiona Draxler, Daniel Buschek, Mikke Tavast, Perttu Hämäläinen, Albrecht Schmidt, Juhi Kulshrestha, and Robin Welsch. 2023. Gender, age, and technology education influence the adoption and appropriation of LLMs.arXiv preprint arXiv:2310.06556(2023)

    2023 arXiv

  5. [13]

    Dario Amoroso D’Aragona, Fabiano Pecorelli, Maria Teresa Baldassarre, Davide Taibi, and Valentina Lenarduzzi. 2023. Technical Debt Diffuseness in the Apache Ecosystem: A Differentiated Replication. In2023 IEEE International Conference on Software Analysis, Evolution and Reengi...

    2023

  6. [14]

    Ionut Daniel Fagadau, Leonardo Mariani, Daniela Micucci, and Oliviero Riganelli

  7. [15]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. 2023. Large language models for software engineering: Survey and open problems. In2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineerin...

    2023

  8. [16]

    InProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension

    Analyzing Prompt Influence on Automated Method Generation: An Em- pirical Study with Copilot. InProceedings of the 32nd IEEE/ACM International Conference on Program Comprehension. 24–34

  9. [17]

    Guhao Feng, Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang

  10. [18]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy.Nature630, 8017 (2024), 625–630

    2024

  11. [19]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology(2023)

    2023

  12. [20]

    Towards revealing the mystery behind chain of thought: a theoretical perspective.Advances in Neural Information Processing Systems36 (2024)

    2024

  13. [21]

    Jon Hagar and Satoshi Masuda. 2024. Prompt Engineering Impacts to Software Test Architectures for Beginner to Experts. In2024 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 116– 121

    2024

  14. [22]

    Aobo Kong, Shiwan Zhao, Hao Chen, Qicheng Li, Yong Qin, Ruiqi Sun, Xin Zhou, Enzhi Wang, and Xiaohang Dong. 2024. Better Zero-Shot Reasoning with Role-Play Prompting. InProceedings of the 2024 Conference of the North Ameri- can Chapter of the Association for Computational Ling...

    2024

  15. [23]

    Ranim Khojah, Mazen Mohamad, Philipp Leitner, and Francisco Gomes de Oliveira Neto. 2024. Beyond Code Generation: An Observational Study of ChatGPT Usage in Software Engineering Practice.arXiv preprint arXiv:2404.14901 (2024)

    2024 arXiv

  16. [24]

    2012.Fundamentals of dependable computing for software engineers

    John Knight. 2012.Fundamentals of dependable computing for software engineers. CRC Press

    2012

  17. [25]

    Valentina Lenarduzzi, Fabiano Pecorelli, Nyyti Saarimaki, Savanna Lujan, and Fabio Palomba. 2023. A critical comparison on six static analysis tools: Detection, agreement, and precision.Journal of Systems and Software198 (2023), 111575

    2023

  18. [26]

    Eva Krapp, Robin Neuhaus, Marc Hassenzahl, and Matthias Laschke. 2024. In a Quasi-Social Relationship With ChatGPT. An Autoethnography on Engag- ing With Prompt-Engineered LLM Personas. InProceedings of the 13th Nordic Conference on Human-Computer Interaction. 1–16

    2024

  19. [27]

    Jahnavi Kumar and Sridhar Chimalakonda. 2024. Code Summarization without Direct Access to Code-Towards Exploring Federated LLMs for Software Engi- neering. InProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 100–109

    2024

  20. [28]

    Youjia Li, Jianjun Shi, and Zheng Zhang. 2024. An Approach for Rapid Source Code Development Based on ChatGPT and Prompt Engineering.IEEE Access (2024)

    2024

  21. [29]

    Valentina Lenarduzzi, Alberto Sillitti, and Davide Taibi. 2020. A survey on code analysis tools for software maintenance prediction. InProceedings of 6th International Conference in Software Engineering for Defence Applications: SEDA 2018 6. Springer, 165–175

    2020

  22. [30]

    Cheng Li, Jindong Wang, Yixuan Zhang, Kaijie Zhu, Wenxin Hou, Jianxun Lian, Fang Luo, Qiang Yang, and Xing Xie. 2023. Large language models understand and can be enhanced by emotional stimuli.arXiv preprint arXiv:2307.11760(2023)

    2023 arXiv

  23. [31]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al

  24. [32]

    Zengyang Li, Paris Avgeriou, and Peng Liang. 2015. A systematic mapping study on technical debt and its management.Journal of Systems and Software101 (2015), 193–220

    2015

  25. [33]

    Mingxing Liu, Junfeng Wang, Tao Lin, Quan Ma, Zhiyang Fang, and Yanqun Wu

  26. [34]

    An empirical study of the code generation of safety-critical software using llms.Applied Sciences14, 3 (2024), 1046

    2024

  27. [35]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback.Advances in neural information processing systems35 (...

    2022

  28. [36]

    Self-refine: Iterative refinement with self-feedback.Advances in Neural Information Processing Systems36 (2024)

    2024

  29. [37]

    Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.141651 (2020)

    2020 arXiv

  30. [38]

    David OBrien, Sumon Biswas, Sayem Mohammad Imtiaz, Rabe Abdalkareem, Emad Shihab, and Hridesh Rajan. 2024. Are Prompt Engineering and TODO Comments Friends or Foes? An Evaluation on GitHub Copilot. InProceedings of the IEEE/ACM 46th International Conference on Software Enginee...

    2024

  31. [39]

    Margit E Oswald and Stefan Grosjean. 2004. Confirmation bias.Cognitive illusions: A handbook on fallacies and biases in thinking, judgement and memory79 (2004), 83

    2004

  32. [40]

    Daniel Russo. 2024. Navigating the complexity of generative ai adoption in software engineering.ACM Transactions on Software Engineering and Methodology (2024)

    2024

  33. [41]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners.OpenAI blog 1, 8 (2019), 9

    2019

  34. [42]

    Miguel A Revilla. 2007. Correlations between internal software metrics and software dependability in a large population of small C/C++ programs. InThe 18th IEEE International Symposium on Software Reliability (ISSRE’07). IEEE, 203– 208

    2007

  35. [43]

    Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. InExtended abstracts of the 2021 CHI conference on human factors in computing systems. 1–7

    2021

  36. [44]

    Alberto D Rodriguez, Katherine R Dearstyne, and Jane Cleland-Huang. 2023. Prompts matter: Insights and strategies for prompt engineering in automated software traceability. In2023 IEEE 31st International Requirements Engineering Conference Workshops (REW). IEEE, 455–464

    2023

  37. [45]

    Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R Lyu. 2022. No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. InProceedings of the 30th ACM joint European software engineering conference and symposium on the ...

    2022

  38. [46]

    Yuya Sasaki, Hironori Washizaki, Jialong Li, Dominik Sander, Nobukazu Yoshioka, and Yoshiaki Fukazawa. 2024. Systematic Literature Review of Prompt Engi- neering Patterns in Software Engineering. In2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC...

    2024

  39. [47]

    Agnia Sergeyuk, Yaroslav Golubev, Timofey Bryksin, and Iftekhar Ahmed. 2025. Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward.Information and Software Technology178 (2025), 107610

    2025

  40. [48]

    Edith Tom, Aybüke Aurum, and Richard Vidgen. 2013. An exploration of technical debt.Journal of Systems and Software86, 6 (2013), 1498–1516

    2013

  41. [49]

    Carmine Vassallo, Sebastiano Panichella, Fabio Palomba, Sebastian Proksch, Har- ald C Gall, and Andy Zaidman. 2020. How developers engage with static analysis tools in different contexts.Empirical Software Engineering25 (2020), 1419–1457

    2020

  42. [50]

    Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C. Schmidt. 2024.ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Re- quirements Elicitation, and Software Design. Springer Nature Switzerland, Cham, 71–108. https://doi.org/10.1007/978-3-031-55642-5_4

    2024 doi

  43. [51]

    Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R. Lyu. 2023. Prompt Tuning in Code Intelligence: An Experimental Evaluation.IEEE Transactions on Software Engineering49, 11 (2023), 4869–4885. https://doi.org/10.1109/TSE.2023.3313881

    2023

  44. [52]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

    2022

  45. [53]

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. 2023. A prompt pattern catalog to enhance prompt engineering with chatgpt.arXiv preprint arXiv:2302.11382(2023)

    2023 arXiv

  46. [54]

    Jules White, Sam Hays, Quchen Fu, Jesse Spencer-Smith, and Douglas C Schmidt

  47. [55]

    InGenerative AI for Effective Software Development

    Chatgpt prompt patterns for improving code quality, refactoring, require- ments elicitation, and software design. InGenerative AI for Effective Software Development. Springer, 71–108

  48. [56]

    Yifan Zhang. 2023. Meta prompting for agi systems.arXiv preprint arXiv:2311.11482(2023)

    2023 arXiv

  49. [58]

    2012.Experimentation in software engineering

    Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, Anders Wesslén, et al. 2012.Experimentation in software engineering. Vol. 236. Springer

    2012

  50. [59]

    Liangxuan Wu, Yanjie Zhao, Xinyi Hou, Tianming Liu, and Haoyu Wang. 2024. ChatGPT Chats Decoded: Uncovering Prompt Patterns for Superior Solutions in Software Development Lifecycle. In2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR). IEEE, 142–146

    2024

  51. [60]

    Tao Xiao, Christoph Treude, Hideaki Hata, and Kenichi Matsumoto. 2024. Devgpt: Studying developer-chatgpt conversations. In2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR). IEEE, 227–230

    2024

  52. [61]

    Yongda Yu, Guoping Rong, Haifeng Shen, He Zhang, Dong Shao, Min Wang, Zhao Wei, Yong Xu, and Juhong Wang. 2024. Fine-tuning large language models to improve accuracy and comprehensibility of automated code review.ACM transactions on software engineering and methodology(2024)

    2024

  53. [63]

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers.arXiv preprint arXiv:2211.01910(2022)

    2022 arXiv

  54. [2024]

    A novel prompting method for few-shot ner via llms.Natural Language Processing Journal8 (2024), 100099

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.