{"id":"ed48b15a-c846-4f8b-a155-6eddd3c0de26","arxiv_id":"2504.13656","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Across 7,624 ChatGPT code files from the DevGPT dataset, Kruskal-Wallis tests found no statistically significant differences in maintainability, reliability, or security issues among Zero-Shot, Few-Shot, Chain-of-Thought, and Persona prompts.","lead":"This study tested whether the way a person writes a prompt for ChatGPT can change the quality of the generated code, using thousands of real developer conversations. It found no statistically significant difference in code maintainability, security, or reliability among four common prompt styles.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Selection bias from excluding 755 uncompilable files without testing pattern-dependent compilation failure threatens the null result.","rationale":"The reader's weakest_assumption identifies exactly the same concern I find most load-bearing: the uncompiled files are excluded without checking whether compilation failure correlates with prompt pattern. This is the single greatest threat to the central null claim because it attacks the validity of the comparison itself, not just a secondary detail. The file-count discrepancy (7583 vs 7624) is a real internal inconsistency that should be corrected, but it does not change the statistical conclusion. The abstract's language, suggesting that prompt structure may not substantially impact quality, is a reporting overstatement relative to a failure to reject the null; however, that is a framing issue, not a threat to the validity of the tests. By contrast, if the exclusion is pattern-dependent, the entire RQ2 analysis is comparing non-representative subsets, and the null result could be an artifact of selection. The paper even acknowledges in Section 6 that biases may be introduced through the classification process, but it does not acknowledge this exclusion bias. Given that the online appendix includes the data needed to perform the check, this is an addressable concern. The reader's CONDITIONAL verdict is therefore appropriate: the paper should be accepted only if this analysis is added and the result remains null. My read does not change the reader's verdict, so I recommend UNCHANGED.","tokens_in":17539,"tokens_out":3154,"duration_ms":31591,"concrete_test":"From the online appendix (figshare), reconstruct the data for all 8379 non-Java files, including the 755 compilation-failed files with their flags. Compute the compilation failure rate for each prompt-pattern group in Table 3 and run a chi-square test of independence between prompt pattern and compilation outcome, using Fisher's exact test where cells are small. If the test is significant at p<0.05, the exclusion is non-random and the Kruskal-Wallis analysis on the surviving 7624 files is biased; the authors would need to re-analyze with all files, treating compilation failure as a distinct outcome, or apply a selection correction. If the test is not significant, the selection-bias concern is resolved and the null result can be accepted as is.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a null result: prompt patterns do not significantly affect maintainability, reliability, or security (Section 4.2). That claim is computed only on the 7624 files that compiled after 755 of 8379 non-Java files were excluded for failing compilation (Section 3.5). The paper never tests whether compilation failure is independent of prompt pattern. If some patterns systematically produce uncompilable or incomplete code, dropping those files differentially removes low-quality outputs from those groups. The Kruskal-Wallis tests then compare surviving subpopulations, not the populations that the prompts actually generate. This is not speculative: the near-significant reliability result (p=0.072, Table 6) and the tiny FS-CoT-Personas group (n=7) mean that removing even a few high-issue files could change the outcome. The Threats to Validity section (Section 6) does not mention this exclusion as a selection threat. Because the headline conclusion is that prompt structure does not matter, an uncontrolled selection mechanism in the dependent variable directly threatens that conclusion: it can create a null where an effect exists or hide an effect that does exist. The check is feasible because the excluded files are flagged in the released dataset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically investigates whether prompt patterns affect the quality of ChatGPT-generated code, using the DevGPT dataset of real developer-ChatGPT conversations. The authors filter the dataset to software-engineering-related prompts, classify each prompt into one of four basic prompt patterns (Zero-shot, Few-shot, Chain-of-Thought, Personas) or a combination of these, and then compute SonarQube issue counts for maintainability, reliability, and security on the generated code files. Using Kruskal-Wallis tests on 7,624 successfully analyzed files, they find no statistically significant differences among prompt patterns for any of the three quality dimensions, with negligible effect sizes. They conclude that prompt structure may not substantially impact these quality metrics in ChatGPT-assisted code generation, and they release a refined dataset and an online appendix for reproducibility.","tokens_in":17701,"tokens_out":4506,"duration_ms":42058,"significance":"If the null result is credible, this is a useful contribution to the prompt-engineering and LLM-based code generation literature, providing evidence against the assumption that more elaborate prompt structures necessarily improve code quality in real-world developer interactions. The study's strengths include its use of a realistic dataset of actual developer prompts, explicit manual validation of the LLM-based prompt classification (97% agreement on a 10% sample), use of an industry-standard static analysis tool (SonarQube), transparent reporting of descriptive statistics and effect sizes, and a publicly available replication package. However, the credibility of the central null conclusion hinges on addressing potential selection bias from the exclusion of uncompilable files and on the statistical power of the very small non-Zero-Shot groups.","major_comments":[{"comment":"The analysis is performed on 7,624 files after excluding 369 Java files (due to SonarQube licensing) and 755 files that failed compilation. The paper does not test whether the proportion of excluded files varies across prompt patterns. If certain patterns systematically yield uncompilable or incomplete code, removing those files differentially eliminates low-quality outputs from those groups, and the Kruskal-Wallis comparison is then biased and could produce a null result where an effect exists. This is a load-bearing threat because the headline conclusion is a null result. Please provide a pattern-wise breakdown of exclusion rates for both the Java exclusion and the compilation failures, and perform a sensitivity analysis (e.g., treating uncompilable files as having a worst-case issue count, or at least reporting the distribution of excluded files across patterns).","section":"Sections 3.5 and 4.2"},{"comment":"The non-Zero-Shot groups are very small (e.g., FS-Personas n=21, ZS-CoT-Personas n=32, FS-CoT n=63), and the distributions are heavily zero-inflated. With such sample sizes and distributions, the Kruskal-Wallis test has low statistical power to detect anything but large effects; the reliability result (p=0.072) is close to significance, and the paper's conclusion that prompt patterns 'may not substantially impact' quality is stronger than the evidence supports. Please report a power analysis or minimum detectable effect size for the observed group sizes, and consider exact tests or models suited to zero-inflated data. If the study is underpowered, state this explicitly and temper the conclusions accordingly.","section":"Section 4.2, Tables 3-6"},{"comment":"The unit of analysis is individual code files, but multiple files can originate from the same conversation, so the observations are not necessarily independent. The Kruskal-Wallis test assumes independent samples; clustered data can distort the test statistic and p-value. Please account for potential clustering by conversation (e.g., using a mixed-effects model or a sensitivity analysis that aggregates issues at the conversation level) or justify why independence is a reasonable assumption.","section":"Section 3.6 and 4.2"}],"minor_comments":[{"comment":"The abstract states that the analysis covers '7583 code files,' but the body (Section 4.2, Table 2) reports 7,624 files. Please correct this numerical inconsistency.","section":"Abstract"},{"comment":"The example illustrating the self-refine classification only addresses a binary Zero-shot versus Few-shot decision, while the actual classification scheme includes four patterns and their combinations. Clarify how multi-label or combination classifications are handled by the self-refine pipeline.","section":"Section 3.4"},{"comment":"There is a typo in the alternative hypothesis: 'Hypotehsis' should be 'Hypothesis.' Also, the significance level is written as '𝜌=0.05'; standard notation is 'α=0.05.'","section":"Section 3.6"},{"comment":"The sentence about the fifth iteration says the 'agreement rate between the feedback step and the initial classifications declined significantly,' which is ambiguous. It would be clearer to state that the number of new suggestions or the frequency of classification changes dropped, indicating convergence.","section":"Section 3.4"},{"comment":"Table 1 reports occurrence counts but not percentages. Adding relative frequencies would make the prominence of Zero-shot prompting easier to interpret.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is generally well-written, transparent, and provides a valuable replication package, but the selection-bias issue from excluding uncompilable files is a genuine threat to the central null claim. The requested sensitivity analysis is feasible with the released data and should be feasible to address in a revision. I am recommending major revision rather than rejection because the core methodology is sound and the issue is fixable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a transparent, careful empirical study, but its central null result is weaker than the abstract lets on. The main concern is real: they excluded 755 files that failed compilation and never tested whether compilation failure correlates with prompt pattern. If some patterns generate uncompilable code more often, dropping those files differentially removes bad outcomes and can create a null where an effect exists. That selection threat is absent from Section 6. The stress-test note holds up.\n\nWhat's new and good: they extend Wu et al. by measuring SonarQube maintainability, reliability, and security and by using Hou et al.'s validated pattern taxonomy. The cleaned DevGPT snapshot with pattern labels and issue counts is a useful artifact, and the manual validation of the LLM classifier (97% agreement on a 10% sample) is more than most papers do. The Kruskal-Wallis analysis with effect sizes is appropriate, and the online appendix is a real plus. Citations to prior work look fair.\n\nSoft spots, in order. First, the compilation-exclusion issue above; it is a genuine threat to the null conclusion, and the fix is easy: report compile-failure rates by pattern and adjust the analysis or interpretation. Second, the groups are extremely unbalanced—6,534 ZS files versus 21 FS-Personas and 7 FS-CoT-Personas. The test is underpowered for rare patterns, so 'no significant difference' is not 'no effect.' The reliability p=0.072 is consistent with a small effect. Third, the abstract says 7,583 files but the body and tables say 7,624. Fourth, the abstract's 'may not substantially impact' overstates a failure to reject the null.\n\nThis paper is for researchers working on prompt engineering and LLM code generation, especially those mining DevGPT. It deserves a serious referee, but the compilation-failure analysis should be a required revision, and the abstract/body discrepancy fixed. I'd conditionally accept it after those changes.","headline":"Useful data artifact and honest null result, but an unchecked compilation-failure exclusion and a loose abstract weaken the central claim.","tokens_in":18262,"tokens_out":3549,"would_cite":true,"duration_ms":32667,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt patterns show no statistically significant effect on ChatGPT code quality, a large empirical study finds.","keywords":["prompt engineering","prompt patterns","code quality","ChatGPT-generated code","DevGPT","empirical software engineering","static analysis","Kruskal-Wallis test"],"falsifier":"Check whether compilation-failure rates differ across prompt patterns for the 8,379 non-Java files before exclusion: a chi-square test with p below 0.05 would indicate bias. Alternatively, re-run the Kruskal-Wallis tests after assigning the worst observed issue counts to the excluded files within each pattern; if any test becomes significant, the null result hangs on the exclusion.","tokens_in":17291,"feed_emoji":"🤖","tokens_out":10361,"duration_ms":81282,"temperature":0.7,"pith_summary":"This paper asks whether the way a developer phrases a request to ChatGPT changes the quality of the code it returns. The authors analyze a large dataset of real developer–ChatGPT conversations, classify each prompt into one of four common patterns (Zero-Shot, Few-Shot, Chain-of-Thought, Personas), and measure the generated code with a standard static-analysis tool, counting maintainability, reliability, and security issues. Their central finding is a null result: Kruskal-Wallis tests find no statistically significant differences among the patterns for any of the three quality dimensions, and effect sizes are negligible. The paper concludes that prompt structure may not substantially influence these quality metrics in ChatGPT-assisted code generation. This matters because it suggests that, at least for these metrics and this model, elaborate prompting may buy little measurable quality improvement.","feed_headline":"7,624 ChatGPT files show prompt patterns don't change code quality","feed_subtitle":"Detailed patterns like Chain-of-Thought matched plain prompts; simple prompts may suffice.","key_machinery":"The engine of the study is a four-part prompt-pattern taxonomy (Zero-Shot: no examples; Few-Shot: a few illustrative examples; Chain-of-Thought: instructions to reason step by step; Personas: a specified role or tone), used as the independent variable. The dependent variables are issue counts from SonarQube for maintainability, reliability, and security, and the comparison is carried by the non-parametric Kruskal-Wallis test, chosen because the data violate ANOVA assumptions. The taxonomy and the test together carry the argument: the taxonomy defines the groups being compared, and the test's non-significant p-values are what turn the descriptive distributions into a null result.","core_discovery":"On the paper's own terms, the discovery is that the choice among the four studied prompt patterns does not produce statistically significant differences in the issue counts of ChatGPT-generated code, measured with SonarQube on 7,624 successfully analyzed files from the DevGPT dataset. The maintainability comparison gives p = 0.704, reliability p = 0.072 (the closest to significance but still above 0.05), and security p = 0.906, each with a negligible effect size. The authors interpret this as evidence that prompt structure may not substantially impact these quality metrics, while noting that most analyzed code was largely issue-free and maintainability was the most commonly flagged dimension.","pith_inferences":["My inference: the exclusion of 755 files that failed to compile, without checking whether compilation failure differs across patterns, could hide a real pattern effect; comparing failure rates across patterns would settle this.","My inference: the null result may be specific to SonarQube's issue-count metrics; prompt patterns could still affect functional correctness, code smells, or downstream repair effort, none of which this study measures.","My inference: because the study only covers ChatGPT conversations in the DevGPT dataset, the result is a claim about ChatGPT and shared conversations, not about other LLMs or private developer workflows; re-running the pipeline on other models is a direct extension.","My inference: the near-zero baseline may partly reflect that shared ChatGPT conversations are curated or successful examples; analyzing failed or abandoned conversations could change the quality picture."],"forward_implications":["Developers can use simple Zero-Shot prompts and expect the same measured maintainability, reliability, and security as with more elaborate patterns, so prompt-engineering effort may be better spent elsewhere.","Prompt structure is not, by itself, a lever for these quality attributes in ChatGPT-generated code; other factors such as task type, model version, and human post-editing are more plausible levers.","The dominance of Zero-Shot prompts (over 10,000 of 12,045) suggests real-world developers rarely use advanced patterns, pointing to an education gap rather than a quality gap.","Maintainability issues were far more common than reliability or security issues, so future empirical work on LLM code quality could concentrate on maintainability first.","Security issue counts near zero across all patterns imply that, within this dataset and as measured by SonarQube, security defects in ChatGPT output are rare, although the paper does not test functional correctness."],"supporting_citations":[{"why":"Provides the DevGPT dataset of developer–ChatGPT conversations that the whole study analyzes.","marker":"[54]"},{"why":"Supplies the four basic prompt patterns (Zero-Shot, Few-Shot, Chain-of-Thought, Personas) that the study classifies and compares.","marker":"[19]"},{"why":"The closest prior work on prompt patterns and software quality; this study extends it by focusing on code generation and quality attributes.","marker":"[53]"},{"why":"The self-refine approach used to build the LLM-based prompt-pattern classifier.","marker":"[31]"},{"why":"The alternative prompt-pattern catalog used by the closest prior work, which the study contrasts when selecting its own categories.","marker":"[48]"},{"why":"One of the studies supporting the working hypothesis that prompt design can affect LLM performance, motivating the investigation.","marker":"[46]"},{"why":"The prior work on ChatGPT prompt patterns for code quality that this study tests empirically; cited as the hypothesis source.","marker":"[50]"}],"fun_headline_variants":["No significant quality differences from prompt patterns in ChatGPT","Prompt patterns don't change code quality in ChatGPT outputs","Study: Prompt choice doesn't affect ChatGPT code quality","Fancy prompts don't improve ChatGPT-generated code quality","7,624 files confirm prompt patterns don't alter code quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 755 files excluded for failing to compile are assumed not to differ systematically across prompt patterns; if some patterns generate uncompilable code more often, the comparison of the remaining files could be biased and a real quality difference masked.","fun_headline_variants_meta":{"raw":{"variants":["No significant quality differences from prompt patterns in ChatGPT","Prompt patterns don't change code quality in ChatGPT outputs","Study: Prompt choice doesn't affect ChatGPT code quality","Fancy prompts don't improve ChatGPT-generated code quality","7,624 files confirm prompt patterns don't alter code quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":2989,"prompt_tokens":887,"completion_tokens":2102,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":2025}},"tokens_in":503,"tokens_out":2102,"duration_ms":13034,"temperature":1.0,"reasoning_tokens":2025,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:02:29.032024+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check whether compilation-failure rates differ across prompt patterns for the 8,379 non-Java files before exclusion: a chi-square test with p below 0.05 would indicate bias. Alternatively, re-run the Kruskal-Wallis tests after assigning the worst observed issue counts to the excluded files within each pattern; if any test becomes significant, the null result hangs on the exclusion.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the studies supporting the working hypothesis that prompt design can affect LLM performance, motivating the investigation."}],"review_version":1}