Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Characterizing Readability Issue Patterns and the Role of Prompt Design in LLM-Generated Code

T0 review · 4 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current LLMs produce code whose measured readability is at least comparable to, and often higher than, human-written code, but the failures are characteristically AI-shaped, and prompt design is a real yet bounded lever.

desk verdict Substantial empirical study of LLM code readability whose headline parity claim rests on an unvalidated proxy model; the qualitative issue patterns and prompt experiments are worth engaging but need revision. read the letter →

arxiv 2605.13280 v2 pith:LIYRG6S4 submitted 2026-05-13 cs.SE cs.AI

Hengzhi Ye , Fengyuan Ran , Weiwei Xu , Minghui Zhou This is my paper · ORCID
classification cs.SEcs.AI
keywords codereadabilityLLM-generatedpromptengineeringassessmentmodelissuepatternsAI-assistedprogrammingempiricalsoftwaretechnicaldebt
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to determine whether LLM-generated code is as readable as human-written code and whether prompt design can be used to improve it. The authors build a multi-feature readability model, generate code from five LLMs across nearly six thousand real-world and competitive programming tasks, and compare against human implementations. Their central claim is that LLM code is statistically at least as readable as human code, but it fails in distinct, AI-specific patterns—over-engineered logic, redundant comments, and unexplained APIs. They also find that function signatures, constraints, and style descriptions matter most in prompts, but that prompt engineering alone is a weak lever. If correct, the work validates AI-assisted coding from a maintainability perspective while exposing a latent readability debt that future tooling must address.

What carries the argument

The central instrument is an integrated readability model: a 61-dimensional feature vector combining textual/lexical, structural/formatting, information-theoretic (program), and visual/geometric metrics, reduced by sequential forward selection and scored by an L2-regularized logistic regression trained on 360 human-annotated snippets. The same model supplies every readability score in the paper. A secondary mechanism is the prompt vector—a seven-bit encoding of which design dimensions a prompt includes—which lets the authors isolate the effect of each prompt dimension through random-forest importance, permutation tests, and ablations.

What would settle it

Recruit a pre-registered panel of experienced developers to rate a random sample of matched LLM/human code pairs (blind to authorship) and compare their rankings to the model's scores. If human raters do not place LLM code at or above human code, or if the model's scores fail to track ratings on individual pairs, the central parity claim fails. A simpler check: strip comments and re-run the model to see whether it rewards surface regularity that human readers may not.

Watch

Extended reading notes

Core claim

The paper claims that, across nearly six thousand matched coding tasks, code generated by five current LLMs receives readability scores statistically higher than human-written code (p<0.001, moderate effect size), and is therefore at least comparable in practical terms. That aggregate parity masks a divergence: poorly rated human code is dominated by missing comments and inconsistent style, while poorly rated LLM code is dominated by excessive complexity and redundant comments, plus three newly identified AI-specific patterns—unknown API usage, redundant variables, and overblanking. On prompts, function signatures, constraints, and style descriptions are the most influential dimensions, yet

Load-bearing premise

The entire argument rests on treating the readability model's score—trained on 360 snippets from a different distribution and never checked against human readers for LLM-generated or open-source code—as human-perceived readability in the target setting.

Editorial extensions

If this is right

  • LLM-generated code can be adopted into workflows without an overall readability penalty; the risk is concentrated in specific, recognizable failure modes rather than average quality.
  • Existing code-review and static-analysis tools, tuned to human-typical issues, need new detectors for excessive complexity, redundant comments, unknown APIs, redundant variables, and overblanking.
  • Adding function signatures, constraints, and style descriptions to prompts is a cheap, partially effective readability fix, but the low explained variance means prompting alone cannot reliably engineer readable code.
  • Readability should join functional correctness as a first-class evaluation and optimization target for code LLMs.
  • Human-written code is more consistent; teams relying on generated code may need style normalization to reduce readability variance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the three new AI-specific patterns are detector-ready: unknown API usage can be flagged by checking imports and comments, redundant variables by dataflow analysis, and overblanking by layout rules. Building these detectors is a direct, testable next step.
  • A human-rating validation study on the same matched pairs would be the natural stress test; if readers do not perceive the parity the model reports, the 'readable as human code' conclusion is an artifact of the proxy.
  • The commission-versus-omission asymmetry—LLMs add clutter, humans leave gaps—may extend beyond readability to other non-functional qualities like security and performance, suggesting AI output often needs subtractive refactoring.
  • Because the prompt study is restricted to single-turn interactions, iterative or conversational prompting—where a developer asks for simplifications—could have a larger effect; testing readability under multi-turn refinement is an open extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper investigates whether LLM-generated Python code is as readable as human-written code, what readability issues each exhibits, and how prompt design affects generated-code readability. The authors build a 61-feature readability model from four established feature families (TF, BWF, PF, DF), train it on the Dorn dataset, and apply it to 5,869 WoC/LeetCode prompt-code pairs (the abstract says 2,735). They report that LLM-generated code is 'at least comparable' and, in the main analysis, statistically higher in readability than human code (Wilcoxon p<0.001, r=0.398). A thematic analysis of 500 pairs identifies distinct issue patterns, with LLM code more prone to Excessive Complexity, Redundant Comments, Unknown APIs, Redundant Variables, and Overblanking. A controlled prompt experiment with 5,248 prompts across seven dimensions finds function signature, constraints, and style description to be the most influential prompt dimensions, while overall prompt impact is limited (RF R²<0.3).

Significance. If the central quantitative result holds, the paper would provide a useful large-scale baseline for an important and understudied non-functional property of LLM-generated code. The mixed-method design is a genuine strength: the RQ2 thematic analysis uses human annotation with substantial inter-rater agreement (Cohen's Kappa 0.87/0.81), and the prompt-variant experiment is carefully controlled. The scale of the benchmark and the proposed readability issue taxonomy are valuable contributions independent of the exact numerical scores. However, the RQ1 and RQ3 conclusions rest entirely on the validity of the automated readability model, and the manuscript does not validate that model on the target distribution or against human readers of the target code. This is a load-bearing construct-validity gap, not a circularity problem: the model was trained on an external dataset, but its use to compare LLM and human code on a different distribution requires domain calibration that is not provided.

major comments (4)
  1. [§4.3, §5, Appendix A.2] The central RQ1 claim—LLM code is 'statistically higher' in readability (p<0.001, r=0.398 in §5)—is computed from a logistic-regression readability model trained on the 360-snippet Dorn dataset (§4.3) and applied to 5,869 WoC/LeetCode snippets without any validation on the target distribution or against human judgments on that distribution. Appendix A.2 (Eq. A.31) shows the score is a weighted linear combination of features whose weights are learned on Dorn. The features heavily reward regular formatting, alignment, and low layout entropy, which are properties LLM output may exhibit for stylistic rather than human-comprehension reasons. A concrete fix is to collect human readability ratings on a held-out sample of WoC/LeetCode pairs and report correlation/agreement between the model and human judgments, or otherwise demonstrate that the model's preferences transfer. Without this, the qua
  2. [Appendix A.2, Table 4, Figure 2] There is an internal inconsistency in the definition of the readability score. Eq. A.35 defines r(s) as a logistic-sigmoid probability in [0,1], and the text says this 'is used as the final readability prediction.' But Table 4 reports average scores such as 2.15, 1.29, and 0.77, and Figure 2 shows negative scores and values outside [0,1]. If the actual score used in §5 is the raw linear combination before the sigmoid, the manuscript needs to say so explicitly and consistently; if it is the sigmoid probability, the tables and figures are incorrect. This matters because all RQ1/RQ3 statistics are computed on this score, and the reader cannot infer which quantity was tested.
  3. [§3.4, Table 5, Table 6] The issue-pattern results in RQ2 are not auditable as rendered. Table 6 is garbled: the row values for Human_bad, LLM_bad, and Total are run together (e.g., 'Human_bad60 35841 24 22 7 6 3'), making it impossible to verify which numbers correspond to DC, RC, IS, EC, PS, PN, MV, CD, and HC. Table 5 also has an unexplained arithmetic issue: the per-column totals match (72+60=132 etc.), but the row sums are 214 for Human_bad and 169 for LLM_bad, totaling 383, not the 500 sampled pairs described in §3.4. The manuscript should clarify the unit of analysis, how ties/exclusions were handled, and provide the full frequency table for each issue pattern by group.
  4. [§3.5, Table 7] The statistical thresholds for RQ3 are inconsistent. Section 3.5 states that t-tests and permutation tests use p<0.1 to determine relevance, but Table 7's footnote says 'Bold values indicate statistical significance (p<0.05).' The reported p-values happen to be far below both thresholds for the significant dimensions, but the protocol should be stated consistently and the threshold choice justified, especially because RQ3's conclusion about which dimensions are 'most influential' is based on this test.
minor comments (6)
  1. [Abstract vs. §3.1/§3.2.1] The abstract reports '2,735 scenarios' while the full text and §3.1/§3.2.1 consistently report 5,869 effective samples and 3,000 WoC pairs plus 2,869 LeetCode pairs. This is a central factual inconsistency that should be corrected.
  2. [Table 3] The All-features row reads '2577.5% 83.8%'—likely '25 77.5%' with a missing separator. This should be formatted properly.
  3. [§4.3] Typo: 'Ddorn dataset' should be 'Dorn dataset'.
  4. [Table 4] The model is referred to as 'Llama 3.1' in §3.3 and elsewhere, but Table 4 lists 'Llama 3'. Please make consistent.
  5. [§7] The random-forest R² is reported only as '<0.3'. Reporting the exact value and confidence interval would clarify how limited the prompt effects actually are, especially since the R² is used to support the 'overall impact remains limited' claim.
  6. [§3.5] The description of permutation testing says the binary labels of the tested dimension are shuffled, but permutation importance typically shuffles the feature values. Clarify whether the shuffling was applied to the feature column or to the target; the current wording is ambiguous.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the readability proxy is independently trained on Dorn and applied out-of-sample; RQ2 uses human annotation; RQ3 uses controlled prompt experiments.

full rationale

The central RQ1 comparison is not circular: the readability model is trained on the Dorn dataset (Section 4.3: 'we conducted experiments on the Ddorn dataset... 360 code snippets') and then applied to 5,869 WoC/LeetCode pairs (Section 5). No parameter of that model is fitted to the LLM-vs-human difference it is used to estimate; the comparison is an out-of-sample application of an independently labeled proxy. The proxy's possible mismatch with the target distribution is an external-validity/construct-validity limitation, not a circular reduction—the authors themselves flag it in Section 8.2.1 ('Although our model integrates dimensions from validated prior work, it may not be exhaustive, as individual developer preferences vary'). RQ2 issue patterns come from independent human annotation with reported Cohen's kappa (Section 3.4), not from the model's fitted weights; using the same four families as annotation dimensions is a categorization choice, not a derivation from the outcome. RQ3 is a controlled prompt-variant experiment (Set B, 5,248 prompts) with random forest importance, t-tests, permutation tests, and ablation; no parameter is fitted to the claimed conclusion. The only self-citation ([99], Licoeval) is a related-work reference about license compliance and is not load-bearing. Therefore no specific equation or fitted value reduces to the target claim, and no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new entities are postulated. 'Readability debt' and 'latent technical debt' are interpretive metaphors for observed patterns, not independent mechanisms with falsifiable handles. The central load-bearing parameters are the readability-model weights and the prompt-analysis hyperparameters.

free parameters (3)
  • Readability model weights (logistic regression w, b) and SFS-selected features = Not reported; all-features model uses 25 selected features with 77.5% accuracy and 83.8% AUC on Dorn
    The logistic regression trained on Dorn labels produces the scores used for all LLM-vs-human comparisons and prompt-effect analyses. The weights are fitted to data and are not reported or validated on the target distribution.
  • Random forest hyperparameters (n_estimators, max_depth, min_samples_split) = 100, 15, 5
    Chosen by the authors for the RQ3 feature-importance analysis; affects the reported importance ranking and R^2.
  • Permutation importance significance threshold = p < 0.1
    Used in Section 3.5 to declare prompt dimensions relevant; a looser threshold than conventional p<0.05 and inconsistent with Table 7's bolded p<0.05.
assumptions (4)
  • domain assumption Code from WoC (Oct 2021 snapshot) and LeetCode solutions before 2022 are human-written.
    Section 3.1 justifies this by the ChatGPT release date, but Codex and GitHub Copilot were already available in 2021, so the baseline may include AI-assisted code.
  • domain assumption The readability model trained on Dorn (360 snippets) yields valid readability measurements for WoC, LeetCode, and LLM-generated code.
    Section 4.3 reports only in-domain 10-fold cross-validation; no external validation on the target distributions or against human ratings.
  • domain assumption The 61 integrated features (TF/BWF/PF/DF) capture code readability, and the logistic-regression probability is a meaningful continuous readability score.
    Section 4 assumes the established metric families are valid and that the model output can be treated as a direct readability measurement.
  • domain assumption Annotator-identified issue patterns from 1,000 sampled pairs (500 WoC + 500 LeetCode) are representative of the full 5,869 scenarios and of LLM code generally.
    Section 3.4 samples 500 pairs per source; generalization beyond the sample and beyond one selected model is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Characterizing Readability Issue Patterns and the Role of Prompt Design in LLM-Generated Code." pith.science (2026). https://pith.science/paper/LIYRG6S4

@misc{pith2026260513280,
  author       = {Pith},
  title        = {Pith review of: Characterizing Readability Issue Patterns and the Role of Prompt Design in LLM-Generated Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LIYRG6S4}},
  note         = {Machine review of arXiv:2605.13280}
}
read the original abstract

Large Language Models (LLMs) are increasingly changing how code is produced, but generated code still requires human review and validation before it can be adapted or integrated into real-world projects. This makes the readability of LLM-generated code a critical concern. Existing studies have mainly focused on functional correctness and task completion of LLM generated code, leaving open questions about whether it is readable, how its readability fails, and to what extent prompt design can improve it. We therefore investigate the readability of LLM-generated code. We first construct a readability assessment model that integrates textual, structural, program, and visual features. Using this model, we compare human-written code with code generated by representative frontier LLMs across 2,735 scenarios derived from World of Code (WoC) and LeetCode. We further characterize readability issue patterns using thematic analysis and examine prompt design associations through controlled prompt-variant experiments. Our results show that current LLMs produce code that is comparable to human-written code in overall readability. However, this aggregate similarity masks distinct issue patterns, including excessive complexity, redundant comments, and unknown API usage. Prompt analysis further shows that function signature, constraints, and style description are most strongly associated with code readability, although the overall role of prompt design remains bounded. These findings reveal latent readability debt in AI-assisted programming, identify prompt design as a lightweight starting point for improving generated-code readability, and motivate automated support for detecting and mitigating readability issues in future development workflows.

Figures

Figures reproduced from arXiv: 2605.13280 by the authors.

Figure 1
Figure 1. Overview of the methodology cross-verification phase. Any discrepancies were resolved through iterative discussion to a consensus on all final expressions. We fi￾nally got a comprehensive set of 5,869 prompts reflecting diverse real-world development tasks. 3.2.2 Set B: Controlled Experimental Prompts (RQ3). Analyzing how specific prompt dimensions affect readability requires a rigor￾ous control of variables. The co… view at source ↗
Figure 2
Figure 2. Comparison of the readability scores of LLM [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Readability score distribution of code generated by [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Feature importance of the Random Forest regres [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Hitchhiker's Guide to Monoculture

    cs.CY 2026-07 conditional novelty 6.0 of 10

    Code written for Kaggle contests has become substantially more similar in syntax since ChatGPT, converging heavily on seed 42, while the semantic diversity of solution approaches has not declined.

Reference graph

Works this paper leans on

114 extracted references · 17 linked inside Pith · cited by 1 Pith paper

  1. [1]

    [n. d.]. LeetCode. https://leetcode.com/problemset/

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shya- mal Anadkat, et al. 2023. Gpt-4 technical report.arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  3. [3]

    Duaa Alawad, Manisha Panta, Minhaz Zibran, and Md Rakibul Islam. 2019. An empirical study of the relationships between code readability and software complexity.arXiv preprint arXiv:1909.01760(2019)

    arXiv 2019

  4. [4]

    André Altmann, Laura Toloşi, Oliver Sander, and Thomas Lengauer. 2010. Per- mutation importance: a corrected feature importance measure.Bioinformatics 26, 10 (2010), 1340–1347

    2010

  5. [5]

    Anthropic. [n. d.]. Claude. https://www.anthropic.com/claude/

  6. [6]

    Anysphere. [n. d.]. Cursor. https://www.cursor.com/

  7. [7]

    Ashley. 2024. What is .cursorrule and How to Use It Effectively. https: //medium.com/towards-agi/what-are-cursor-rules-and-how-to-use-them- ec558468d139

    2024

  8. [8]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732(2021)

    arXiv 2021

Show all 114 references
  1. [9]

    Alberto Bacchelli and Christian Bird. 2013. Expectations, outcomes, and chal- lenges of modern code review. In2013 35th International Conference on Software Engineering (ICSE). IEEE, 712–721

    2013

  2. [10]

    2006.Pattern recognition and machine learning

    Christopher M Bishop and Nasser M Nasrabadi. 2006.Pattern recognition and machine learning. Vol. 4. Springer

    2006

  3. [11]

    O’Reilly Media, Inc

    Dustin Boswell and Trevor Foucher. 2011.The art of readable code. " O’Reilly Media, Inc. "

    2011

  4. [12]

    Leo Breiman. 2001. Random forests.Machine learning45, 1 (2001), 5–32

    2001

  5. [13]

    Kiran Busch, Alexander Rochlitzer, Diana Sola, and Henrik Leopold. 2023. Just tell me: Prompt engineering in business process management. InInternational Conference on Business Process Modeling, Development and Support. Springer, 3–11

    2023

  6. [14]

    Raymond PL Buse and Westley R Weimer. 2008. A metric for software readability. InProceedings of the 2008 international symposium on Software testing and analysis. 121–130

    2008

  7. [15]

    Raymond PL Buse and Westley R Weimer. 2009. Learning a metric for code readability.IEEE Transactions on software engineering36, 4 (2009), 546–558

    2009

  8. [16]

    Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H Pater- son, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye movements in code reading: Relaxing the linear order. In2015 ieee 23rd international conference on program comprehension. IEEE, 255–265

    2015

  9. [17]

    Mark Chen. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    2021 arXiv

  10. [18]

    Tristan Coignion, Clément Quinton, and Romain Rouvoy. 2024. A performance study of llm-generated code on leetcode. InProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 79–89

    2024

  11. [19]

    Martha E Crosby and Jan Stelovsky. 2002. How do we read algorithms? A case study.Computer23, 1 (2002), 25–35

    2002

  12. [20]

    Pablo Roberto Fernandes de Oliveira, Rohit Gheyi, José Aldo Silva da Costa, and Márcio Ribeiro. 2024. Assessing Python Style Guides: An Eye-Tracking Study with Novice Developers. InSimpósio Brasileiro de Engenharia de Software (SBES). SBC, 136–146

    2024

  13. [21]

    George Digkas, Alexander Chatzigeorgiou, Apostolos Ampatzoglou, and Paris Avgeriou. 2020. Can clean new code reduce technical debt density?IEEE Transactions on Software Engineering48, 5 (2020), 1705–1721

    2020

  14. [22]

    Xi Ding, Rui Peng, Xiangping Chen, Yuan Huang, Jing Bian, and Zibin Zheng

  15. [23]

    2011.Software development and professional practice

    John Dooley and John Zukowski. 2011.Software development and professional practice. Springer

    2011

  16. [24]

    Jonathan Dorn. 2012. A general software readability model.MCS Thesis available at (web.eecs.umich.edu/~weimerw/students/dorn-mcs-paper.pdf)5 (2012), 11–14

    2012

  17. [25]

    James L Elshoff and Michael Marcotty. 1982. Improving computer program readability to aid modification.Commun. ACM25, 8 (1982), 512–521

    1982

  18. [26]

    Tom Fawcett. 2006. An introduction to ROC analysis.Pattern recognition letters 27, 8 (2006), 861–874

    2006

  19. [27]

    Louie Giray. 2023. Prompt engineering with ChatGPT: a guide for academic writers.Annals of biomedical engineering51, 12 (2023), 2629–2633

    2023

  20. [28]

    Github. [n. d.]. Copilot. https://github.com/features/copilot

  21. [29]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al . 2024. DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196(2024)

    2024 arXiv

  22. [30]

    Isabelle Guyon and André Elisseeff. 2003. An introduction to variable and feature selection.Journal of machine learning research3, Mar (2003), 1157– 1182

    2003

  23. [31]

    1977.Elements of Software Science (Operating and program- ming systems series)

    Maurice H Halstead. 1977.Elements of Software Science (Operating and program- ming systems series). Elsevier Science Inc

    1977

  24. [32]

    Ardis Hanson. 2017. Negative case analysis.The international encyclopedia of communication research methods(2017), 1–2

    2017

  25. [33]

    Mohammad Hassany, Jiaze Ke, Peter Brusilovsky, Arun Balajiee Lek- shmi Narayanan, and Kamil Akhuseyinoglu. 2024. Authoring Worked Examples for JAVA Programming with Human AI Collaboration. InProceedings of the 39th ACM/SIGAPP Symposium on Applied Computing. 101–103

    2024

  26. [34]

    Roberta Heale and Dorothy Forbes. 2013. Understanding triangulation in research.Evidence-based nursing16, 4 (2013), 98–98

    2013

  27. [35]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 1–79

    2024

  28. [36]

    Chao Hu, Yitian Chai, Hao Zhou, Fandong Meng, Jie Zhou, and Xiaodong Gu

  29. [37]

    Yuan Huang, Nan Jia, Junhuai Shu, Xinyu Hu, Xiangping Chen, and Qiang Zhou. 2020. Does your code need comment?Software: Practice and Experience 50, 3 (2020), 227–245

    2020

  30. [38]

    InProceedings of the 39th IEEE/ACM International Conference on Auto- mated Software Engineering

    How Effectively Do Code Language Models Understand Poor-Readability Code?. InProceedings of the 39th IEEE/ACM International Conference on Auto- mated Software Engineering. 795–806

  31. [39]

    Ciera Jaspan and Collin Green. 2023. Defining, measuring, and managing technical debt.IEEE Software40, 03 (2023), 15–19

    2023

  32. [40]

    Reza Iranzad and Xiao Liu. 2025. A review of random forest-based feature selection methods for data science education and applications.International Journal of Data Science and Analytics20, 2 (2025), 197–211

    2025

  33. [41]

    Can Jin, Hongwu Peng, Shiyu Zhao, Zhenting Wang, Wujiang Xu, Ligong Han, Jiahui Zhao, Kai Zhong, Sanguthevar Rajasekaran, and Dimitris N Metaxas

  34. [42]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation.ACM computing surveys55, 12 (2023), 1–38

    2023

  35. [43]

    John Johnson, Sergio Lubo, Nishitha Yedla, Jairo Aponte, and Bonita Sharif

  36. [44]

    Sungmin Kang, Louis Milliken, and Shin Yoo. 2024. Identifying inaccurate descriptions in llm-generated code comments via test execution.arXiv preprint arXiv:2406.14836(2024)

    2024 arXiv

  37. [45]

    Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen

  38. [46]

    From llms to llm-based agents for software engineering: A survey of current, challenges and future.arXiv preprint arXiv:2408.02479(2024)

    2024 arXiv

  39. [47]

    Dawn Lawrie, Christopher Morrell, Henry Feild, and David Binkley. 2006. What’s in a Name? A Study of Identifiers. In14th IEEE international conference on program comprehension (ICPC’06). IEEE, 3–12

    2006

  40. [48]

    Dawn Lawrie, Christopher Morrell, Henry Feild, and David Binkley. 2007. Effec- tive identifier names for comprehension and memory.Innovations in Systems and Software Engineering3, 4 (2007), 303–318

    2007

  41. [49]

    Valentina Lenarduzzi, Terese Besker, Davide Taibi, Antonio Martini, and Francesca Arcelli Fontana. 2021. A systematic literature review on technical debt prioritization: Strategies, processes, factors, and tools.Journal of Systems and Software171 (2021), 110827

    2021

  42. [50]

    Ron Kohavi et al. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. InIjcai, Vol. 14. Montreal, Canada, 1137–1145

    1995

  43. [51]

    Dawn Lawrie, Henry Feild, and David Binkley. 2006. Syntactic identifier con- ciseness and consistency. In2006 Sixth IEEE International Workshop on Source Code Analysis and Manipulation. IEEE, 139–148. Ye et al

    2006

  44. [52]

    Sherlock A Licorish, Ansh Bajpai, Chetan Arora, Fanyu Wang, and Kla Tan- tithamthavorn. 2025. Comparing Human and LLM Generated Code: The Jury is Still Out!arXiv preprint arXiv:2501.16857(2025)

    2025 arXiv

  45. [53]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems36 (2023), 21558–21572

    2023

  46. [54]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.ACM computing surveys 55, 9 (2023), 1–35

    2023

  47. [55]

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al . 2023. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161(2023)

    2023 arXiv

  48. [56]

    Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. 2023. GPT detectors are biased against non-native English writers.Patterns4, 7 (2023)

    2023

  49. [57]

    Yuxing Ma, Chris Bogart, Sadika Amreen, Russell Zaretzki, and Audris Mockus

  50. [58]

    Yuxing Ma, Tapajit Dey, Chris Bogart, Sadika Amreen, Marat Valiev, Adam Tutko, David Kennard, Russell Zaretzki, and Audris Mockus. 2021. World of code: enabling a research workflow for mining and analyzing the universe of open source VCS data.Empirical Software Engineering26 (...

    2021

  51. [59]

    Ggaliwango Marvin, Nakayiza Hellen, Daudi Jjingo, and Joyce Nakatumba- Nabende. 2023. Prompt engineering in large language models. InInternational conference on data intelligence and cognitive informatics. Springer, 387–402

    2023

  52. [60]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambro- sio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation.arXiv preprint arXiv:2102.04664(2021)

    2021 arXiv

  53. [61]

    Michael R Lyu, Baishakhi Ray, Abhik Roychoudhury, Shin Hwei Tan, and Patanamon Thongtanunam. 2024. Automatic programming: Large language models and beyond.ACM Transactions on Software Engineering and Methodology (2024)

    2024

  54. [62]

    Leon Moonen. 2001. Generating robust parsers using island grammars. In Proceedings eighth working conference on reverse engineering. IEEE, 13–22

    2001

  55. [63]

    In2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)

    World of code: an infrastructure for mining the universe of open source VCS data. In2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR). IEEE, 143–154

  56. [64]

    OpenAI. [n. d.]. ChatGPT release. https://openai.com/index/chatgpt/

  57. [65]

    2007.Beautiful code Leading programmers explain how they think

    Andy Oram and Greg Wilson. 2007.Beautiful code Leading programmers explain how they think. O’Reilly Media, Inc

    2007

  58. [66]

    Orni Meerbaum-Salant, Michal Armoni, and Mordechai Ben-Ari. 2011. Habits of programming in scratch. InProceedings of the 16th annual joint conference on Innovation and technology in computer science education. 168–172

    2011

  59. [67]

    Prabhaker Mishra, Uttam Singh, Chandra M Pandey, Priyadarshni Mishra, and Gaurav Pandey. 2019. Application of student’s t-test, analysis of variance, and covariance.Annals of cardiac anaesthesia22, 4 (2019), 407–411

    2019

  60. [68]

    Sebastiano Panichella, Venera Arnaoudova, Massimiliano Di Penta, and Giuliano Antoniol. 2015. Would static analysis tools help developers with code reviews?. In2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, 161–170

    2015

  61. [69]

    Delano Oliveira, Reydne Santos, Benedito De Oliveira, Martin Monperrus, Fer- nando Castor, and Fernanda Madeiral. 2024. Understanding Code Understand- ability Improvements in Code Reviews.IEEE Transactions on Software Engineer- ing(2024)

    2024

  62. [70]

    Felix Petersen, Debarghya Mukherjee, Yuekai Sun, and Mikhail Yurochkin

  63. [71]

    Valentina Piantadosi, Fabiana Fierro, Simone Scalabrino, Alexander Serebrenik, and Rocco Oliveto. 2020. How does code readability change during software evolution?Empirical Software Engineering25 (2020), 5374–5412

    2020

  64. [72]

    Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, and Andrea De Lucia. 2014. Do they really smell bad? a study on developers’ perception of bad code smells. In2014 IEEE International conference on software maintenance and evolution. IEEE, 101–110

    2014

  65. [73]

    Dangfeng Pan, Zhensu Sun, Cenyuan Zhang, David Lo, and Xiaoning Du. 2025. The Hidden Cost of Readability: How Code Formatting Silently Consumes Your LLM Budget.arXiv preprint arXiv:2508.13666(2025)

    2025 arXiv

  66. [74]

    David Ramel. 2023. GitHub Copilot AI Tech Upgraded, Already Generates 61% of Java Code. https://visualstudiomagazine.com/articles/2023/02/15/copilot- upgrade.aspx

    2023

  67. [75]

    Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models.Advances in neural information processing systems34 (2021), 11054–11070

    2021

  68. [76]

    Rodrigo REBOUÇAS DE ALMEIDA, Christoph TREUDE, and Uirá KULESZA

  69. [77]

    Simone Scalabrino, Gabriele Bavota, Christopher Vendome, Mario Linares- Vásquez, Denys Poshyvanyk, and Rocco Oliveto. 2017. Automatically assessing code understandability: How far are we?. In2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). I...

    2017

  70. [78]

    Simone Scalabrino, Mario Linares-Vásquez, Rocco Oliveto, and Denys Poshy- vanyk. 2018. A comprehensive model for code readability.Journal of Software: Evolution and Process30, 6 (2018), e1958

    2018

  71. [79]

    Daryl Posnett, Abram Hindle, and Premkumar Devanbu. 2011. A simpler model of software readability. InProceedings of the 8th working conference on mining software repositories. 73–82

    2011

  72. [80]

    Vaclav Rajlich and Prashant Gosavi. 2004. Incremental change in object-oriented programming.IEEE software21, 4 (2004), 62–69

    2004

  73. [81]

    Claude E Shannon. 1948. A mathematical theory of communication.The Bell system technical journal27, 3 (1948), 379–423

    1948

  74. [82]

    Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models.arXiv preprint arXiv:2309.05922(2023)

    2023 arXiv

  75. [83]

    Wannita Takerngsaksiri, Chakkrit Tantithamthavorn, Micheal Fu, Jirat Pa- suksmit, Kun Chen, and Ming Wu. 2025. Code readability in the age of large language models: An industrial case study from atlassian. In2025 IEEE In- ternational Conference on Software Maintenance and Evol...

    2025

  76. [84]

    Gareth Terry, Nikki Hayfield, Victoria Clarke, Virginia Braun, et al. 2017. The- matic analysis.The SAGE handbook of qualitative research in psychology2, 17-37 (2017), 25

    2017

  77. [85]

    Edith Tom, Aybüke Aurum, and Richard Vidgen. 2013. An exploration of technical debt.Journal of Systems and Software86, 6 (2013), 1498–1516

    2013

  78. [86]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971(2023)

    2023 arXiv

  79. [87]

    Simone Scalabrino, Mario Linares-Vasquez, Denys Poshyvanyk, and Rocco Oliveto. 2016. Improving code readability models with textual features. In2016 IEEE 24th International Conference on Program Comprehension (ICPC). IEEE, 1–10

    2016

  80. [88]

    Mor Shamy and Dror G Feitelson. 2023. Identifying lines and interpreting vertical jumps in eye tracking studies of reading text and code.ACM Transactions on Applied Perception20, 2 (2023), 1–20

    2023

  81. [89]

    Chong Wang, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2024. Llms meet library evolution: Evaluating deprecated api usage in llm-based code completion.arXiv preprint arXiv:2406.09834(2024)

    2024 arXiv

  82. [90]

    Andreas Stefik and Susanna Siebert. 2013. An empirical investigation into programming language syntax.ACM Transactions on Computing Education (TOCE)13, 4 (2013), 1–40

    2013

  83. [91]

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652(2021)

    2021 arXiv

  84. [92]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models.Advances in neural information processing systems35 (2022), 24824–24837

    2022

  85. [93]

    J White. 2023. A prompt pattern catalog to enhance prompt engineering with ChatGPT.arXiv preprint arXiv:2302.11382(2023)

    2023 arXiv

  86. [94]

    A Wayne Whitney. 1971. A direct method of nonparametric measurement selection.IEEE transactions on computers100, 9 (1971), 1100–1103

    1971

  87. [95]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017)

    2017

  88. [96]

    Antonio Vitale, Emanuela Guglielmi, Rocco Oliveto, and Simone Scalabrino

  89. [97]

    Personalized Code Readability Assessment: Are We There Yet?arXiv preprint arXiv:2503.07870(2025)

    2025 arXiv

  90. [98]

    xAI. [n. d.]. Grok. https://grok.com/

  91. [99]

    Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khy- athi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. 2023. How far can camels go? exploring the state of instruction tuning on open resources. InProceedings of the 37th...

    2023

  92. [100]

    Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang

  93. [101]

    Xiang Zhang, Juntai Cao, Chenyu You, and Dujian Ding. 2025. Why prompt design matters and works: A complexity analysis of prompt search space in llms. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 32525–32555

    2025

  94. [102]

    Li Zhong and Zilong Wang. 2024. Can llm replace stack overflow? a study on ro- bustness and reliability of large language model code generation. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 21841–21849

    2024

  95. [103]

    Yangtian Zi, Luisa Li, Arjun Guha, Carolyn Anderson, and Molly Q Feldman

  96. [104]

    Frank Wilcoxon. 1945. Individual comparisons by ranking methods.Biometrics bulletin1, 6 (1945), 80–83. The Readability Spectrum: Patterns, Issues, and Prompt Effects in LLM-Generated Code

    1945

  97. [105]

    Michael Williams and Tami Moser. 2019. The art of coding and thematic exploration in qualitative research.International management review15, 1 (2019), 45–55

    2019

  98. [106]

    Scott N Woodfield, Hubert E Dunsmore, and Vincent Y Shen. 1981. The effect of modularization and comments on program comprehension. InProceedings of the 5th international conference on Software engineering. 215–223

    1981

  99. [108]

    Weiwei Xu, Kai Gao, Hao He, and Minghui Zhou. 2025. Licoeval: Evaluating llms on license compliance in code generation. In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 1665–1677

    2025

  100. [110]

    A survey on large language model (llm) security and privacy: The good, the bad, and the ugly.High-Confidence Computing(2024), 100211

    2024

  101. [114]

    InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering

    “I Would Have Written My Code Differently’: Beginners Struggle to Understand LLM-Generated Code. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 1479–1488. Ye et al. A Readability Model Table A.1: Construction of the Readabili...

  102. [2019]

    In2019 IEEE International conference on software maintenance and evolution (ICSME)

    An empirical study assessing source code readability in comprehension. In2019 IEEE International conference on software maintenance and evolution (ICSME). IEEE, 513–523

  103. [2021]

    Post-processing for individual fairness.Advances in Neural Information Processing Systems34 (2021), 25944–25955

    2021

  104. [2023]

    InProceedings of the 16th International Conference on Cooperative and Human Aspects of Software Engineering, Melbourne, Australia

    What’s behind tight deadlines? Business causes of technical debt.(2023). InProceedings of the 16th International Conference on Cooperative and Human Aspects of Software Engineering, Melbourne, Australia. 14–15

    2023

  105. [2024]

    Do code summarization models process too much information? function signature may be all that is needed.ACM Transactions on Software Engineering and Methodology33, 6 (2024), 1–35

    2024

  106. [2025]

    InCompanion Proceedings of the ACM on Web Conference 2025

    Apeer: Automatic prompt engineering enhances large language model reranking. InCompanion Proceedings of the ACM on Web Conference 2025. 2494– 2502

    2025

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.