Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In real-world LLM-integrated applications, prompts behave like a distinct software artifact: they grow mainly by addition and modification during feature work, are rarely documented, and can introduce contradictory instructions.

desk verdict First large-scale empirical study of prompt evolution in open-source repos, with a genuinely useful change taxonomy, but the headline maintenance-activity split rests on an LLM classifier whose per-class accuracy is never reported. read the letter →

arxiv 2412.17298 v2 pith:JUPM4LMM submitted 2024-12-23 cs.SE

Mahan Tafreshipour , Aaron Imani , Eric Huang , Eduardo Almeida , Thomas Zimmermann , Iftekhar Ahmed This is my paper · ORCID
classification cs.SE
keywords promptengineeringLLM-integratedapplicationsevolutionempiricalsoftwaremaintenancecommitmessagedocumentationcomponentslogicalconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that prompts in applications built around large language models (LLMs) are a real, evolving software artifact with measurable maintenance patterns. Analyzing 1,262 prompt changes across 243 open-source repositories, it finds that developers mostly grow prompts by adding new components or modifying existing ones, and that such changes cluster in feature-development work. It also finds that only 21.9% of prompt changes are mentioned in commit messages, that a small but real share of changes introduce logical inconsistencies such as contradictory instructions, and that when a change is executed the LLM sometimes responds exactly as intended, sometimes not at all, and sometimes in an unintended direction. If these results hold, prompt engineering needs its own validation and testing tooling rather than being treated as an ad hoc string-editing task.

What carries the argument

The load-bearing machinery is a change-extraction and classification pipeline. Starting from an existing dataset of developer-written prompts, the paper cleans it, filters to 243 active repositories, clones each repository, and pairs consecutive versions of prompt-containing files using the ROUGE-L similarity metric to isolate 1,262 distinct prompt changes. It then codes those changes with a refined six-component taxonomy (Directive, Role, Example, Output Instruction, Consideration, Additional Information) and an eight-type change taxonomy (addition, modification, removal, rephrase, formatting, correction, generalization, restructure), reporting a 65% average Jaccard index for inter-rater agreement on multi-label coding. For maintenance-activity questions it classifies commits with a quantized LLM following a prior method whose reported overall accuracy is 51%, and for impact questions it uses chain-of-thought LLM prompting to flag potential inconsistencies, followed by manual review, before attempting to execute changed prompts in the 7 projects with usable tests.

What would settle it

Take a random sample of the 931 prompt-changing commits, have two independent human experts classify each as feature addition, bug fix, or refactor using the paper's own definitions, and compare against the LLM labels: agreement near the reported 51% overall accuracy would invalidate the maintenance-activity distribution, while agreement above roughly 80% would support it. The same audit applied to the 205 LLM-flagged inconsistency candidates would confirm or refute the 15 manually verified inconsistencies.

Watch

Extended reading notes

Core claim

The central claim is that prompt evolution in real projects follows an expansive, component-focused pattern rather than a corrective or structural one. Developers add new instructions or constraints most often (30.1% of changes) and modify existing ones (25.5%); they remove content only rarely (8.8%). The component most frequently touched, at 60.4% of changes, is the paper's refined 'Consideration' category—instructions and constraints that guide model behavior—while the core 'Directive' that defines the overall task almost never changes. The paper further claims that 59.7% of prompt-changing commits are feature-development commits, that 78.1% of prompt changes go undocumented in commit messages, that prompt changes can produce logical inconsistencies (the paper manually confirms 15 cases from 205 LLM-flagged candidates), and that in the 7 projects that could be executed the LLM response matched the change's intent in 4 cases, stayed unchanged in 2, and deviated in 1. Together these observations define prompts as a new artifact class with its own failure modes.

Load-bearing premise

The load-bearing premise is that the automated LLM commit classifier assigns maintenance activities correctly often enough on this dataset; the paper reports only a prior method's overall 51% accuracy and does not give precision and recall for its own 931 commits, so a systematic mislabeling could change the headline feature-development share.

Editorial extensions

If this is right

  • Because additions and modifications dominate, prompt maintenance tooling should focus on insert-edit operations rather than deletions or structural rewrites.
  • Since the 'Consideration' component absorbs 60.4% of changes, automated consistency checks could concentrate on newly added instructions and constraints.
  • With 78.1% of prompt changes undocumented, commit-message generation and prompt-diff documentation tools would address a real traceability gap.
  • Given confirmed inconsistencies and misaligned LLM responses, prompt changes should be validated by executing representative inputs before merge, not just reviewed as text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the 51% accuracy of the prior commit classifier suggests the 59.7% feature-development share should be re-measured on a human-labeled sample before being used as a baseline by later work.
  • Editorial extension: the paper's prompt-component taxonomy, especially the category it calls 'Consideration,' could be transferred to non-Python projects and to prompts assembled at runtime from many string fragments, where version-to-version pairing is harder.
  • Editorial extension: the 7-of-101 execution result implies that building prompt regression harnesses with frozen example inputs would let teams detect 'no response change' and 'unintended response change' automatically, turning the paper's observations into testable properties.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper reports an empirical study of how developer-written prompts evolve in open-source LLM-integrated applications. Using the PromptSet dataset and Git history, the authors collected 1,262 prompt changes across 243 GitHub repositories. They apply qualitative coding to categorize change types and prompt components, use a 4-bit quantized Llama3 70B model to classify prompt-changing commits into maintenance activities, extract and categorize commit messages mentioning prompt changes, and perform manual/LLM-based analyses of logical inconsistencies and of whether prompt changes produce the intended LLM response. The main findings are that prompts evolve primarily through additions and modifications, that most prompt-changing commits occur during feature development (59.7%), that only 21.9% of prompt changes are explicitly documented in commit messages, and that prompt changes can introduce logical inconsistencies and sometimes fail to achieve their intended effect on LLM responses.

Significance. If the findings hold, this is a useful first descriptive baseline for prompt evolution as a software maintenance activity. The paper is transparent about its extraction and coding pipeline, reports inter-rater reliability for the qualitative coding, and makes a concrete contribution by characterizing prompts as a distinct software artifact with measurable change patterns. The use of PromptSet and the manual coding of change types are strengths. However, the significance is tempered by the fact that several headline quantitative claims rest on LLM-based classifiers whose accuracy on this study's data is not reported, and by the very small sample underlying the RQ9 conclusion.

major comments (4)
  1. [IV-E1, V-B (RQ4/Observation 4)] The headline distribution of prompt-changing commits across maintenance activities (feature 59.7%, fix 18.9%, refactor 14.5%) is produced by a 4-bit quantized Llama3 70B classifier whose only reported accuracy is the 51% figure from prior work (Section IV-E1). The paper does not report precision, recall, or a confusion matrix for this model on the 931 commits in this study, and the "thorough manual review" mentioned in Section VI is not described with any procedure, sample size, or agreement statistic. Because RQ5 and Observation 4 are downstream of this same labeling, the missing validation is load-bearing; systematic confusion between feature and refactor commits, for example, could change the 59.7% estimate substantially. The authors should provide a validation set with per-class F1 or a confusion matrix on a random sample of this study's commits and discuss the sensitivity of the RQ4/RQ5 conclusions to classification error.
  2. [IV-F1, V-C (RQ6)] The 21.9% documentation rate is the basis for the claim that most prompt changes are undocumented. This estimate rests on an LLM-based extraction of commit messages referencing prompt changes (Section IV-F1), but no validation or accuracy measure is reported for this extraction step. False negatives would inflate the undocumented share, while false positives would deflate it. The authors should validate the extraction on a random sample (e.g., manually labeling 100-200 commits) or report precision/recall for the few-shot classifier, and adjust the RQ6 estimate accordingly.
  3. [V-D (RQ9), Abstract] The abstract states that "misalignment often occurs between prompt changes and LLM responses," and Observation 9 says prompt modifications "do not consistently achieve their intended effects." This is based on 7 projects (Section V-D), of which 2 showed no response change and 1 showed an unintended change; the other 4 aligned as intended. The paper acknowledges this sample is limited in Section VI, but the abstract presents the finding without the caveat. The authors should either weaken the generalization (e.g., "in a small sample, we observed...") or substantially enlarge the set of executable projects before claiming that misalignment is common.
  4. [V-B (RQ4)] The percentages in RQ4 sum to 93.1% (59.7 + 18.9 + 14.5). The remaining 6.9% of commits is not accounted for. If the LLM classifier produces an "other" category or fails to classify some commits, this should be reported explicitly; otherwise the distribution is incomplete and the denominator for the percentages is unclear.
minor comments (6)
  1. [IV-D] The population for the sample size calculation is given as 1,263 while the final dataset is stated as 1,262 (Section IV-C); please confirm which number is correct.
  2. [V-A (RQ2, Table II)] The text says Consideration accounts for 60.4% of "all prompt changes," but the denominator for the component percentages is the sum of component-specific changes (942), not the 1,463 total changes in Table I; rephrase to "all component-specific changes" or provide a consistent denominator.
  3. [Figure 6] The axes of the pairwise counts heatmap are not defined in the caption; clarify whether rows and columns are change types or component-change pairs and how the proportional frequency is normalized.
  4. [IV-F1] The few-shot prompt used to identify prompt-related commit messages is not shown; including the prompt template would improve reproducibility.
  5. [VI (Internal Validity)] The claim of a "thorough manual review" of the inconsistency-detection LLM results would be easier to assess if the number of cases reviewed and any disagreement counts were reported.
  6. [II-B, IV-D] The refined component "Consideration" is introduced in Section IV but the reader first encounters it in Section II-B's discussion of the original component set; a brief definition in Section II would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical pipeline is not a derivation chain, and the only flagged issue is an unvalidated LLM commit classifier, which is a validity risk rather than a circular step.

full rationale

This is an empirical measurement study, not a derivation chain. The prompt-change categories in Table I come from open coding with negotiated agreement and a 65% Jaccard inter-rater reliability check, not from the study's conclusions. The component taxonomy is adapted from Schulhoff et al.'s external framework, and the refinement (merging Output Formatting and Style Specifications, adding Consideration) is presented as data-driven coding rather than as a theorem. The only same-author citations are [17] and [28], used to justify the LLM-based commit classifier in Section IV-E1; that classifier's labels are an input measurement, and the study's prompt-change distribution is not present in those cited papers, so the central claims do not reduce to the self-citation. The paper does flag a limitation in Section VI ('we conducted a thorough manual review of results from LLMs used for commit classification') without giving the procedure, sample size, or agreement statistic; this is a reproducibility and measurement-validity concern, not circularity, because no fitted parameter is renamed as a prediction and no equation is defined in terms of a target conclusion. The RQ9 seven-project sample and the RQ8 manual validation are small but externally checkable, and the inconsistency findings were manually confirmed. Accordingly, no circular step can be exhibited under the required standard.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

This empirical study does not rest on fitted parameters or mathematical derivations, but its central claims depend on several dataset filtering and LLM-based classification assumptions. The main methodological choices are the 15-word prompt filter, the 65% inter-rater acceptance threshold, and the reliance on PromptSet and an LLM classifier with 51% accuracy.

free parameters (2)
  • Minimum prompt length threshold = 15 words
    Prompts shorter than 15 words are filtered out; this choice affects which prompt changes are analyzed and may exclude valid short prompts (Section IV-B).
  • Inter-rater reliability acceptance threshold = 65% Jaccard index
    An averaged Jaccard index of 65% is treated as acceptable agreement for multi-label coding; a higher threshold might have changed the codebook or coding outcomes (Section IV-D).
assumptions (4)
  • domain assumption PromptSet accurately extracts developer-written prompts from Python files
    The dataset is used to identify prompt versions from file history, but the original authors made no quality assessment, and manual inspection found fragmented entries (Section IV-A).
  • domain assumption LLM-based commit classification with 51% accuracy is sufficient for maintenance activity labels
    The paper classifies commits into feature, fix, and refactor using a 4-bit quantized Llama3 70B model with reported state-of-the-art accuracy of 51%; a near-half error rate could bias the distribution of prompt changes across activities (Section IV-E).
  • domain assumption ROUGE-L similarity captures meaningful prompt changes
    Pairwise comparison of consecutive prompt versions uses ROUGE-L to detect changes; the choice of change threshold is not reported (Section IV-C).
  • domain assumption Seven successfully executed projects are representative of prompt change impact
    The RQ9 analysis of alignment between intended and actual LLM responses is based on 7 of 101 candidate projects; the paper acknowledges this limitation (Section V-D, Threats to Validity).
invented entities (1)
  • Consideration prompt component
    purpose: A new category of prompt components grouping instructions and constraints that guide the LLM's task execution; added to the original six components from Schulhoff et al.
    The component was introduced after empirical observation of prompts that did not fit the existing taxonomy; inter-rater reliability is 65% Jaccard, but there is no external benchmark validating the category (Section II-B, Section V-A RQ2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories." pith.science (2026). https://pith.science/paper/JUPM4LMM

@misc{pith2026241217298,
  author       = {Pith},
  title        = {Pith review of: Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JUPM4LMM}},
  note         = {Machine review of arXiv:2412.17298}
}
read the original abstract

The adoption of Large Language Models (LLMs) is reshaping software development as developers integrate these LLMs into their applications. In such applications, prompts serve as the primary means of interacting with LLMs. Despite the widespread use of LLM-integrated applications, there is limited understanding of how developers manage and evolve prompts. This study presents the first empirical analysis of prompt evolution in LLM-integrated software development. We analyzed 1,262 prompt changes across 243 GitHub repositories to investigate the patterns and frequencies of prompt changes, their relationship with code changes, documentation practices, and their impact on system behavior. Our findings show that developers primarily evolve prompts through additions and modifications, with most changes occurring during feature development. We identified key challenges in prompt engineering: only 21.9% of prompt changes are documented in commit messages, changes can introduce logical inconsistencies, and misalignment often occurs between prompt changes and LLM responses. These insights emphasize the need for specialized testing frameworks, automated validation tools, and improved documentation practices to enhance the reliability of LLM-integrated applications.

Figures

Figures reproduced from arXiv: 2412.17298 by the authors.

Figure 1
Figure 1. Example of developer-written prompt B. Prompt Components A prompt often consists of multiple components, each contributing to eliciting a precise and relevant response from the model. Schulhoff et al. [45] identified six common compo￾nents used by practitioners: (1) Directive—defines the desired task with instructions or questions; (2) Examples—illustrate the expected output, aiding in few-shot learning; (3) Output … view at source ↗
Figure 2
Figure 2. Example of prompt components introduced by Schul [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overall methodology in isolation. Many of such prompts contain only a few words. Due to these limitations, we performed a data-cleaning phase before using PromptSet in our study. B. Data Cleaning and Filtering We started the data cleaning by removing empty prompts from the dataset. Next, we filtered out prompts that were not in English or contained non-ASCII characters. Finally, we eliminated incomplete and distorte… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Inconsistency in prompt statements introduced by directive modification (murchie85/gpt [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Example of refined prompt components A. Theme 1: Prompt Change Types and Patterns To address the research questions related to this theme, we present the outcomes of our qualitative analysis process. RQ1: What are the categories of changes made to prompts? Our qualitat…
Figure 6
Figure 6. Figure 6: Proportional Distribution of Changes Across Prompt Compo [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Distribution of prompt component changes over software [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Distribution of component-independent changes over software [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Secure Code Generation at Scale with Reflexion

    cs.CR 2025-11 conditional novelty 4.0 of 10

    Three-round reflexion prompting raises LLM-generated code security from 70.74% to 79.43% as measured by the ICD static detector on the Instruct Prime benchmark.

Reference graph

Works this paper leans on

58 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Abridge: Medical conversation summarization,

    Abridge AI, Inc., “Abridge: Medical conversation summarization,” 2021, accessed: 2023-11-03. [Online]. Available: https://www.abridge.com/

    work page 2021

  2. [2]

    Multi-language software development in the llm era: Insights from practitioners’ conversations with chatgpt,

    L. Aguiar, M. Paixao, R. Carmo, E. Soares, A. Leal, M. Freitas, and E. Gama, “Multi-language software development in the llm era: Insights from practitioners’ conversations with chatgpt,” in Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , ser. ESEM ’24. New York, NY , USA: Association for Comput...

    arXiv 2024

  3. [3]

    Semantic scholar,

    Allen Institute for AI, “Semantic scholar,” 2015, accessed: 2024-11-03. [Online]. Available: https://www.semanticscholar.org/

    work page 2015

  4. [4]

    GPT Store Statistics & Facts,

    D. H. Bæk, “GPT Store Statistics & Facts,” 2024, accessed: 2024-10-31. [Online]. Available: https://seo.ai/blog/gpt-store-statistics-facts

    work page 2024

  5. [5]

    Demystifying issues, causes and solutions in llm open-source projects,

    Y . Cai, P. Liang, Y . Wang, Z. Li, and M. Shahin, “Demystifying issues, causes and solutions in llm open-source projects,” arXiv preprint arXiv:2409.16559, 2024

    arXiv 2024

  6. [6]

    Corbin and A

    J. Corbin and A. Strauss, Basics of qualitative research . sage, 2015, vol. 14

    work page 2015

  7. [7]

    Exploring prompt engineering prac- tices in the enterprise,

    M. Desmond and M. Brachman, “Exploring prompt engineering prac- tices in the enterprise,” arXiv preprint arXiv:2403.08950 , 2024

    arXiv 2024

  8. [8]

    Fira: fine-grained graph-based code change representation for automated commit message generation,

    J. Dong, Y . Lou, Q. Zhu, Z. Sun, Z. Li, W. Zhang, and D. Hao, “Fira: fine-grained graph-based code change representation for automated commit message generation,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 970–981

    work page 2022

Show all 58 references
  1. [9]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  2. [10]

    A comparison of features for automatic readability assessment,

    L. Feng, M. Jansche, M. Huenerfauth, and N. Elhadad, “A comparison of features for automatic readability assessment,” in Coling 2010: Posters, 2010, pp. 276–284

    2010

  3. [11]

    Promptbreeder: Self-referential self-improvement via prompt evolution,

    C. Fernando, D. Banarse, H. Michalewski, S. Osindero, and T. Rockt ¨aschel, “Promptbreeder: Self-referential self-improvement via prompt evolution,” arXiv preprint arXiv:2309.16797 , 2023

    2023 arXiv

  4. [12]

    What makes good in-context demonstrations for code intelligence tasks with llms?

    S. Gao, X.-C. Wen, C. Gao, W. Wang, H. Zhang, and M. R. Lyu, “What makes good in-context demonstrations for code intelligence tasks with llms?” in 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp. 761–773

    2023

  5. [13]

    GitHub Copilot,

    GitHub, Inc., “GitHub Copilot,” https://github.com/features/copilot, 2021, accessed: [your access date, e.g., 2024-11-03]

    2021

  6. [14]

    Rethinking software engineering in the era of foundation models: A curated catalogue of challenges in the development of trustworthy fmware,

    A. E. Hassan, D. Lin, G. K. Rajbahadur, K. Gallaba, F. R. Cogo, B. Chen, H. Zhang, K. Thangarajah, G. Oliva, J. J. Lin, W. M. Abdullah, and Z. M. J. Jiang, “Rethinking software engineering in the era of foundation models: A curated catalogue of challenges in the development of...

    2024

  7. [15]

    Quantifying the persona effect in llm simulations,

    T. Hu and N. Collier, “Quantifying the persona effect in llm simulations,” arXiv preprint arXiv:2402.10811 , 2024

    2024 arXiv

  8. [16]

    Ibm watson assistant,

    IBM, “Ibm watson assistant,” https://www.ibm.com/products/ watsonx-assistant, accessed: 2024-11-03

    2024

  9. [17]

    Context conquers parameters: Outperforming proprietary llm in commit message generation,

    A. Imani, I. Ahmed, and M. Moshirpour, “Context conquers parameters: Outperforming proprietary llm in commit message generation,” arXiv preprint arXiv:2408.02502, 2024

    2024 arXiv

  10. [18]

    Inc., “Cursor,” 2024, version 1.0

    A. Inc., “Cursor,” 2024, version 1.0. [Online]. Available: https: //www.cursor.com/

    2024

  11. [19]

    Elicit: The ai research assistant,

    O. Inc., “Elicit: The ai research assistant,” 2021, accessed: 2024-11-03. [Online]. Available: https://elicit.org/

    2021

  12. [20]

    Research rabbit,

    R. R. Inc., “Research rabbit,” 2021, accessed: 2024-11-03. [Online]. Available: https://www.researchrabbit.ai/

    2021

  13. [21]

    Tabnine: Ai code assistant,

    T. Inc., “Tabnine: Ai code assistant,” 2024, accessed: 2024-11-03. [Online]. Available: https://www.tabnine.com/

    2024

  14. [22]

    Infermedica: Medical triage and symptom checker,

    Infermedica, “Infermedica: Medical triage and symptom checker,” 2021, accessed: 2023-11-03. [Online]. Available: https://www.infermedica. com/

    2021

  15. [23]

    Identifying and mitigating vulnerabilities in llm-integrated applica- tions,

    F. Jiang, Z. Xu, L. Niu, B. Wang, J. Jia, B. Li, and R. Poovendran, “Identifying and mitigating vulnerabilities in llm-integrated applica- tions,” arXiv preprint arXiv:2311.16153 , 2023

    2023 arXiv

  16. [24]

    The promises and perils of mining github,

    E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, and D. Damian, “The promises and perils of mining github,” in Proceedings of the 11th Working Conference on Mining Software Repositories, ser. MSR 2014. New York, NY , USA: Association for Computing Machinery, 2...

    2014

  17. [25]

    Kai-gpt: Banking’s first purpose-built language model,

    Kasisto, “Kai-gpt: Banking’s first purpose-built language model,” https: //kasisto.com/products/kai-gpt/, accessed: 2024-11-03

    2024

  18. [26]

    Label Studio,

    Label Studio, “Label Studio,” https://labelstud.io/, accessed: 2024-11-03

    2024

  19. [27]

    The measurement of observer agreement for categorical data,

    J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” biometrics, pp. 159–174, 1977

    1977

  20. [28]

    Only diff is not enough: Generating commit messages leveraging reasoning and action of large language model,

    J. Li, D. Farag ´o, C. Petrov, and I. Ahmed, “Only diff is not enough: Generating commit messages leveraging reasoning and action of large language model,” Proceedings of the ACM on Software Engineering , vol. 1, no. FSE, pp. 745–766, 2024

    2024

  21. [29]

    Contradoc: Understanding self- contradictions in documents with large language models,

    J. Li, V . Raheja, and D. Kumar, “Contradoc: Understanding self- contradictions in documents with large language models,” arXiv preprint arXiv:2311.09182, 2023

    2023 arXiv

  22. [30]

    Spell: Semantic prompt evolution based on a llm,

    Y . B. Li and K. Wu, “Spell: Semantic prompt evolution based on a llm,” arXiv preprint arXiv:2310.01260 , 2023

    2023 arXiv

  23. [31]

    Prompts are programs too! understanding how developers build software containing prompts,

    J. T. Liang, M. Lin, N. Rao, and B. A. Myers, “Prompts are programs too! understanding how developers build software containing prompts,” arXiv preprint arXiv:2409.12447 , 2024

    2024 arXiv

  24. [32]

    A large-scale survey on the usability of ai programming assistants: Successes and challenges,

    J. T. Liang, C. Yang, and B. A. Myers, “A large-scale survey on the usability of ai programming assistants: Successes and challenges,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , ser. ICSE ’24. New York, NY , USA: Association for Comp...

    2024

  25. [33]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  26. [34]

    Atom: Commit message generation based on abstract syntax tree and hybrid ranking,

    S. Liu, C. Gao, S. Chen, L. Y . Nie, and Y . Liu, “Atom: Commit message generation based on abstract syntax tree and hybrid ranking,” IEEE Transactions on Software Engineering , vol. 48, no. 5, pp. 1800–1817, 2020

    2020

  27. [35]

    Rethinking the role of demonstrations: What makes in-context learning work?

    S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?” arXiv preprint arXiv:2202.12837 , 2022

    2022 arXiv

  28. [36]

    Beyond the comfort zone: Emerging solutions to overcome challenges in integrating llms into software products,

    N. Nahar, C. K ¨astner, J. Butler, C. Parnin, T. Zimmermann, and C. Bird, “Beyond the comfort zone: Emerging solutions to overcome challenges in integrating llms into software products,” arXiv preprint arXiv:2410.12071, 2024

    2024 arXiv

  29. [37]

    Coregen: Contextualized code representation learning for commit message gener- ation,

    L. Y . Nie, C. Gao, Z. Zhong, W. Lam, Y . Liu, and Z. Xu, “Coregen: Contextualized code representation learning for commit message gener- ation,” Neurocomputing, vol. 459, pp. 97–107, 2021

    2021

  30. [38]

    Nuance dax: Ambient clinical intelligence,

    Nuance Communications, Inc., “Nuance dax: Ambient clinical intelligence,” 2021, accessed: 2023-11-03. [Online]. Available: https: //www.nuance.com/healthcare/ambient-clinical-intelligence.html

    2021

  31. [39]

    An empirical study of the non-determinism of chatgpt in code generation,

    S. Ouyang, J. M. Zhang, M. Harman, and M. Wang, “An empirical study of the non-determinism of chatgpt in code generation,” ACM Transactions on Software Engineering and Methodology , 2024

    2024

  32. [40]

    Phantom: Curating github for engineered software projects using time- series clustering,

    P. Pickerill, H. J. Jungen, M. Ochodek, M. Ma ´ckowiak, and M. Staron, “Phantom: Curating github for engineered software projects using time- series clustering,” Empirical Software Engineering , vol. 25, pp. 2897– 2929, 2020

    2020

  33. [41]

    Promptset: A programmer’s prompting dataset,

    K. Pister, D. J. Paul, I. Joshi, and P. Brophy, “Promptset: A programmer’s prompting dataset,” in Proceedings of the 1st International Workshop on Large Language Models for Code , 2024, pp. 62–69

    2024

  34. [42]

    The probabilistic basis of jaccard’s index of similarity,

    R. Real and J. M. Vargas, “The probabilistic basis of jaccard’s index of similarity,” Systematic biology, vol. 45, no. 3, pp. 380–385, 1996

    1996

  35. [43]

    A systematic survey of prompt engineering in large language models: Techniques and applications,

    P. Sahoo, A. K. Singh, S. Saha, V . Jain, S. Mondal, and A. Chadha, “A systematic survey of prompt engineering in large language models: Techniques and applications,” arXiv preprint arXiv:2402.07927 , 2024

    2024 arXiv

  36. [44]

    Future of software development with generative ai,

    J. Sauvola, S. Tarkoma, M. Klemettinen, J. Riekki, and D. Doermann, “Future of software development with generative ai,” Automated Soft- ware Engineering, vol. 31, no. 1, p. 26, 2024

    2024

  37. [45]

    The prompt report: A systematic survey of prompting techniques,

    S. Schulhoff, M. Ilie, N. Balepur, K. Kahadze, A. Liu, C. Si, Y . Li, A. Gupta, H. Han, S. Schulhoff et al., “The prompt report: A systematic survey of prompting techniques,” arXiv preprint arXiv:2406.06608 , 2024

    2024 arXiv

  38. [46]

    Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks,

    J. Shin, C. Tang, T. Mohati, M. Nayebi, S. Wang, and H. Hem- mati, “Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks,” arXiv preprint arXiv:2310.10508, 2023

    arXiv

  39. [47]

    Basics of qualitative research techniques,

    A. Strauss and J. Corbin, “Basics of qualitative research techniques,” 1998

    1998

  40. [48]

    How do software engineers understand code changes? an exploratory study in industry,

    Y . Tao, Y . Dang, T. Xie, D. Zhang, and S. Kim, “How do software engineers understand code changes? an exploratory study in industry,” in Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering , ser. FSE ’12. New York, NY , USA:...

    2012

  41. [49]

    Report from an experiment: Impact of documentation on maintenance,

    E. Tryggeseth, “Report from an experiment: Impact of documentation on maintenance,” Empirical Software Engineering , vol. 2, no. 2, pp. 201–207, 1997

    1997

  42. [50]

    Unveiling chatgpt’s usage in open source projects: A mining-based study,

    R. Tufano, A. Mastropaolo, F. Pepe, O. Dabi ´c, M. Di Penta, and G. Bavota, “Unveiling chatgpt’s usage in open source projects: A mining-based study,” in 2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR) , 2024, pp. 571–583

    2024

  43. [51]

    Large language models as software components: A taxonomy for llm-integrated applications,

    I. Weber, “Large language models as software components: A taxonomy for llm-integrated applications,” ArXiv, vol. abs/2406.10300,

    arXiv

  44. [52]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems , vol. 35, pp. 24 824–24 837, 2022

    2022

  45. [53]

    A prompt pattern catalog to enhance prompt engineering with chatgpt,

    J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” ArXiv, vol. abs/2302.11382,

    arXiv

  46. [54]

    Bloomberggpt: A large language model for finance,

    S. Wu, O. Irsoy, S. Lu, V . Dabravolski, M. Dredze, S. Gehrmann, P. Kambadur, D. Rosenberg, and G. Mann, “Bloomberggpt: A large language model for finance,” arXiv preprint arXiv:2303.17564 , 2023

    2023 arXiv

  47. [55]

    Sifid: Reassess summary factual inconsistency detection with llm,

    J. Yang, H. Liu, W. Guo, Z. Rao, Y . Xu, and D. Niu, “Sifid: Reassess summary factual inconsistency detection with llm,” arXiv preprint arXiv:2403.07557, 2024

    2024 arXiv

  48. [56]

    Detecting code comment inconsistencies using llm and program analysis,

    Y . Zhang, “Detecting code comment inconsistencies using llm and program analysis,” in Companion Proceedings of the 32nd ACM Inter- national Conference on the Foundations of Software Engineering , 2024, pp. 683–685

    2024

  49. [2023]

    Available: https://api.semanticscholar.org/CorpusID: 257079092

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 257079092

  50. [2024]

    Available: https://api.semanticscholar.org/CorpusID: 270559976

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 270559976

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.