Pith. sign in

REVIEW 3 major objections 5 minor 132 references

Agentic pull requests are at least as clean as human ones, a 489-repository study finds.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:32 UTC pith:TCY33KEY

load-bearing objection Useful descriptive study of agentic PRs, but the headline defect-proneness claim is undermined by a merged-only selection bias; worth refereeing after sensitivity analyses. the 3 major comments →

arxiv 2607.21832 v1 pith:TCY33KEY submitted 2026-07-23 cs.SE cs.LG

How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests

Iren Mazloomzadeh , Mohammad Mehdi Morovati , Foutse Khomh This is my paper
classification cs.SE cs.LG
keywords AI coding agentspull requestssoftware qualitydefect pronenessmerge ratesoftware development lifecycleempirical studyPython repositories
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that AI coding agents' contributions are not a software-quality liability. Using 220,612 closed pull requests from 489 popular Python repositories, including 9,428 agent-generated PRs, it tracks merge outcomes, task types, and bug-inducing commits across four equal quarters of each repository's agentic activity. The headline result is that agentic PRs show bug-inducing rates comparable to, and often lower than, human PRs, with mostly non-significant statistical differences and negligible effect sizes. The paper also finds that agent acceptance varies by agent (Claude highest, Devin lowest) but is not systematically tied to lifecycle stage, and that agents cluster in documentation, dependency, and testing tasks. If the defect result is right, it undercuts the assumption that AI-generated code quietly increases long-term maintenance burden.

Core claim

On the paper's own terms, the central discovery is that AI coding agents, across roughly a year of adoption in open-source Python projects, maintain defect proneness no worse than humans: agentic PRs consistently show equal or lower bug-inducing commit rates in every development quarter, and only the fourth-quarter difference reaches statistical significance, with negligible practical effect. Surrounding this, the study shows that merge probability differs strongly across agents (estimated 84.3% for Claude down to 43.0% for Devin) but shows no statistically significant change across lifecycle quarters, and that mergeability is task-dependent: localized declarative tasks merge more readily, w

What carries the argument

The central analytical device is the agentic activity span: the time between a repository's first and most recent pull request from a given agent, divided into four equal quarters. This lets the authors align repositories of different ages on a common lifecycle clock. Around it, the paper fits a binomial generalized linear mixed model with agent and quarter as fixed effects and repository as a random intercept, labels development tasks through unsupervised topic modeling, and flags bug-inducing commits using a keyword-based bug-fix search followed by a bug-inducing commit detection algorithm, applied to a balanced sample of 2,275 merged agentic and 2,275 merged human pull requests.

Load-bearing premise

The entire agent-versus-human comparison rests on the assumption that PRs are correctly labeled as agentic or human from branch prefixes, bot author names, and 'Co-Authored-By: Claude' text; if many agentic PRs lack these markers and are counted as human, the defect and merge comparisons are biased.

What would settle it

Manually verify PR authorship on a random sample of the 489 repositories, including PRs whose metadata carries no agent marker. If unreported agentic PRs (submitted from a bot but without the expected branch/bot markers, or by developers who manually copied agent output) exhibit bug-inducing rates materially above the labeled agentic group, the comparable-defect conclusion fails. A simpler check: count how many PRs in the 'human' group were authored by accounts that later merge agent-generated code, and see whether the distribution of branch names in the human group overlaps the reserved agent

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If agentic PRs are genuinely no more defect-prone, maintainers can review AI-generated changes with the same risk posture as human ones, at least for Python projects above 100 stars.
  • The stability of merge rates across lifecycle quarters implies agent adoption does not degrade project quality or review practice over time.
  • Since agents dominate documentation, dependency, and testing tasks, teams can immediately target those task families for automation and expect the highest acceptance.
  • The faster merge times and narrower scope of agentic PRs imply that a large portion of agent contributions bypass deep review precisely because they are low-risk; risk is concentrated in a few context-heavy categories.
  • Comparisons with human PRs show negligible effect sizes, so differences in commit counts, file counts, and comments should not be read as meaningful productivity differences.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper compares labeled against unlabeled PRs, not against a ground-truth audit of agent authorship; if the missing-marker rate is high, the 'comparable defect' finding could be an artifact. A stratified manual audit of PR authorship in a subsample of these repositories would quantify that risk.
  • The Q2-to-Q3 drop in agentic bug-inducing rate could reflect either model improvements or reviewer adaptation; separating agent versions and review experience would tell which mechanism drives it.
  • The task taxonomy suggests a testable extension: if agents are assigned open issues instead of self-selecting tasks, merge rates should fall toward the low end of the observed range, because self-selected tasks are biased toward the easy categories.
  • The comparable-defect result is limited to Python, GitHub, and >100-star repositories; replicating on JavaScript/TypeScript or on enterprise repositories would test whether the pattern generalizes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies agentic pull requests (PRs) from the AIDev dataset, covering 9,428 PRs by five coding agents across 489 popular Python repositories. It poses three research questions: (RQ1) how agentic PR mergeability varies across agents and over a repository-lifecycle quarter decomposition; (RQ2) what development tasks agentic PRs address, using BERTopic topic modeling and a validation sample of 100 PRs; and (RQ3) how merged agentic PRs compare with merged human PRs on complexity metrics and on bug-inducing-commit propensity, using mixed-effects models, Mann–Whitney U tests, and SZZ-based defect detection. The main findings are that merge rates differ across agents but not systematically across lifecycle quarters, that agents are used mostly for documentation, dependency, and testing tasks, and that merged agentic PRs exhibit mostly negligible or non-significant differences from human PRs, with comparable or lower bug-inducing rates.

Significance. If the results hold, the paper provides a useful longitudinal characterization of AI coding agents in open-source development: it offers one of the first quarterly lifecycle analyses of agentic PRs, compares five agents, and includes a defect-proneness analysis. The replication package, use of effect sizes alongside p-values, and explicit treatment of multiple testing are strengths. The central defect claim, however, rests on a merged-only comparison that is vulnerable to collider/selection bias, and the agent-vs-human identification rules are acknowledged but unquantified. These issues are load-bearing for the headline conclusions, so the paper needs substantial additional analysis before its main claims can be accepted.

major comments (3)
  1. [§4.3.4, Table 12, Fig. 12; §3.3.3] Finding 7 and the abstract's 'comparable or lower defect proneness' claim are based on comparing bug-inducing rates between 2,275 merged agentic and 2,275 merged human PRs. This is a selection/collider comparison: agentic merge probabilities range from 43.0% (Devin) to 84.3% (Claude) in Table 3, while human PRs merge at roughly 84–85%, so the merged agentic sample is a quality-filtered subset, whereas the merged human sample is nearly representative. Section 6 lists the exclusion of open PRs as a threat but does not address conditioning on merge status. Without a selection model, a comparison of PRs that passed review, or a robustness check on minimally-reviewed PRs, Finding 7 overstates the defect-proneness comparison. This is the core claim of RQ3 and needs to be addressed.
  2. [§3.2, §6] The entire agent-vs-human contrast depends on identifying agentic PRs by branch prefixes ('head:codex/', 'head:copilot/', 'head:cursor/'), the Devin bot author, and Claude's 'Co-Authored-By: Claude' body text; human PRs are defined as all non-bot PRs without these markers. Section 6 acknowledges that 'some agent-assisted contributions may not be explicitly identifiable through repository metadata,' but the magnitude of the resulting misclassification is never quantified. If even a small fraction of agentic PRs lack the markers and are counted as human, all merge-rate, task-prevalence, and defect comparisons are biased. The authors should provide a validation of the identification rules, for example by manually inspecting a sample of PRs classified as human for hidden agent markers, or by comparing with commit-level agent signatures.
  3. [§4.2.1–4.2.3, Fig. 8] The temporal task-trend claims in RQ2 (e.g., 'increasing,' 'decreasing,' 'stable' merge ratios over quarters) are supported only by visual inspection of Fig. 8 and by narrative interpretation; no statistical test is applied to these quarterly trends. Several task categories have no observations in some quarters (e.g., token management in Q2), and the Empirical Bayes normalization handles group-size differences but not the temporal trend significance. This limits the strength of Finding 5 and the longitudinal conclusions about task-specific evolution. The authors should either add formal trend tests (e.g., a logistic GLMM with task-by-quarter interaction) or explicitly label these as descriptive observations without directional claims.
minor comments (5)
  1. [§4.3.1, Table 7] The text states that Cursor's changed-LOC difference from human PRs is non-significant and reports 'Cliff’s δ = 0.375', but Table 7 lists p=0.375 and Cliff's δ=-0.044. The 0.375 is the p-value, not the effect size. This is a factual error that should be corrected.
  2. [§1, §7, §6] Typos: 'Finlay' should be 'Finally' (end of Section 1); 'agenic' should be 'agentic' (Section 7); 'well-representated' should be 'well-represented' (Section 6).
  3. [§4.1, Table 4] The quarter-specific EMMs in Table 4 are labeled '1st Q' through '4th Q'; the text refers to 'third quarter' and 'fourth quarter' consistently, but the term 'quarter' is overloaded with calendar quarters in Section 6's sampling description. Clarify that lifecycle quarters are relative to each repository's agentic activity span.
  4. [§3.3.2] The BERTopic configuration selects k=36 based on CV and UMass coherence, but the validation sample of 100 PRs is small and the inter-rater agreement is computed on the final labels. The paper should report the topic distribution of the validation sample and per-topic accuracy, since some topics may be poorly validated.
  5. [§4.2, Fig. 6] Fig. 6 is described as showing 'merged and rejected agentic PRs across different development tasks,' but the caption in the text does not clarify whether counts or proportions are shown. The figure should include axis labels or a note on normalization.

Circularity Check

0 steps flagged

No significant circularity: the study is observational, with conclusions drawn from independent GitHub data and standard statistical/topic-modeling methods, not from parameters fitted to the target outcomes.

full rationale

This paper is an observational empirical study, not a derivation or prediction exercise. The central outputs—merge probabilities by agent and quarter, task-type distributions from BERTopic, and bug-inducing commit comparisons—are computed directly from GitHub PR metadata and repository mining. No fitted parameter is renamed as a prediction: the GLMM estimates agent and quarter effects, but those estimates are presented as descriptive summaries (emmeans) and are not used to predict the very outcomes they were fit on. The topic model (BERTopic with k=36 chosen by coherence scores) is a data-driven clustering step; the later findings about task prevalence and task-dependent merge rates are summaries of those clusters, not equivalences built into the model. The defect-proneness analysis uses SZZ on merged PRs; while the merged-only design raises a selection-bias threat to validity, that is a statistical inference concern, not a definitional circularity: the paper does not define 'agentic PR quality' in terms of merge status and then rediscover it. The paper's self-citations (e.g., [1, 80] for bug-fix/SZZ methodology, [106] for LLM summarization) are methodological and externally grounded in prior published work; they are not invoked to forbid alternatives or to supply the paper's conclusions. The acknowledged limitation that 'some agent-assisted contributions may not be explicitly identifiable through repository metadata' (Section 6) is a measurement threat, not a circular step. Overall, the findings could in principle have come out differently given the same methodology, so the derivation chain is not forced by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

No formal axioms; the ledger captures dataset, classification, and measurement assumptions plus the topic-modeling hyperparameter choices that the empirical claims depend on.

free parameters (2)
  • Number of BERTopic topics (k) = 36
    Selected from range 10-60 by highest C_V coherence (0.624) and better UMass score; a modeling choice that determines the task taxonomy used throughout RQ2.
  • MMR diversity in topic representation = 0.5
    Hand-set in the chained representation model; affects topic labels and interpretability, though not the merge-rate estimates directly.
axioms (6)
  • domain assumption Repositories with >100 GitHub stars are representative of agentic software development
    Section 3.2; excludes smaller or less active projects, so findings may not generalize.
  • domain assumption A repository's agentic lifecycle can be divided into four equal time quarters from first to most recent agentic PR
    Section 3.2 and Section 4.1; if project activity is uneven, equal quarters may not align with meaningful lifecycle stages.
  • domain assumption Agentic PRs are identifiable by branch prefixes, Devin bot author, and 'Co-Authored-By: Claude' body marker
    Section 3.2; false negatives and false positives in this classification would bias all comparisons.
  • domain assumption Human-generated PRs are all closed PRs that are not bots and lack agent markers
    Section 3.2; agent-assisted PRs without markers would be misclassified as human.
  • domain assumption SZZ (PyDriller implementation) correctly identifies bug-inducing commits
    Section 6; the paper acknowledges known precision 0.39 and recall 0.67 without issue-date filtering, and errors in either group affect the defect-proneness comparison.
  • domain assumption BERTopic topics (k=36) after manual merging represent meaningful development-task categories
    Section 3.3.2; validated on 100 PRs with 87% agreement, but label noise propagates into merge-ratio and trend estimates.

pith-pipeline@v1.3.0-alltime-deepseek · 39124 in / 11919 out tokens · 111942 ms · 2026-08-01T06:32:36.743755+00:00 · methodology

0 comments
read the original abstract

Recent advances in large language models and their rapid adoption across software engineering tasks have made Artificial Intelligence (AI) coding agents an integral component of modern software development workflows. While developers increasingly benefit from these coding agents, their impact on software quality remains insufficiently understood. In particular, how agentic contributions evolve across the software development lifecycle has not been thoroughly investigated. This study aims to characterize agentic pull requests (PR) in comparison to human generated PRs and to examine how their properties change across different stages of the development lifecycle. Using the AIDev dataset, we first analyze how differences in merge rates between agentic and human generated PRs vary over time. We then identify the types of development tasks where AI coding agents are predominantly applied and investigate how these task distributions evolve across development quarters. Finally, we compare a set of key characteristics of agentic and human generated PRs, focusing on their implications for software quality and their temporal dynamics. Overall, our findings provide an empirical and longitudinal perspective on the role of AI coding agents in software development, offering a more nuanced understanding of their benefits and limitations in real-world practices.

Figures

Figures reproduced from arXiv: 2607.21832 by Foutse Khomh, Iren Mazloomzadeh, Mohammad Mehdi Morovati.

Figure 1
Figure 1. Figure 1: Workflow of PR generated by AI coding agents [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: High-level view of the used methodology for this paper [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of 𝐶𝑉 Coherence scores across varying number of topics (𝑘) for different Embedding Models. these analyses enable us to identify the tasks most frequently addressed by coding agents and determine how the mergeability of PRs associated with these tasks changes as software systems evolve. We conduct topic modeling on the complete set of 9,428 agentic PRs extracted within previous steps (please see … view at source ↗
Figure 4
Figure 4. Figure 4: Agentic-PRs’ Merge rate distributions across agents [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: PRs merge rate over the whole period that coding agents have been used [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Merged vs rejected agentic PRs per topic [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Agentic PRs’ merge ratio per topic Proc. ACM Meas. Anal. Comput. Syst., Vol. 37, No. 4, Article 111. Publication date: July 2026 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Distribution of Agentic PRs’ merge ratio across topics per quarter [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Boxplot distributions of the complexity metrics for A-PRs H-PRs across different agents [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Complexity metrics comparing aggregated merged agentic PRs with human generated PRs [PITH_FULL_IMAGE:figures/full_fig_p028_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Complexity metrics comparing aggregated merged agentic PRs with human generated PRs per quarter [PITH_FULL_IMAGE:figures/full_fig_p029_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Bug-Inducing behavior of agentic vs human generated PRs over time [PITH_FULL_IMAGE:figures/full_fig_p033_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

132 extracted references · 1 canonical work pages

  1. [1]

    Mouna Abidi, Md Saidur Rahman, Moses Openja, and Foutse Khomh. 2021. Are multi-language design smells fault-prone? An empirical study.ACM Transactions on Software Engineering and Methodology (TOSEM)30, 3 (2021), 1–56

    2021

  2. [2]

    Shyam Agarwal, Hao He, and Bogdan Vasilescu. 2026. AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development.arXiv preprint arXiv:2601.13597(2026)

    arXiv 2026

  3. [3]

    Roozbeh Aghili, Heng Li, and Foutse Khomh. 2023. Studying the characteristics of AIOps projects on GitHub.Empirical Software Engineering28, 6 (2023), 143

    2023

  4. [4]

    Iyad AlAgha. 2021. Topic modeling and sentiment analysis of Twitter discussions on COVID-19 from spatial and temporal perspectives. Journal of Information Science Theory and Practice9, 1 (2021), 35–53

    2021

  5. [5]

    Khairul Alam and Banani Roy. 2026. Analyzing GitHub Issues and Pull Requests in nf-core Pipelines: Insights into nf-core Pipeline Repositories.arXiv preprint arXiv:2601.09612(2026)

    arXiv 2026

  6. [6]

    Khairul Alam, Banani Roy, Chanchal K Roy, and Kartik Mittal. 2025. An empirical investigation on the challenges in scientific workflow systems development.Empirical Software Engineering30, 5 (2025), 151

    2025

  7. [7]

    Kamel Alrashedy and Ahmed Binjahlan. 2024. How do software engineering researchers use github? an empirical study of artifacts & impact. In2024 IEEE International Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 118–130

    2024

  8. [8]

    Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. 2024. Automated unit test improvement using large language models at meta. InCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. 185–196

    2024

  9. [9]

    Zakarea Alshara, Anas Shatnawi, Hamzeh Eyal-Salman, Abdelhak-Djamel Seriai, and Maad Shatnawi. 2022. Pi-link: A ground-truth dataset of links between pull-requests and issues in github.IEEE Access11 (2022), 697–710

    2022

  10. [10]

    Andrea Arcuri and Lionel Briand. 2014. A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering.Software Testing, Verification and Reliability24, 3 (2014), 219–250

    2014

  11. [11]

    Ander Artola Velasco, Efstratios Tsirtsis, Nastaran Okati, and Manuel Gomez Rodriguez. 2025. Is Your LLM Overcharging You? Tokenization, Transparency, and Incentives.arXiv preprint arXiv:2505.21627(2025)

    Pith/arXiv arXiv 2025

  12. [12]

    Lyes Attouche, Mohamed-Amine Baazizi, Dario Colazzo, Giorgio Ghelli, Carlo Sartiani, and Stefanie Scherzinger. 2024. Validation of modern JSON schema: Formalization and complexity.Proceedings of the ACM on Programming Languages8, POPL (2024), 1451–1481

    2024

  13. [13]

    Alberto Bacchelli and Christian Bird. 2013. Expectations, outcomes, and challenges of modern code review. In2013 35th International Conference on Software Engineering (ICSE). IEEE, 712–721

    2013

  14. [14]

    Deepika Badampudi, Michael Unterkalmsteiner, and Ricardo Britto. 2023. Modern code reviews—Survey of literature and practice. ACM Transactions on Software Engineering and Methodology32, 4 (2023), 1–61

    2023

  15. [15]

    Slimane Bellaouar, Mohammed Mounsif Bellaouar, and Issam Eddine Ghada. 2021. Topic modeling: Comparison of LSA and LDA on scientific publications. InProceedings of the 2021 4th International Conference on Data Storage and Data Engineering. 59–64

    2021

  16. [16]

    Keith H Bennett and Václav T Rajlich. 2000. Software maintenance and evolution: a roadmap. InProceedings of the Conference on the Future of Software Engineering. 73–87

    2000

  17. [17]

    Christian Bird, Nachiappan Nagappan, Brendan Murphy, Harald Gall, and Premkumar Devanbu. 2011. Don’t touch my code! Examining the effects of ownership on software quality. InProceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. 4–14

    2011

  18. [18]

    Benjamin M Bolker. 2015. Linear and generalized linear mixed models.Ecological statistics: contemporary theory and application2015 (2015), 309–333. Proc. ACM Meas. Anal. Comput. Syst., Vol. 37, No. 4, Article 111. Publication date: July 2026. How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests•111:39

    2015

  19. [19]

    Hudson Borges, Andre Hora, and Marco Tulio Valente. 2016. Understanding the factors that impact the popularity of GitHub repositories. In2016 IEEE international conference on software maintenance and evolution (ICSME). IEEE, 334–344

    2016

  20. [20]

    Hudson Borges and Marco Tulio Valente. 2018. What’s in a github star? understanding repository starring practices in a social coding platform.Journal of Systems and Software146 (2018), 112–129

    2018

  21. [21]

    Mark Chen. 2021. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374(2021)

    Pith/arXiv arXiv 2021

  22. [22]

    Xiang Chen, Chaoyang Gao, Chunyang Chen, Guangbei Zhang, and Yong Liu. 2025. An empirical study on challenges for llm application developers.ACM Transactions on Software Engineering and Methodology34, 7 (2025), 1–37

    2025

  23. [23]

    Claude CodeDevin. 2025. Claude Code official website. https://www.claude.com/product/claude-code. Accessed: 2025-11-15

    2025

  24. [24]

    OpenAI Codex. 2025. OpenAI Codex official website. https://openai.com/codex/. Accessed: 2025-11-15

    2025

  25. [25]

    GitHub Copilot. 2025. GitHub Copilot official website. https://github.com/copilot. Accessed: 2025-11-15

    2025

  26. [26]

    Domenico Cotroneo, Luigi De Simone, Pietro Liguori, Roberto Natella, and Nematollah Bidokhti. 2019. How bad can a bug get? an empirical analysis of software failures in the openstack cloud computing platform. InProceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineerin...

    2019

  27. [27]

    1999.Mathematical methods of statistics

    Harald Cramér. 1999.Mathematical methods of statistics. Vol. 9. Princeton university press

    1999

  28. [28]

    Cursor. 2025. Cursor official website. https://cursor.com/. Accessed: 2025-11-15

    2025

  29. [29]

    Hugo Da Gião, André Flores, Rui Pereira, and Jácome Cunha. 2024. Chronicles of ci/cd: A deep dive into its usage over time.arXiv preprint arXiv:2402.17588(2024)

    Pith/arXiv arXiv 2024

  30. [30]

    Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. 2023. Github copilot ai pair programmer: Asset or liability?Journal of Systems and Software203 (2023), 111734

    2023

  31. [31]

    Mian Muhammad Danyal, Sarwar Shah Khan, Muzammil Khan, Subhan Ullah, Muhammad Bilal Ghaffar, and Wahab Khan. 2024. Sentiment analysis of movie reviews based on NB approaches using TF–IDF and count vectorizer.Social network analysis and mining 14, 1 (2024), 87

    2024

  32. [32]

    Alexandre Decan, Tom Mens, Maëlick Claes, and Philippe Grosjean. 2016. When GitHub meets CRAN: An analysis of inter-repository package dependency problems. In2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER), Vol. 1. IEEE, 493–504

    2016

  33. [33]

    GitHub developer guideline documentation. 2021. GitHub REST API. https://developer.github.com/v3/. Accessed: 2025-09-05

    2021

  34. [34]

    Devin. 2025. Devin official website. https://devin.ai/. Accessed: 2025-11-15

    2025

  35. [35]

    Thomas Dohmke, Marco Iansiti, and Greg Richards. 2023. Sea change in software development: Economic and productivity analysis of the ai-powered developer lifecycle.arXiv preprint arXiv:2306.15033(2023)

    Pith/arXiv arXiv 2023

  36. [36]

    Sabry E Farrag. 2026. The Productivity-Reliability Paradox: Specification-Driven Governance for AI-Augmented Software Development. arXiv preprint arXiv:2605.01160(2026)

    Pith/arXiv arXiv 2026

  37. [37]

    Oscar Fridh and Szymon Stypa. 2022. Classification of pull requests using transformers.LU-CS-EX(2022)

    2022

  38. [38]

    Haoyu Gao, Peerachai Banyongrakkul, Hao Guan, Mansooreh Zahedi, and Christoph Treude. 2026. On Autopilot? An Empirical Study of Human-AI Teaming and Review Practices in Open Source.arXiv preprint arXiv:2601.13754(2026)

    arXiv 2026

  39. [39]

    Georgios Gousios, Martin Pinzger, and Arie van Deursen. 2014. An exploratory study of the pull-based software development model. InProceedings of the 36th international conference on software engineering. 345–355

    2014

  40. [40]

    Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure.arXiv preprint arXiv:2203.05794 (2022)

    Pith/arXiv arXiv 2022

  41. [41]

    Angular Commit Guidelines. 2026. Conventional Commits: A specification for adding human and machine readable meaning to commit messages. https://www.conventionalcommits.org/en/v1.0.0/. Accessed: 2026-04-01

    2026

  42. [42]

    Haryadi S Gunawi, Mingzhe Hao, Tanakorn Leesatapornwongsa, Tiratat Patana-Anake, Thanh Do, Jeffry Adityatama, Kurnia J Eliazar, Agung Laksono, Jeffrey F Lukman, Vincentius Martin, et al. 2014. What bugs live in the cloud? a study of 3000+ issues in cloud systems. InProceedings of the ACM symposium on cloud computing. 1–14

    2014

  43. [43]

    Thomas J Hardcastle and Krystyna A Kelly. 2013. Empirical Bayesian analysis of paired high-throughput sequencing data with a beta-binomial distribution.BMC bioinformatics14, 1 (2013), 135

    2013

  44. [44]

    Ahmed E Hassan, Hao Li, Dayi Lin, Bram Adams, Tse-Hsun Chen, Yutaro Kashiwa, and Dong Qiu. 2025. Agentic Software Engineering: Foundational Pillars and a Research Roadmap.arXiv preprint arXiv:2509.06216(2025)

    Pith/arXiv arXiv 2025

  45. [45]

    Hao He, Courtney Miller, Shyam Agarwal, Christian Kästner, and Bogdan Vasilescu. 2025. Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projects.arXiv preprint arXiv:2511.04427(2025)

    arXiv 2025

  46. [46]

    Runzhi He, Hao He, Yuxia Zhang, and Minghui Zhou. 2023. Automating dependency updates in practice: An exploratory study on github dependabot.IEEE Transactions on Software Engineering49, 8 (2023), 4004–4022

    2023

  47. [47]

    John Healy and Leland McInnes. 2024. Uniform manifold approximation and projection.Nature Reviews Methods Primers4, 1 (2024), 82

    2024

  48. [48]

    Kim Herzig and Andreas Zeller. 2013. The impact of tangled code changes. In2013 10th Working Conference on Mining Software Repositories (MSR). IEEE, 121–130. Proc. ACM Meas. Anal. Comput. Syst., Vol. 37, No. 4, Article 111. Publication date: July 2026. 111:40•Mazloomzadeh et al

    2013

  49. [49]

    Andre Hora and Romain Robbes. 2026. Are Coding Agents Generating Over-Mocked Tests? An Empirical Study.arXiv preprint arXiv:2602.00409(2026)

    arXiv 2026

  50. [50]

    Kosei Horikawa, Hao Li, Yutaro Kashiwa, Bram Adams, Hajimu Iida, and Ahmed E Hassan. 2025. Agentic Refactoring: An Empirical Study of AI Coding Agents.arXiv preprint arXiv:2511.04824(2025)

    arXiv 2025

  51. [51]

    Louis M Hsu and Ronald Field. 2003. Interrater agreement measures: Comments on Kappan, Cohen’s Kappa, Scott’s 𝜋, and Aickin’s 𝛼. Understanding Statistics2, 3 (2003), 205–219

    2003

  52. [52]

    Haoming Huang, Pongchai Jaisri, Shota Shimizu, Lingfeng Chen, Sota Nakashima, and Gema Rodríguez-Pérez. 2026. More Code, Less Reuse: Investigating Code Quality and Reviewer Sentiment towards AI-generated Pull Requests.arXiv preprint arXiv:2601.21276(2026)

    arXiv 2026

  53. [53]

    Peng Huang, William J Bolosky, Abhishek Singh, and Yuanyuan Zhou. 2015. Confvalley: A systematic configuration validation framework for cloud services. InProceedings of the Tenth European Conference on Computer Systems. 1–16

    2015

  54. [54]

    Ajay Kumar Jha, Sunghee Lee, and Woo Jin Lee. 2017. Developer mistakes in writing android manifests: An empirical study of configuration errors. In2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 25–36

    2017

  55. [55]

    Yikun Jiang, Huanyu Wang, Lei Xie, Hanbin Zhao, Chao Zhang, Hui Qian, and John C Lui. 2024. D-llm: A token adaptive computing resource allocation strategy for large language models.Advances in Neural Information Processing Systems37 (2024), 1725–1749

    2024

  56. [56]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770(2023)

    Pith/arXiv arXiv 2023

  57. [57]

    Thomas Joshi, Shayan Chowdhury, and Fatih Uysal. 2025. Swe-bench-cl: Continual learning for coding agents.arXiv preprint arXiv:2507.00014(2025)

    Pith/arXiv arXiv 2025

  58. [58]

    Vigdis By Kampenes, Tore Dybå, Jo E Hannay, and Dag IK Sjøberg. 2007. A systematic review of effect size in software engineering experiments.Information and Software Technology49, 11-12 (2007), 1073–1086

    2007

  59. [59]

    Oleksii Kononenko, Tresa Rose, Olga Baysal, Michael Godfrey, Dennis Theisen, and Bart De Water. 2018. Studying pull request merges: A case study of shopify’s active merchant. InProceedings of the 40th international conference on software engineering: software engineering in practice. 124–133

    2018

  60. [60]

    Naveen Krishnan. 2025. Ai agents: Evolution, architecture, and real-world applications.arXiv preprint arXiv:2503.12687(2025)

    Pith/arXiv arXiv 2025

  61. [61]

    Anand Kumar, Vishal Khare, Deepak Sharma, Satyam Kumar, Vijay Saini, Anshul Yadav, Sachendra Jain, Ankit Rana, Pratham Verma, Vaibhav Meena, et al. 2025. Intuition to Evidence: Measuring AI’s True Impact on Developer Productivity.arXiv preprint arXiv:2509.19708 (2025)

    arXiv 2025

  62. [62]

    Sandra Kumi, Charles Snow, Richard K Lomotey, and Ralph Deters. 2024. Uncovering concerns of citizens through machine learning and social network sentiment analysis.Ieee Access12 (2024), 94885–94913

    2024

  63. [63]

    Saveliy Lavrentiev and Alexey Dukhanov. 2026. Token Cost Optimization in an LLM Agent for JSON-Based System Modeling from Dialogue History. In2026 International Russian Smart Industry Conference (SmartIndustryCon). IEEE, 854–860

    2026

  64. [64]

    Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, et al. 2025. Gemini embedding: Generalizable embeddings from gemini.arXiv preprint arXiv:2503.07891 (2025)

    Pith/arXiv arXiv 2025

  65. [65]

    Valentina Lenarduzzi, Vili Nikkola, Nyyti Saarimäki, and Davide Taibi. 2021. Does code quality affect pull request acceptance? an empirical study.Journal of Systems and Software171 (2021), 110806

    2021

  66. [66]

    Russell Lenth. 2023. emmeans: Estimated Marginal Means, aka Least-Squares Means_.R package version 1.8. 5(2023)

    2023

  67. [67]

    Hao Li, Haoxiang Zhang, and Ahmed E Hassan. 2025. The rise of ai teammates in software engineering (se) 3.0: How autonomous coding agents are reshaping software engineering.arXiv preprint arXiv:2507.15003(2025)

    Pith/arXiv arXiv 2025

  68. [68]

    Shuqing Li, Yechang Wu, Yi Liu, Dinghua Wang, Ming Wen, Yida Tao, Yulei Sui, and Yepang Liu. 2020. An exploratory study of bugs in extended reality applications on the web. In2020 IEEE 31st International symposium on software reliability engineering (ISSRE). IEEE, 172–183

    2020

  69. [69]

    Zhixing Li, Yue Yu, Tao Wang, Gang Yin, Shanshan Li, and Huaimin Wang. 2021. Are you still working on this? An empirical study on pull request abandonment.IEEE Transactions on Software Engineering48, 6 (2021), 2173–2188

    2021

  70. [70]

    Yuhao Liu, Yingnan Zhou, Hanfeng Zhang, Zhiwei Chang, Sihan Xu, Yan Jia, Wei Wang, Juncheng Hu, and Zheli Liu. 2024. Rethinking software misconfigurations in the real world: an empirical study and literature analysis.arXiv preprint arXiv:2412.11121(2024)

    Pith/arXiv arXiv 2024

  71. [71]

    Shan Lu, Soyeon Park, Eunsoo Seo, and Yuanyuan Zhou. 2008. Learning from mistakes: a comprehensive study on real world concurrency bug characteristics. InProceedings of the 13th international conference on Architectural support for programming languages and operating systems. 329–339

    2008

  72. [72]

    Guillermo Macbeth, Eugenia Razumiejczyk, and Rubén Daniel Ledesma. 2011. Cliff’s Delta Calculator: A non-parametric effect size program for two groups of observations.Universitas Psychologica10, 2 (2011), 545–555

    2011

  73. [73]

    Iren Mazloomzadeh, Mohammad Mehdi Morovati, and Foutse Khomh. 2026. Replication package of the paper. https://github.com/ mazloomzadeh/AI-Agent-Analysis. Accessed: 2026-06-01

    2026

  74. [74]

    Mary L McHugh. 2012. Interrater reliability: the kappa statistic.Biochemia medica22, 3 (2012), 276–282

    2012

  75. [75]

    Mary L McHugh. 2013. The chi-square test of independence.Biochemia medica23, 2 (2013), 143–149. Proc. ACM Meas. Anal. Comput. Syst., Vol. 37, No. 4, Article 111. Publication date: July 2026. How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests•111:41

    2013

  76. [76]

    Atif M Memon. 2007. An event-flow model of GUI-based applications for testing.Software testing, verification and reliability17, 3 (2007), 137–157

    2007

  77. [77]

    Xiangrui Meng. 2013. Scalable simple random sampling and stratified sampling. InInternational conference on machine learning. PMLR, 531–539

    2013

  78. [78]

    David Mimno, Hanna Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011. Optimizing semantic coherence in topic models. InProceedings of the 2011 conference on empirical methods in natural language processing. 262–272

    2011

  79. [79]

    Florent Moriconi, Thomas Durieux, Jean-Rémy Falleri, Raphaël Troncy, and Aurélien Francillon. 2025. GHALogs: Large-scale dataset of GitHub Actions runs. In2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE, 669–673

    2025

  80. [80]

    Mohammad Mehdi Morovati, Amin Nikanjam, Foutse Khomh, and Zhen Ming Jiang. 2023. Bugs in machine learning-based systems: a faultload benchmark.Empirical Software Engineering28, 3 (2023), 62

    2023

Showing first 80 references.