pith. sign in

arxiv: 2606.24429 · v1 · pith:U3PDU7JYnew · submitted 2026-06-23 · 💻 cs.SE · cs.AI

Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories

Arsham Khosravani , Audris Mockus This is my paper

Pith reviewed 2026-06-25 22:56 UTC · model grok-4.3

classification 💻 cs.SE cs.AI
keywords AI coding agentsopen source repositoriesdetection frameworkcommit analysisprevalence estimationClaude Codepull requestsbot detection
0
0 comments X

The pith

Multi-method detection identifies 30 times more AI coding agent commits than bot-account lookup alone.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a multi-layered detection framework that combines configuration-file scanning, commit-message analysis, author-identity matching, and bot-signature lookup across 180 million Git repositories. It classifies agent traces into four behavioral types and shows that no single method captures most activity. Bot-account lookup recovers only 3.3 percent of the 850,157 Claude Code commits found by the combined methods. The framework is validated with 495 hand labels that supply per-cell precision and Wilson intervals. Commit and pull-request censuses capture nearly disjoint populations performing different work, with commit agents doing maintenance and PR agents doing feature work.

Core claim

Multi-method detection identifies 850,157 Claude Code commits in one snapshot, of which bot-account lookup recovers only 28,154 (3.3 percent), establishing a 30x relative-recall gap. Across snapshots from December 2024 to April 2026, commit-attributed agents generate over 320,000 commits per month, with Claude Code leading (886,122 commits across 17,295 projects) and dominating silent configuration-file-only adoption. Compared with an independent pull-request census, the two channels capture nearly disjoint populations: a PR census misses 79 percent of commit-detected Claude Code adopters and essentially all Codex adopters, and the observed work profile follows deployment and detection mode

What carries the argument

multi-layered detection framework integrating configuration-file scanning, commit-message analysis, author-identity matching, and bot-signature lookup, classifying traces into four behavioral types

If this is right

  • Single-signal prevalence estimates for AI coding agents are biased low by at least a factor of 30.
  • Commit-deployed in-editor agents surface as maintenance work while PR-deployed cloud agents surface as feature work.
  • Silent configuration-file-only adoption accounts for thousands of projects and is invisible to bot or PR methods.
  • Different detection channels capture disjoint agent populations, so no single channel represents overall adoption.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Future prevalence studies should combine multiple signals rather than rely on any one channel.
  • The work profile observed depends more on deployment mode than on the specific agent tool.
  • Extending the four-pattern framework to additional agent types could test whether the 30x gap is general across tools.

Load-bearing premise

The 495 hand-validated labels with per-cell precision and Wilson intervals are representative of the true positive and false positive rates of the four detection patterns across the full 180 million repository corpus and across time snapshots.

What would settle it

Re-running the full multi-method pipeline on a later snapshot and finding that bot-account lookup recovers substantially more than 3.3 percent of the total Claude Code commits would indicate the reported recall gap does not hold.

Figures

Figures reproduced from arXiv: 2606.24429 by Arsham Khosravani, Audris Mockus.

Figure 1
Figure 1. Figure 1: Overview of the multi-method AI-agent detection framework. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Monthly AI-attributed commit volume by agent across V2412–V2604. Total agent activity rises [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) PR counts (AIDev) versus commit counts (this work) per agent, log scale. (b) Devin’s task-type [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Type D configuration-file blob counts per agent, V2412 vs V2604 (log scale). Established conven [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
read the original abstract

Generative AI coding agents are entering the open-source supply chain, yet their diverse and often invisible traces leave their prevalence poorly understood. We introduce a multi-layered detection framework that integrates configuration-file scanning, commit-message analysis, author-identity matching, and bot-signature lookup across World of Code (180M+ Git repositories), classifying agent traces into four behavioral types. No single method captures more than a fraction of activity: multi-method detection identifies 850,157 Claude Code commits in one snapshot, of which bot-account lookup_the signal most adoption studies rely on_recovers only 28,154 (3.3%), a 30x relative-recall gap, so single-signal prevalence estimates are biased low by at least this factor. Every detection pattern is hand-validated (495 labels) with per-cell precision and Wilson confidence intervals. Across snapshots from December 2024 to April 2026, commit-attributed agents generate over 320,000 commits per month; Claude Code leads (886,122 commits across 17,295 projects) and dominates silent, configuration-file-only adoption (21,078 projects). Compared against an independent pull-request census (AIDev), the two channels capture nearly disjoint agent populations_a PR census misses 79% of commit-detected Claude Code adopters and essentially all Codex adopters_and different kinds of work: PR-deployed cloud agents (Codex, Cursor) surface as feature work, while commit-deployed in-editor agents (Claude Code, OpenHands, Aider) surface as maintenance. The observed work profile follows deployment and detection mode rather than the tool itself, so no single channel is representative.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript presents a multi-method detection framework integrating configuration-file scanning, commit-message analysis, author-identity matching, and bot-signature lookup across 180M+ Git repositories in World of Code. It classifies AI coding agent traces into four behavioral types and reports that multi-method detection identifies 850,157 Claude Code commits in one snapshot, while bot-account lookup recovers only 28,154 (3.3%), yielding a 30x relative-recall gap. Every detection pattern is hand-validated on 495 labels with per-cell precision and Wilson confidence intervals. Comparisons to an independent PR census (AIDev) show nearly disjoint populations (PR misses 79% of commit-detected Claude Code adopters) and work profiles tied to deployment mode rather than tool.

Significance. If the validation generalizes, the work demonstrates that single-signal prevalence estimates for generative AI coding agents are biased low by at least an order of magnitude and that commit versus PR channels capture distinct populations and work types. The scale of the census and the explicit multi-method plus hand-validation approach provide a concrete empirical basis for revising adoption studies in software supply chains.

major comments (2)
  1. [Validation / Methods] Validation / Methods section: The sampling frame for the 495 hand-validated labels (stratified random vs. convenience sampling, stratification by detection pattern, repository size, or temporal coverage) is not described. This is load-bearing for the per-cell precision estimates and Wilson intervals that underwrite the 30x recall-gap claim across the full 180M-repository corpus.
  2. [Results] Results, disjoint-population paragraph: The statement that a PR census misses 79% of commit-detected Claude Code adopters requires an explicit description of the overlap calculation, including whether the two datasets cover identical time windows and repositories; without this the disjoint-population claim cannot be evaluated.
minor comments (1)
  1. [Abstract] Abstract: The four behavioral types are referenced but not named or briefly defined; adding one-sentence characterizations would improve standalone readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the validation sampling and overlap calculation. These points improve the transparency of the manuscript. We address each major comment below.

read point-by-point responses

  1. Referee: [Validation / Methods] Validation / Methods section: The sampling frame for the 495 hand-validated labels (stratified random vs. convenience sampling, stratification by detection pattern, repository size, or temporal coverage) is not described. This is load-bearing for the per-cell precision estimates and Wilson intervals that underwrite the 30x recall-gap claim across the full 180M-repository corpus.

    Authors: The referee is correct that the manuscript does not describe the sampling frame for the 495 hand-validated labels. We will revise the Validation / Methods section to provide a complete description of the sampling procedure, including the sampling method (stratified random or otherwise), the stratification variables used (detection pattern, repository size, temporal coverage), and the selection process from the larger set of detected instances. This addition will support evaluation of the per-cell precision estimates and Wilson intervals. revision: yes

  2. Referee: [Results] Results, disjoint-population paragraph: The statement that a PR census misses 79% of commit-detected Claude Code adopters requires an explicit description of the overlap calculation, including whether the two datasets cover identical time windows and repositories; without this the disjoint-population claim cannot be evaluated.

    Authors: We agree that the overlap calculation requires explicit description. We will revise the Results section to detail the overlap computation, including the matching criteria on repositories and agent identifiers, confirmation that both the commit detection and AIDev PR census cover identical time windows (December 2024 to April 2026), and the exact formula yielding the 79% figure (proportion of commit-detected Claude Code adopters without a match in the PR census). revision: yes

Circularity Check

0 steps flagged

No circularity: empirical counts and hand validation are independent of any self-referential derivation

full rationale

The paper performs a multi-method census on an external corpus (World of Code) and reports raw detection counts plus hand-labeled precision on 495 samples with Wilson intervals. No equations, fitted parameters, predictions derived from fits, or load-bearing self-citations appear in the derivation chain. The 30x recall gap is a direct ratio of observed counts (850157 vs 28154), not a constructed quantity. Validation labels are produced by human inspection external to the detection rules, and the AIDev comparison is an independent external census. The representativeness concern raised by the skeptic is a question of sampling validity, not circularity by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Review performed on abstract only; no free parameters, invented entities, or explicit axioms are stated in the provided text.

axioms (1)
  • domain assumption The four detection signals (config files, commit messages, author identity, bot signatures) are reliable indicators of AI agent activity with acceptably low false-positive rates.
    The entire multi-method framework and the 30x gap claim rest on this unstated premise about signal validity.

pith-pipeline@v0.9.1-grok · 5835 in / 1375 out tokens · 24690 ms · 2026-06-25T22:56:08.030848+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 8 canonical work pages · 3 internal anchors

  1. [1]

    The Impact of AI on Developer Productivity: Evidence from GitHub Copilot

    Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. The impact of ai on developer pro- ductivity: Evidence from github copilot.arXiv preprint arXiv:2302.06590, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Composer 2 technical report.arXiv preprint arXiv:2603.24477, 2026

    Cursor Research Team, Aaron Chan, Ahmed Shalaby, Alexander Wettig, Aman Sanger, Andrew Zhai, Anurag Ajay, Ashvin Nair, Charlie Snell, Chen Lu, et al. Composer 2 technical report.arXiv preprint arXiv:2603.24477, 2026

    work page arXiv 2026

  3. [3]

    Replit agent.https://docs.replit.com/replitai/agent, 2024

    Replit. Replit agent.https://docs.replit.com/replitai/agent, 2024. Autonomous app-building agent; announced September 2024. Accessed June 2026

    2024

  4. [4]

    OpenHands: An Open Platform for AI Software Developers as Generalist Agents

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. OpenHands: An open platform for AI software developers as generalist agents, 2024. arXiv:2407.16741

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  5. [5]

    Speed at the cost of quality: How cursor ai increases short-term velocity and long-term complexity in open-source projects

    Hao He, Courtney Miller, Shyam Agarwal, Christian K¨ astner, and Bogdan Vasilescu. Speed at the cost of quality: How cursor ai increases short-term velocity and long-term complexity in open-source projects. InProceedings of the 23rd International Conference on Mining Software Repositories (MSR ’26), 2026. arXiv:2511.04427

    work page arXiv 2026

  6. [6]

    Cybersecurity risks of ai-generated code.Center for Security and Emerging Technology, November 2024

    Jessica Ji, Jenny Jun, Maggie Wu, and Rebecca Gelles. Cybersecurity risks of ai-generated code.Center for Security and Emerging Technology, November 2024. Issue Brief

    2024

  7. [7]

    Agentic Much? Adoption of Coding Agents on GitHub

    Romain Robbes, Th´ eo Matricon, Thomas Degueule, Andre Hora, and Stefano Zacchiroli. Agentic much? adoption of coding agents on github.arXiv preprint arXiv:2601.18341, 2026. 18 Project Pre-adoption (months) Post-adoption (months) −6−5−4−3−2−1 +1 +2 +3 +4 +5 all-hands-ai oh-aci 0 0 0 0 0 41 84 61 125 39 7 jisaf location-map 0 0 0 0 0 20 129 0 0 0 0 runtime...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  8. [8]

    World of code: Enabling a research workflow for mining and analyzing the universe of open source vcs data.Empirical Software Engineering, 26(2):1–42, 2021

    Yuxing Ma, Tapajit Dey, Chris Bogart, Sadika Amreen, Marat Valiev, Adam Tutko, David Kennard, Russell Zaretzki, and Audris Mockus. World of code: Enabling a research workflow for mining and analyzing the universe of open source vcs data.Empirical Software Engineering, 26(2):1–42, 2021

    2021

  9. [9]

    Hao Li, Haoxiang Zhang, and Ahmed E. Hassan. AIDev: Studying AI coding agents on GitHub. In Proceedings of the 23rd International Conference on Mining Software Repositories (MSR ’26), 2026. arXiv:2602.09185; dataset: huggingface.co/datasets/hao-li/AIDev

    work page arXiv 2026

  10. [10]

    Survey reveals AI’s impact on the developer experience.https://github.blog/ news-insights/research/survey-reveals-ais-impact-on-the-developer-experience/, 2023

    GitHub. Survey reveals AI’s impact on the developer experience.https://github.blog/ news-insights/research/survey-reveals-ais-impact-on-the-developer-experience/, 2023. Wakefield Research survey of 500 U.S. developers; reports 92% AI-tool usage. Accessed June 2026

    2023

  11. [11]

    Detecting and characterizing bots that commit code

    Tapajit Dey, Sara Mousavi, Eduardo Ponce, Tanner Fry, Bogdan Vasilescu, Anna Filippova, and Au- dris Mockus. Detecting and characterizing bots that commit code. InIEEE/ACM 17th International Conference on Mining Software Repositories (MSR), pages 209–219, 2020

    2020

  12. [12]

    A ground-truth dataset and clas- sification model for detecting bots in github issue and pr comments.Journal of Systems and Software, 175:110911, 2021

    Mehdi Golzadeh, Alexandre Decan, Damien Legay, and Tom Mens. A ground-truth dataset and clas- sification model for detecting bots in github issue and pr comments.Journal of Systems and Software, 175:110911, 2021

    2021

  13. [13]

    Wiese, Ivanilton Polato, Ana Paula Chaves, and Marco A

    Mairieli Wessel, Bruno Mendes de Souza, Igor Steinmacher, Igor S. Wiese, Ivanilton Polato, Ana Paula Chaves, and Marco A. Gerosa. The power of bots: Characterizing and understanding bots in OSS projects.Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):182:1–182:19, 2018

    2018

  14. [14]

    Identifying auto-generated code by using machine learning techniques

    Kento Shimonaka, Soichi Sumi, Yoshiki Higo, and Shinji Kusumoto. Identifying auto-generated code by using machine learning techniques. InInternational Workshop on Empirical Software Engineering in Practice (IWESEP), 2016

    2016

  15. [15]

    Zero- shot detection of machine-generated codes.arXiv preprint arXiv:2310.05103, 2023

    Xianjun Yang, Kexun Zhang, Haifeng Chen, Linda Petzold, William Yang Wang, and Wei Cheng. Zero- shot detection of machine-generated codes.arXiv preprint arXiv:2310.05103, 2023

    work page arXiv 2023

  16. [16]

    Detecting LLM-generated code with subtle modification by adversarial training.arXiv preprint arXiv:2507.13123, 2025

    Xin Yin, Xinrui Li, Chao Ni, Xiaodan Xu, and Xiaohu Yang. Detecting LLM-generated code with subtle modification by adversarial training.arXiv preprint arXiv:2507.13123, 2025

    work page arXiv 2025

  17. [17]

    An empirical evaluation of GitHub Copilot’s code suggestions

    Nhan Nguyen and Sarah Nadi. An empirical evaluation of GitHub Copilot’s code suggestions. In IEEE/ACM 19th International Conference on Mining Software Repositories (MSR), pages 1–5, 2022

    2022

  18. [18]

    A dataset and an approach for identity resolution of 38 million author IDs extracted from 2b git commits

    Tanner Fry, Tapajit Dey, Andrey Karnauch, and Audris Mockus. A dataset and an approach for identity resolution of 38 million author IDs extracted from 2b git commits. InIEEE/ACM 17th International Conference on Mining Software Repositories (MSR), pages 518–522, 2020

    2020

  19. [19]

    A complete set of related git repositories identified via community detection approaches based on shared commits

    Audris Mockus, Diomidis Spinellis, Zoe Kotti, and Gabriel John Dusing. A complete set of related git repositories identified via community detection approaches based on shared commits. InIEEE/ACM 17th International Conference on Mining Software Repositories (MSR), pages 513–517, 2020. 19

    2020

  20. [20]

    CodeRabbit: AI code reviews.https://www.coderabbit.ai/, 2024

    CodeRabbit. CodeRabbit: AI code reviews.https://www.coderabbit.ai/, 2024. AI code-review agent. Accessed June 2026

    2024

  21. [21]

    Aider: AI pair programming in your terminal.https://github.com/Aider-AI/aider,

    Paul Gauthier. Aider: AI pair programming in your terminal.https://github.com/Aider-AI/aider,

  22. [22]

    Accessed June 2026

    Open-source terminal coding tool. Accessed June 2026

    2026

  23. [23]

    Ecosyste.ms: An open api service for software ecosystems.https://ecosyste.ms/

    Ecosyste.ms. Ecosyste.ms: An open api service for software ecosystems.https://ecosyste.ms/. Repository-metadata index across software forges. Accessed June 2026

    2026

  24. [24]

    Windsurf: The agentic IDE.https://windsurf.com/, 2024

    Codeium. Windsurf: The agentic IDE.https://windsurf.com/, 2024. Agentic IDE, originally released by Codeium. Accessed June 2026

    2024

  25. [25]

    Claude code.https://www.anthropic.com/claude-code, 2025

    Anthropic. Claude code.https://www.anthropic.com/claude-code, 2025. Agentic command-line coding tool; research preview announced February 2025 alongside Claude 3.7 Sonnet. Accessed June 2026

    2025

  26. [26]

    Introducing Codex: A cloud-based software engineering agent.https://openai.com/index/ introducing-codex/, 2025

    OpenAI. Introducing Codex: A cloud-based software engineering agent.https://openai.com/index/ introducing-codex/, 2025. Research preview announced May 16, 2025; powered by codex-1. Accessed June 2026

    2025

  27. [27]

    Gemini cli.https://github.com/google-gemini/gemini-cli, 2025

    Google. Gemini cli.https://github.com/google-gemini/gemini-cli, 2025. Computer software

    2025

  28. [28]

    Dependabot.https://github.com/dependabot, 2019

    GitHub. Dependabot.https://github.com/dependabot, 2019. Automated dependency-update bot. Accessed June 2026

    2019

  29. [29]

    Github copilot coding agent.https://github.blog/changelog/ 2025-09-25-copilot-coding-agent-is-now-generally-available/, 2025

    GitHub. Github copilot coding agent.https://github.blog/changelog/ 2025-09-25-copilot-coding-agent-is-now-generally-available/, 2025. Autonomous SWE agent; announced May 2025, generally available September 2025. Accessed June 2026

    2025

  30. [30]

    Jules: An autonomous coding agent.https://jules.google/, 2025

    Google. Jules: An autonomous coding agent.https://jules.google/, 2025. Asynchronous coding agent from Google Labs; public beta May 2025. Accessed June 2026

    2025

  31. [31]

    Introducing devin, the first AI software engineer.https://cognition.ai/blog/ introducing-devin, 2024

    Cognition. Introducing devin, the first AI software engineer.https://cognition.ai/blog/ introducing-devin, 2024. Autonomous software-engineering agent; announced March 2024. Accessed June 2026

    2024

  32. [32]

    Conventional commits 1.0.0.https://www.conventionalcommits.org/en/v1

    Conventional Commits. Conventional commits 1.0.0.https://www.conventionalcommits.org/en/v1. 0.0/, 2023. Specification for structured commit messages. Accessed June 2026

    2023

  33. [33]

    Audris Mockus and Lawrence G. Votta. Identifying reasons for software changes using historic databases. InProceedings of the International Conference on Software Maintenance (ICSM), pages 120–130, 2000

    2000

  34. [34]

    When do changes induce fixes? InProceed- ings of the 2005 International Workshop on Mining Software Repositories (MSR), pages 1–5, 2005

    Jacek ´Sliwerski, Thomas Zimmermann, and Andreas Zeller. When do changes induce fixes? InProceed- ings of the 2005 International Workshop on Mining Software Repositories (MSR), pages 1–5, 2005

    2005

  35. [35]

    Edwin B. Wilson. Probable inference, the law of succession, and statistical inference.Journal of the American Statistical Association, 22(158):209–212, 1927

    1927

  36. [36]

    Developer certificate of origin, version 1.1.https://developercertificate.org/,

    Linux Foundation. Developer certificate of origin, version 1.1.https://developercertificate.org/,

  37. [37]

    Accessed June 2026. 20

    2026