Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:JP74FV3Krecord.jsonopen to challenge →
read the original abstract
Despite widespread adoption, the impact of AI tools on software development in the wild remains understudied. We conduct a randomized controlled trial (RCT) to understand how AI tools at the February-June 2025 frontier affect the productivity of experienced open-source developers. 16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience. Each task is randomly assigned to allow or disallow usage of early 2025 AI tools. When AI tools are allowed, developers primarily use Cursor Pro, a popular code editor, and Claude 3.5/3.7 Sonnet. Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%--AI tooling slowed developers down. This slowdown also contradicts predictions from experts in economics (39% shorter) and ML (38% shorter). To understand this result, we collect and evaluate evidence for 20 properties of our setting that a priori could contribute to the observed slowdown effect--for example, the size and quality standards of projects, or prior developer experience with AI tooling. Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design.
This paper has not been read by Pith yet.
Forward citations
Cited by 40 Pith papers
-
Flaws in the LLM Automation Narrative
A new code-writing data analysis benchmark shows human experts outperforming a frontier LLM on average with lower performance variance.
-
From Runnable to Shippable: Multi-Agent Test-Driven Development for Generating Full-Stack Web Applications from Requirements
TDDev automates the full TDD loop for web app generation from requirements, delivering 34-48 percentage point quality gains and zero manual intervention in user studies.
-
To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study
AI-generated code requires less maintenance than human code, with humans handling the majority of changes that are mostly feature extensions rather than bug fixes.
-
Using LLMs in Software Design: An Empirical Study of GitHub and A Practitioner Survey
Developers use LLMs like ChatGPT mainly for knowledge acquisition and code generation at the detailed design level, reporting benefits such as better technology selection and early flaw detection alongside limitations...
-
Problem Reductions at Scale: Agentic Integration of Computationally Hard Problems
A harness for AI agents enabled construction of a Rust library with 100+ problem types and 200+ reduction rules for NP-hard problems in three months.
-
Agentic Much? Adoption of Coding Agents on GitHub
Coding agents reached 22-29% adoption in GitHub projects within months of release, with agent-assisted commits larger and focused on features and bug fixes.
-
AI Writes Faster Than Humans Can Review: A Longitudinal Study of an Enterprise 2x Mandate
Longitudinal panel study of 802 developers shows an enterprise AI coding mandate doubled per-capita merged pull requests to 2.09x baseline, with gains associated with AI adoption and accumulated use while review proce...
-
Govern the Repository, Not the Agent: Measuring Ecosystem-Level Risk in AI-Native Software
Study of 930k+ agent PRs shows repository explains ~50% of integration friction variance, with agents concentrating it twice as much as humans (ICC 0.30 vs 0.16) after controls.
-
Life After Benchmark Saturation: A Case Study of CORE-Bench
Using CORE-Bench as a case study, the paper shows that saturated benchmarks can still deliver insights on efficiency, reliability, model-scaffold differences, and human collaboration even after accuracy plateaus, and ...
-
Rule Taxonomy and Evolution in AI IDEs: A Mining and Survey Study
Mixed-methods study creates taxonomy of AI IDE rules from 7310 instances, analyzes evolution drivers, and reports that rule updates raise average artifact compliance from 49.14% to 72.13%.
-
How to Compare the Security of Code Written by Humans to LLM-generated Code
Proposes and open-sources an automated framework for species-fair security comparisons of human-written, LLM-generated, and hybrid code, validated via feasibility study.
-
From Prompting to Verification: How Experience Shapes Vibe Coding Practices
Survey of 162 vibe coders finds perceptions of AI code quality similar across experience levels but motivations, interaction styles, and quality assurance practices diverge, revealing a perception-action gap.
-
Design and Report Benchmarks for Knowledge Work
Proposes a three-step benchmark design method (define work activity, specify tested setting, score work product) derived from work studies and O*NET, demonstrated via three case analyses.
-
One Developer Is All You Need: A Case Study of an AI-Augmented One-Person Squad in a Brownfield Enterprise
Case study reports one staff engineer with four AI agents delivering a four-person-scoped brownfield project in half the planned time under Spec-Driven Development, with high code acceptance and major cost savings.
-
To What Extent Does Agent-generated Code Require Maintenance? An Empirical Study
AI-generated code requires less maintenance than human-written code, mostly involving feature additions by humans rather than bug fixes.
-
Generative artificial intelligence reduces social welfare through model collapse
A game-theoretic model shows that individually rational adoption of generative AI causes model collapse that reduces collective social welfare for important tasks, with habit formation creating spillovers from low-sta...
-
EditFlow: Benchmarking and Optimizing Code Edit Recommendation Systems via Reconstruction of Developer Flows
EditFlow reconstructs temporal developer editing flows from code changes to benchmark and optimize AI code edit recommenders so they align with natural incremental reasoning rather than static snapshots.
-
Adoption and Impact of Command-Line AI Coding Agents: A Study of Microsoft's Early 2026 Rollout of Claude Code and GitHub Copilot CLI
Observational study of Claude Code and GitHub Copilot CLI at Microsoft finds social-network-driven adoption, activity-linked retention, and a persistent 24% lift in merged pull requests among adopters.
-
A Penny for Your Prompts: Experiments Detecting and Mitigating LLM Usage by Survey Respondents
Experiments on 250 participants show LLM-assisted survey responses range from under 10% on Prolific to over 80% on Mechanical Turk, with identifiable characteristics and partial mitigation effects.
-
Pomona: Continuous Code Quality Improvement via Small, Automated Changes at Bloomberg
Pomona automates discovery and repair of small code quality issues via agent skills, achieving 15 of 17 PRs merged with median close time under 2 hours in a one-month Bloomberg team deployment.
-
Faster than the Team, Faster than the Customer: Tool Integration, Collaboration, and Organisational Lag in AI-assisted RE
AI tools for requirements engineering outpace organizational integration and customer processes at the studied firm, so time savings and workflow changes remain concentrated at the individual level.
-
Using Biometrics to Understand AI-Assisted Coding Performance and its Perception
A multisite biometric study finds lower cognitive engagement under AI assistance via EEG and blink rate, with physiological-performance links present only in the non-AI condition.
-
One Developer Is All You Need: A Case Study of an AI-Augmented One-Person Squad in a Brownfield Enterprise
One experienced developer with four AI agents delivered a four-person scoped brownfield project in half the time at over 85% lower staffing cost under a spec-driven process.
-
Assistance to Autonomy: A Systematic Literature Review of Agentic AI across the Software Development Life Cycle
Systematic review of agentic AI in the SDLC finds output verifiability drives industrial adoption in later phases, with Planner-Executor-Reviewer as the dominant pattern, plus a new multi-agent LLM screening pipeline ...
-
A meta-analysis of the effect of generative AI on productivity and learning in programming
Meta-analysis of 23 studies shows moderate productivity gains from GenAI coding assistants (Hedges' g=0.33) but no significant effect on learning (g=0.14).
-
The Productivity-Reliability Paradox: Specification-Driven Governance for AI-Augmented Software Development
The Productivity-Reliability Paradox arises because AI code generators produce variable output while developers lack sufficient specification discipline, making governance models focused on specifications the binding ...
-
More Is Different: Toward a Theory of Emergence in AI-Native Software Ecosystems
AI-native software ecosystems exhibit emergent behaviors best explained by complex adaptive systems theory, requiring new ecosystem-level monitoring and seven testable propositions that may extend or replace Lehman's laws.
-
Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems
Claude Code centers on a model-tool while-loop surrounded by permission systems, context compaction, extensibility hooks, subagent delegation, and session storage; the same design questions yield different answers in ...
-
Beyond Functional Correctness: Design Issues in AI IDE-Generated Large-Scale Projects
AI IDEs with structured guidance can produce functional large-scale code but frequently introduce design flaws such as duplication, complexity, and principle violations that risk long-term maintainability.
-
The Fast and Spurious: Developer Productivity with GenAI
Survey of 415 developers finds GenAI accelerates coding output but redistributes effort into review and verification, making net productivity gains appear spurious at current adoption levels.
-
Vibe Coding in Product Teams: Reconfiguring AI-Assisted Workflows, Prototyping, and Collaboration
Interviews reveal a four-stage vibe coding workflow that accelerates prototyping while introducing tensions between quick efficiency and reflective design intention, plus asymmetries in trust and ownership.
-
How to Compare the Security of Code Written by Humans to LLM-generated Code
Proposes and validates via feasibility study an open-source automated framework for reproducible, species-fair security comparisons of human-written, LLM-generated, and hybrid code.
-
At What Cost? Software Developers' Well-Being in the Age of GenAI
The paper proposes shifting GenAI research in software engineering from narrow performance metrics to also include developer well-being, social context, and sustainable productivity.
-
Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Development
Agentic Agile-V uses Agile-V as backbone and a Specify-Constrain-Orchestrate-Prove-Evolve-Verify loop to convert AI agent conversations into traceable engineering artifacts with acceptance evidence.
-
LLM-Assisted Repository-Level Generation with Structured Spec-Driven Engineering
Structured specifications as LLM inputs can make high-quality repository-level code generation feasible with better verifiability than natural language prompts.
-
IACDM: Interactive Adversarial Convergence Development Methodology -- A Structured Framework for AI-Assisted Software Development
IACDM is an 8-phase methodology using external verification agents and three pillars to close the verification gap in stochastic LLM-based software development.
-
From Determinism to Delegation: AI-Native Software Engineering and the Evolution of the Agentic Engineer
Software engineering is undergoing a paradigm shift to AI-native practices centered on agentic systems rather than traditional code.
-
The Rise of AI-Native Software Engineering: Implications for Practice, Education, and the Future Workforce
A systematic review of 48 papers on AI in software engineering synthesizes evidence into frameworks for AI-native practice, a nine-dimension competency model, a four-phase curriculum roadmap, and an agenda of research...
-
The Agentic Web Requires New Normative Infrastructure
The agentic web requires new normative infrastructure of laws, norms, and practices to allow user-delegated AI agents to access online properties without being blocked as malicious bots.
-
Reshaping Undergraduate Computer Science Education in the Generative AI Era
The paper recommends shifting CS education from implementation skills to AI-native competencies, fundamental concepts, and critical evaluation based on workshop consensus.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.