pith. machine review for the scientific record. sign in

arxiv: 2604.09388 · v2 · submitted 2026-04-10 · 💻 cs.SE · cs.AI

Recognition: unknown

The AI Codebase Maturity Model: From Assisted Coding to Fully Autonomous Systems

Andy Anderson
Authors on Pith no claims yet

Pith reviewed 2026-05-10 16:55 UTC · model grok-4.3

classification 💻 cs.SE cs.AI
keywords AI-assisted codingmaturity modelfeedback loopsautonomous systemstesting infrastructurecodebase evolutionmulti-agent orchestration
0
0 comments X

The pith

The intelligence of AI coding systems resides in the infrastructure of tests, metrics, and feedback loops that surround the model.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper presents the AI Codebase Maturity Model, a six-level framework that describes how codebases advance from basic AI-assisted coding to fully autonomous operation. Each level is defined by the specific feedback loop topology required before the next stage becomes feasible. Experience maintaining real projects shows that teams cannot skip levels and that adding reliable mechanisms for instructions, testing, and metrics drives progress. The model positions testing coverage and execution reliability as the single most important investment for reaching higher autonomy.

Core claim

The paper claims that the capabilities of an AI-driven development system are determined by the infrastructure of instructions, tests, metrics, and feedback loops rather than by the underlying AI model. It introduces the ACMM with six sequential levels, each enabled by a distinct feedback loop topology, and validates the progression through a 100-day experience report on two projects that achieved 91 percent code coverage, 74 CI/CD workflows, and bug-to-fix times under 30 minutes while reaching full autonomy at Level 6.

What carries the argument

The AI Codebase Maturity Model (ACMM), a six-level framework in which each level is defined by the feedback loop topology that must be present before the next level can be reached.

Load-bearing premise

The six levels represent necessary sequential stages that cannot be skipped, derived from experience with two specific projects.

What would settle it

A documented case in which a team reached full autonomy in AI-driven development without first establishing the feedback mechanisms required by one or more intermediate levels.

read the original abstract

AI coding tools are widely adopted, but most teams plateau at prompt-and-review without a framework for systematic progression. This paper presents the AI Codebase Maturity Model (ACMM), a 6-level framework describing how codebases evolve from basic AI-assisted coding to fully autonomous systems. Inspired by CMMI, each level is defined by its feedback loop topology - the specific mechanisms that must exist before the next level becomes possible. I validate the model through a 100-day experience report maintaining KubeStellar Console, a CNCF Kubernetes dashboard built from scratch with Claude Code (Opus) and GitHub Copilot, and through the initial production deployment of Hive - an open-source multi-agent orchestration system that realizes Level 6: full autonomy. The system currently operates with 74 CI/CD workflows, 32 nightly test suites, 91% code coverage, and achieves bug-to-fix times under 30 minutes - 24 hours a day. The central finding: the intelligence of an AI-driven development system resides not in the AI model itself, but in the infrastructure of instructions, tests, metrics, and feedback loops that surround it. You cannot skip levels, and at each level, the thing that unlocks the next one is another feedback mechanism. Testing - the volume of test cases, the coverage thresholds, and the reliability of test execution - proved to be the single most important investment in the entire journey. v2 extends the model from 5 to 6 levels, adding Level 6 (Fully Autonomous) with Hive as reference implementation and Beads for cross-agent memory continuity, plus throughput acceleration data showing 5x PR throughput and 37x issue throughput from Level 2 to Level 6.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 3 minor

Summary. The paper proposes the AI Codebase Maturity Model (ACMM), a six-level framework (extending a prior five-level version) for the progression of AI-driven codebases from basic assisted coding to fully autonomous operation. Levels are defined by feedback-loop topologies, with the central claim that system intelligence resides in the surrounding infrastructure of instructions, tests, metrics, and loops rather than the underlying AI model, that levels cannot be skipped, and that each successive level is unlocked by adding another feedback mechanism. Validation rests on a 100-day experience report developing KubeStellar Console with Claude Code and Copilot, plus Hive as a Level-6 reference implementation achieving 74 CI/CD workflows, 32 nightly suites, 91% coverage, sub-30-minute bug fixes, and reported throughput gains of 5x PRs and 37x issues from Level 2 to Level 6.

Significance. If the necessity of the sequential progression holds, the model supplies a concrete, actionable roadmap for teams adopting AI coding tools, shifting focus from model capability to testable infrastructure investments (especially testing volume and reliability). The experience report supplies specific, quantitative outcomes (coverage, workflow counts, throughput multipliers) that illustrate potential gains at higher levels and credit the author for documenting a sustained, production-grade trajectory including cross-agent memory via Beads.

major comments (3)
  1. [Validation section (100-day report)] Validation section (100-day KubeStellar/Hive report): the load-bearing claim that 'you cannot skip levels' (abstract, §1, conclusion) is supported only by correlation along one successful trajectory; no attempts to bypass a level, no stalled progress when a mechanism was absent, and no counterexamples or alternative paths are reported, leaving the sequential necessity an untested extrapolation.
  2. [Model definition sections] Model definition (levels and topologies): the six levels and their feedback-loop characterizations are constructed directly from the observed progression in the author's two projects, creating circularity where the model is fitted to the data used to derive it rather than independently validated.
  3. [Throughput data (v2 extension)] Throughput acceleration data: the reported 5x PR and 37x issue gains from Level 2 to Level 6 are presented without baseline controls, confounding factors (team size, project scope changes), or statistical analysis, weakening attribution to the maturity progression itself.
minor comments (3)
  1. [Introduction] The paper cites inspiration from CMMI but provides no explicit mapping or comparison of the ACMM levels to CMMI process areas or maturity dimensions.
  2. [Model sections] Notation for feedback-loop topologies is introduced without a summary table or diagram that would allow readers to compare topologies across the six levels at a glance.
  3. [Conclusion] The experience report would benefit from an explicit limitations subsection acknowledging single-author, single-organization scope and absence of multi-case or controlled validation.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for the constructive and detailed review. We address each major comment below with point-by-point responses. Revisions have been made where they strengthen the manuscript without altering its core claims as an experience report.

read point-by-point responses

  1. Referee: Validation section (100-day KubeStellar/Hive report): the load-bearing claim that 'you cannot skip levels' (abstract, §1, conclusion) is supported only by correlation along one successful trajectory; no attempts to bypass a level, no stalled progress when a mechanism was absent, and no counterexamples or alternative paths are reported, leaving the sequential necessity an untested extrapolation.

    Authors: We agree that the evidence for the non-skippability claim rests on inductive observation from one sustained, successful trajectory rather than controlled experiments or counterexamples. The claim is grounded in the logical dependency that each level adds a distinct feedback mechanism (e.g., reliable test execution at Level 3 is prerequisite for metric-driven self-optimization at Level 4). We have added a dedicated Limitations subsection in the Discussion that explicitly labels the sequential necessity as an extrapolation from observed patterns and calls for future multi-project studies to test bypass attempts. The core claim is retained as a hypothesis derived from the data, with the added caveats. revision: partial

  2. Referee: Model definition (levels and topologies): the six levels and their feedback-loop characterizations are constructed directly from the observed progression in the author's two projects, creating circularity where the model is fitted to the data used to derive it rather than independently validated.

    Authors: The levels were indeed derived iteratively from the KubeStellar Console and Hive projects. To address circularity, we have revised the Model Definition section to (a) explicitly describe the derivation process, (b) map each feedback-loop topology to established concepts in software engineering literature (e.g., test automation, continuous feedback in DevOps), and (c) note that the topologies are intended as generalizable patterns rather than project-specific artifacts. These changes reduce the appearance of post-hoc fitting while preserving the experience-report nature of the work. revision: yes

  3. Referee: Throughput acceleration data: the reported 5x PR and 37x issue gains from Level 2 to Level 6 are presented without baseline controls, confounding factors (team size, project scope changes), or statistical analysis, weakening attribution to the maturity progression itself.

    Authors: The throughput figures are observational metrics collected from the same project across levels and are not the result of a controlled study. We acknowledge the presence of confounders such as growing team proficiency with AI tools and evolving project scope. We have inserted explicit qualifying language in the Results and a new Limitations paragraph stating that the multipliers are indicative rather than causally proven, and that statistical analysis is not applicable to this single-trajectory dataset. The data remain as reported but are now framed with appropriate caveats. revision: partial

standing simulated objections not resolved
  • Absence of deliberate bypass attempts, stalled-progress cases, or counterexamples for the 'cannot skip levels' claim, which would require experimental designs outside the scope of the current single-trajectory experience report.

Circularity Check

1 steps flagged

ACMM 'cannot skip levels' claim is definitional rather than empirically derived from skip attempts

specific steps
  1. self definitional [Abstract]
    "each level is defined by its feedback loop topology - the specific mechanisms that must exist before the next level becomes possible. ... You cannot skip levels, and at each level, the thing that unlocks the next one is another feedback mechanism."

    The levels are defined such that each requires the prior feedback mechanisms to exist before the next becomes possible; the assertion that levels cannot be skipped therefore follows by construction from the model's definitional structure, without requiring separate evidence that skipping was attempted and failed.

full rationale

The paper constructs a 6-level model where progression is explicitly tied to the addition of specific feedback loop mechanisms, with the necessity of the sequence embedded in how the levels are defined. The validation consists of a single 100-day experience report on the author's projects that followed this progressive addition of mechanisms. No independent derivation or counterexamples are provided to establish that the sequence is necessary rather than one possible path. The central claim that intelligence resides in the surrounding infrastructure and that levels cannot be skipped reduces to the model's own definitional assumptions fitted to the observed trajectory.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The model rests on the domain assumption that feedback loop topology is the primary determinant of maturity level, with no free parameters or new physical entities introduced.

axioms (1)
  • domain assumption Feedback loop topology determines the possible maturity level of an AI coding system and prevents skipping stages.
    This organizing principle is stated directly in the abstract as the basis for the six levels.
invented entities (1)
  • Six-level AI Codebase Maturity Model no independent evidence
    purpose: To classify and guide progression of AI-assisted codebases
    The levels are postulated based on the author's experience rather than derived from prior theory.

pith-pipeline@v0.9.0 · 5603 in / 1304 out tokens · 73261 ms · 2026-05-10T16:55:45.531000+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    Capability Maturity Model, Version 1.1,

    M. C. Paulk, B. Curtis, M. B. Chrissis, and C. V. Weber, “Capability Maturity Model, Version 1.1,” IEEE Software, vol. 10, no. 4, pp. 18–27, 1993

    1993

  2. [2]

    CMMI for Development, Version 1.3,

    CMMI Institute, “CMMI for Development, Version 1.3,” Software Engineering Institute, Carnegie Mellon University, 2010

    2010

  3. [3]

    The Challenge of ‘Good Enough’ Software,

    J. Bach, “The Challenge of ‘Good Enough’ Software,” American Programmer, vol. 8, no. 10, 1995

    1995

  4. [4]

    Forsgren, J

    N. Forsgren, J. Humble, and G. Kim, Accelerate: The Science of Lean Software and DevOps. IT Revolution Press, 2018

    2018

  5. [5]

    GitHub Copilot: Your AI pair programmer,

    GitHub, “GitHub Copilot: Your AI pair programmer,” 2021. [Online]. Available: https:// github.com/features/copilot

    2021

  6. [6]

    Claude Code,

    Anthropic, “Claude Code,” 2025. [Online]. Available: https://docs.anthropic.com/en/docs/claude- code

    2025

  7. [7]

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

    C. E. Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” arXiv preprint arXiv:2310.06770, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    The Impact of AI on Developer Productivity: Evidence from GitHub Copilot

    S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot,” arXiv preprint arXiv:2302.06590, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  9. [9]

    Productivity Assessment of Neural Code Completion,

    A. Ziegler et al., “Productivity Assessment of Neural Code Completion,” in Proc. of the 6th ACM SIGPLAN International Symposium on Machine Programming, 2022

    2022

  10. [10]

    MLOps: Continuous delivery and automation pipelines in machine learning,

    Google Cloud, “MLOps: Continuous delivery and automation pipelines in machine learning,”

  11. [11]

    Available: https://cloud.google.com/architecture/mlops-continuous-delivery-and- automation-pipelines-in-machine-learning

    [Online]. Available: https://cloud.google.com/architecture/mlops-continuous-delivery-and- automation-pipelines-in-machine-learning

  12. [12]

    Eghbal, Working in Public: The Making and Maintenance of Open Source Software

    N. Eghbal, Working in Public: The Making and Maintenance of Open Source Software. Stripe Press, 2020

    2020

  13. [13]

    KubeStellar Console,

    KubeStellar Project, “KubeStellar Console,” 2025. [Online]. Available: https:// console.kubestellar.io. Source code: https://github.com/kubestellar/console

    2025

  14. [14]

    W. S. Humphrey, Managing the Software Process. Addison-Wesley, 1989

    1989

  15. [15]

    The Five Levels: From Spicy Autocomplete to the Dark Factory,

    D. Shapiro, “The Five Levels: From Spicy Autocomplete to the Dark Factory,” Jan. 2026. [Online]. Available: https://www.danshapiro.com/blog/2026/01/the-five-levels-from-spicy- autocomplete-to-the-software-factory/

    2026

  16. [16]

    The Dark Software Factory,

    BCG Platinion, “The Dark Software Factory,” 2026. [Online]. Available: https:// www.bcgplatinion.com/insights/the-dark-software-factory

    2026

  17. [17]

    Built by Agents, Tested by Agents, Trusted by Whom?

    Stanford Law School CodeX, “Built by Agents, Tested by Agents, Trusted by Whom?” Feb. 2026. [Online]. Available: https://law.stanford.edu/2026/02/08/built-by-agents-tested-by-agents-trusted-by- whom/

    2026

  18. [18]

    AI-SDLC Maturity Model: Traditional to Autonomous Development,

    ELEKS, “AI-SDLC Maturity Model: Traditional to Autonomous Development,” 2025. [Online]. Available: https://eleks.com/blog/ai-sdlc-maturity-model/

    2025

  19. [19]

    AI Development Maturity Model,

    G. Gondim, “AI Development Maturity Model,” DEV Community, 2025. [Online]. Available: https://dev.to/ggondim/ai-development-maturity-model-4i47

    2025

  20. [20]

    AI Maturity Model for Software Engineering Teams (AI-MM SET),

    Gigacore, “AI Maturity Model for Software Engineering Teams (AI-MM SET),” GitHub, 2025. [Online]. Available: https://github.com/Gigacore/AI-Maturity-Model

    2025

  21. [21]

    Hive: Multi-Agent Orchestration for Autonomous Codebases,

    KubeStellar Project, “Hive: Multi-Agent Orchestration for Autonomous Codebases,” 2026. [Online]. Available: https://github.com/kubestellar/hive

    2026

  22. [22]

    Beads: Memory upgrade for your coding agent,

    S. Yegge, “Beads: Memory upgrade for your coding agent,” 2026. [Online]. Available: https:// github.com/steveyegge/beads

    2026