pith. sign in

arxiv: 2606.19042 · v1 · pith:PFZKQNPDnew · submitted 2026-06-17 · 💻 cs.SE · cs.AI

Where Did the Variability Go? From Vibe Coding to Product Lines by Regeneration

Xhevahire T\"ernava This is my paper

Pith reviewed 2026-06-26 20:04 UTC · model grok-4.3

classification 💻 cs.SE cs.AI
keywords variabilityvibe codingsoftware product linesLLM generationbinding timeregenerationderivation engine
0
0 comments X

The pith

Variability in LLM-generated programs is resolved entirely at generation time rather than left in the compiled code.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper studies what happens to traditional software variability when an LLM generates complete programs from natural language prompts in the emerging practice called vibe coding. An analysis of ten C/C++ projects found near-zero variability remaining inside the generated artifacts at compile time or runtime. All decisions are instead fixed at the single moment the model produces the source code. The authors introduce Variability by Regeneration as a product-line method in which the LLM itself generates a dedicated, dead-code-free binary for each variant from a declarative specification, while a dispatcher routes requests to the correct binary. This relocates variability management from the code artifacts into the initial specification.

Core claim

In vibe coding, an exploratory study of 10 projects reveals near-zero in-artifact variability. All variability decisions are bound at generation time when the LLM produces the source code. Variability by Regeneration treats the LLM as a derivation engine that produces a dedicated binary for each variant from a declarative specification, with a variant dispatcher handling routing, offering an alternative to classical software product line derivation.

What carries the argument

Variability by Regeneration (VbR), where the LLM serves as the derivation engine that generates a purpose-built binary for each variant from a declarative specification.

Load-bearing premise

The near-zero in-artifact variability observed in the ten projects is caused by the generation-time binding rather than by project selection or prompt style.

What would settle it

Observing substantial compile-time conditionals or runtime variability mechanisms in a larger or differently sampled collection of vibe-coded projects would falsify the central claim.

Figures

Figures reproduced from arXiv: 2606.19042 by Xhevahire T\"ernava.

Figure 1
Figure 1. Figure 1: The VbR pipeline instantiated for the wc product family. 3.3 Instantiating VbR: The wc Product Family To demonstrate VbR, we use the Unix wc (word count) utility [16], simple to show in full yet rich on variability. Its implementation is in the replication package.2 Specification. The top of [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
read the original abstract

In vibe coding, an emerging AI-driven paradigm, an LLM generates an entire program from a natural language prompt, but what happens to the variability that traditional software engineering carefully builds into code? To answer this question, we conducted an exploratory analysis on 10 vibe coded C/C++ projects, which suggests that there is near-zero in-artifact variability, i.e., at compile and runtime. All variability decisions are resolved at a single new binding time, generation time, the moment the LLM produces the source code. Rather than treating this as a defect to fix, we propose Variability by Regeneration (VbR), to our knowledge the first product-line approach in which the LLM acts as the derivation engine, generating a purpose-built, free of dead code binary for each variant from a declarative specification, while a variant dispatcher transparently routes user requests to the matching binary. We formalise VbR, contrast it with classical SPL derivation, and demonstrate its full pipeline on a wc product family. For SPL engineering, variability in AI-generated software belongs in the specification, not in the code.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper claims that in 'vibe coding' (LLM generation of entire programs from natural-language prompts), near-zero in-artifact variability is observed at compile and runtime because all variability decisions are resolved at a new 'generation time' binding point. This is supported by an exploratory analysis of 10 C/C++ projects. The authors propose Variability by Regeneration (VbR), a product-line approach in which the LLM serves as the derivation engine to produce purpose-built, dead-code-free binaries from a declarative specification, routed by a variant dispatcher; they formalize VbR, contrast it with classical SPL derivation, and demonstrate the full pipeline on a wc product family, concluding that variability belongs in the specification rather than the generated code.

Significance. If the core observation generalizes and VbR proves practical, the work could reframe SPL engineering for AI-generated software by relocating variability decisions to the prompt/specification layer and leveraging regeneration for derivation. The wc demonstration and formal contrast with classical approaches provide a concrete starting point and credit the conceptual novelty of treating the LLM itself as the derivation mechanism.

major comments (3)
  1. [Exploratory analysis section] Exploratory analysis of 10 projects: the reported near-zero in-artifact variability lacks any stated project selection criteria, quantitative metrics (e.g., variability measures, error rates), or contrast conditions (different prompt styles or traditional implementations), so the attribution of the outcome specifically to generation-time binding rather than selection or prompting effects remains unisolated and correlational. This observation is load-bearing for the motivation and central claim of VbR.
  2. [VbR formalization] VbR formalization and contrast with classical SPL: the proposal that the LLM acts as derivation engine is presented without addressing how the variant dispatcher is itself implemented or whether it introduces new sources of variability or runtime overhead; this is load-bearing for the claim that VbR yields 'free of dead code' binaries in a practical product-line setting.
  3. [wc demonstration] wc product-family demonstration: the pipeline is shown for a single example but provides no details on declarative-specification construction, prompt templates for regeneration, or quantitative comparison of generated variants against a baseline SPL, limiting assessment of whether the approach scales beyond the illustrative case.
minor comments (2)
  1. [Abstract] The abstract's phrasing 'to our knowledge the first product-line approach' would benefit from a brief literature positioning statement to avoid appearing as an unsubstantiated novelty claim.
  2. [Introduction or formalization section] Notation for binding times (generation time vs. compile/runtime) is introduced informally; a small table contrasting the three would improve clarity for readers unfamiliar with SPL terminology.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on our exploratory study and VbR proposal. We address each major comment point-by-point below.

read point-by-point responses

  1. Referee: [Exploratory analysis section] Exploratory analysis of 10 projects: the reported near-zero in-artifact variability lacks any stated project selection criteria, quantitative metrics (e.g., variability measures, error rates), or contrast conditions (different prompt styles or traditional implementations), so the attribution of the outcome specifically to generation-time binding rather than selection or prompting effects remains unisolated and correlational. This observation is load-bearing for the motivation and central claim of VbR.

    Authors: The analysis is explicitly exploratory, intended to surface the phenomenon rather than provide causal proof. Project selection was based on availability of vibe-coded C/C++ projects in public repositories; we will document the specific repositories and selection process in the revision. We will also incorporate basic quantitative metrics, such as the number of conditional compilation directives and runtime configuration options observed in the artifacts. However, as the study does not include contrast conditions, we will emphasize the correlational nature and the need for future controlled studies. This constitutes a partial revision focused on transparency rather than altering the core claim. revision: partial

  2. Referee: [VbR formalization] VbR formalization and contrast with classical SPL: the proposal that the LLM acts as derivation engine is presented without addressing how the variant dispatcher is itself implemented or whether it introduces new sources of variability or runtime overhead; this is load-bearing for the claim that VbR yields 'free of dead code' binaries in a practical product-line setting.

    Authors: We will revise the formalization section to specify that the variant dispatcher is a lightweight, non-variable component (e.g., a thin wrapper using command-line arguments or environment variables to select the appropriate pre-generated binary). This implementation does not embed feature variability or dead code, as each binary is specialized at generation time. Regarding overhead, we will note that any routing cost is minimal and constant, independent of the number of variants, unlike traditional SPLs with runtime variability. This addresses the practicality concern without changing the central idea. revision: yes

  3. Referee: [wc demonstration] wc product-family demonstration: the pipeline is shown for a single example but provides no details on declarative-specification construction, prompt templates for regeneration, or quantitative comparison of generated variants against a baseline SPL, limiting assessment of whether the approach scales beyond the illustrative case.

    Authors: We agree that additional details are required. In the revision, we will provide the full declarative specification for the wc product family, sample prompt templates used for regeneration, and a quantitative comparison including metrics such as binary size reduction and feature inclusion rates compared to a baseline using traditional preprocessor-based variability. This will better illustrate the pipeline and support claims of practicality. revision: yes

Circularity Check

0 steps flagged

No circularity; observational interpretation and proposal remain independent of inputs

full rationale

The paper reports an exploratory analysis of 10 projects that observed near-zero in-artifact variability and interprets the pattern as decisions being resolved at generation time. It then proposes and formalizes Variability by Regeneration as a new product-line approach. No equations, fitted parameters, or self-citations appear in the provided text as load-bearing steps that reduce the central claim to its own definitions or prior outputs. The attribution is presented as a data-driven suggestion rather than a derivation that loops back by construction, and the VbR proposal is a distinct conceptual contribution.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no free parameters, axioms, or invented entities are identifiable from the provided text.

pith-pipeline@v0.9.1-grok · 5716 in / 1097 out tokens · 26124 ms · 2026-06-26T20:04:35.798523+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 25 canonical work pages · 1 internal anchor

  1. [1]

    In: Proc

    Acher, M., Duarte, J.A.G., Jézéquel, J.: On programming variability with large language model-based assistant. In: Proc. SPLC. pp. 8–14. ACM (2023).https: //doi.org/10.1145/3579027.3608972

    work page doi:10.1145/3579027.3608972 2023

  2. [2]

    In: Proc

    Acher, M., Martinez, J.: Generative AI for reengineering variants into software product lines: An experience report. In: Proc. SPLC. pp. 57–66. ACM (2023). https://doi.org/10.1145/3579028.3609016

    work page doi:10.1145/3579028.3609016 2023

  3. [3]

    Claude Code Documentation (2025),https://code.claude.com/docs/en/memory

    Anthropic: How Claude remembers your project: CLAUDE.md files. Claude Code Documentation (2025),https://code.claude.com/docs/en/memory

    2025

  4. [4]

    Springer (2013).https://doi.org/10.100 7/978-3-642-37521-7

    Apel, S., Batory, D., Kästner, C., Saake, G.: Feature-Oriented Software Product Lines: Concepts and Implementation. Springer (2013).https://doi.org/10.100 7/978-3-642-37521-7

    2013

  5. [5]

    Berger, T., Nair, D., Rublack, R., Atlee, J.M., Czarnecki, K., Wąsowski, A.: Three cases of feature-based variability modeling in industry. pp. 302–319. Springer (2014).https://doi.org/https://doi.org/10.1007/978-3-319-11653-2_19

    work page doi:10.1007/978-3-319-11653-2_19 2014

  6. [6]

    Online software (2024),https://busybox.net/

    BusyBox: BusyBox: The Swiss army knife of embedded Linux. Online software (2024),https://busybox.net/

    2024

  7. [7]

    In: Capilla, R., Bosch, J., Kang, K.C

    Capilla, R., Bosch, J.: Binding time and evolution. In: Capilla, R., Bosch, J., Kang, K.C. (eds.) Systems and Software Variability Management: Concepts, Tools and Experiences, pp. 57–73. Springer (2013).https://doi.org/10.1007/978-3-642 -36583-6_4

    work page doi:10.1007/978-3-642 2013

  8. [8]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H.P., Kaplan, J., Ed- wards, H., Burda, Y., Joseph, N., Brockman, G., et al.: Evaluating large language models trained on code (2021).https://doi.org/10.48550/arXiv.2107.03374

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2107.03374 2021

  9. [9]

    In: Proc

    Chou, Y., Jiang, B., Chen, Y.W., Weng, M., Jackson, V., Zimmermann, T., Jones, J.A.: Building software by rolling the dice: A qualitative study of vibe coding. In: Proc. ESEC/FSE (2026).https://doi.org/10.48550/arXiv.2512.22418

    work page doi:10.48550/arxiv.2512.22418 2026

  10. [10]

    Addison-Wesley (2001)

    Clements, P., Northrop, L.: Software Product Lines: Practices and Patterns. Addison-Wesley (2001)

    2001

  11. [11]

    AI-generated code: A large-scale study of defects, vulnerabilities, and complexity

    Cotroneo, D., Improta, C., Liguori, P.: Human-written vs. AI-generated code: A large-scale study of defects, vulnerabilities, and complexity. In: Proc. ISSRE. pp. 252–263. IEEE (2025).https://doi.org/10.1109/ISSRE66568.2025.00035

    work page doi:10.1109/issre66568.2025.00035 2025

  12. [12]

    Cursor Documentation (2025),https://cursor.com/docs/rules

    Cursor Inc.: Rules: system-level instructions for Agent. Cursor Documentation (2025),https://cursor.com/docs/rules

    2025

  13. [13]

    Addison-Wesley (2000)

    Czarnecki, K., Eisenecker, U.W.: Generative Programming: Methods, Tools, and Applications. Addison-Wesley (2000)

    2000

  14. [14]

    In: Proc

    Czarnecki, K., Grünbacher, P., Rabiser, R., Schmid, K., Wąsowski, A.: Cool fea- tures and tough decisions: A comparison of variability modeling approaches. In: Proc. 6th International Workshop on Variability Modelling of Software-Intensive Systems (VaMoS). pp. 173–182. ACM (2012).https://doi.org/10.1145/211014 7.2110167

    work page doi:10.1145/211014 2012

  15. [15]

    Fawzy, A., Tahir, A., Blincoe, K.: Vibe coding in practice: Motivations, challenges, and a future outlook–a grey literature review (2025).https://doi.org/10.48550 /arXiv.2510.00328

    arXiv 2025

  16. [16]

    FreeSoftwareFoundation:GNUCoreutils:CoreGNUutilities.FreeSoftwareFoun- dation (2026),https://www.gnu.org/software/coreutils/, version 9.10

    2026

  17. [17]

    September 2024–August 2025 data Xh

    GitHub: Octoverse: A new developer joins GitHub every second as AI leads Type- Scrip to#1 (2025),https://github.blog/news-insights/octoverse/octovers e-a-new-developer-joins-github-every-second-as-ai-leads-typescript-t o-1/, accessed: 2025-12-11. September 2024–August 2025 data Xh. Tërnava

    2025

  18. [18]

    In: Proc

    Greiner, S., Schmid, K., Berger, T., Krieter, S., Meixner, K.: Generative AI and software variability–A research vision. In: Proc. VaMoS. pp. 71–76. ACM (2024). https://doi.org/10.1145/3634713.3634722

    work page doi:10.1145/3634713.3634722 2024

  19. [20]

    In: IEEE/ACM 20th Interna- tional Conference on Mining Software Repositories (MSR)

    Jesse, K., Ahmed, T., Devanbu, P.T., Morgan, E.: Large language models and simple, stupid bugs. In: Proc. MSR. pp. 563–575. IEEE (2023).https://doi.or g/10.1109/MSR59073.2023.00082

    work page doi:10.1109/msr59073.2023.00082 2023

  20. [21]

    In: Proc

    Ji, W., Berger, T., Antkiewicz, M., Czarnecki, K.: Maintaining feature traceability with embedded annotations. In: Proc. SPLC. pp. 61–70. ACM (2015).https: //doi.org/10.1145/2791060.2791107

    work page doi:10.1145/2791060.2791107 2015

  21. [22]

    Kang,K.C.,Cohen,S.G.,Hess,J.A.,Novak,W.E.,Peterson,A.S.:Feature-oriented domain analysis (FODA) feasibility study. Tech. Rep. CMU/SEI-90-TR-021, Soft- ware Engineering Institute (1990).https://doi.org/10.21236/ADA235785

    work page doi:10.21236/ada235785 1990

  22. [23]

    X (Twitter) (Feb 2025),https://x.com/karpathy/s tatus/1886192184808149383

    Karpathy, A.: Vibe coding. X (Twitter) (Feb 2025),https://x.com/karpathy/s tatus/1886192184808149383

    arXiv 2025

  23. [24]

    CoRR abs/2112.07279(2021).https://doi.org/10.48550/arXiv.2112.07279, jour- nal reference: Journal of Systems and Software, 2023

    Lesoil, L., Acher, M., Blouin, A., Jézéquel, J.: The interaction between in- puts and configurations fed to software systems: An empirical study. CoRR abs/2112.07279(2021).https://doi.org/10.48550/arXiv.2112.07279, jour- nal reference: Journal of Systems and Software, 2023

    work page doi:10.48550/arxiv.2112.07279 2021

  24. [25]

    In: Proc

    Liebig, J., Apel, S., Lengauer, C., Kästner, C., Schulze, M.: An analysis of the variability in forty preprocessor-based software product lines. In: Proc. ICSE. pp. 105–114 (2010).https://doi.org/10.1145/1806799.1806819

    work page doi:10.1145/1806799.1806819 2010

  25. [26]

    Luu, D.: The growth of command line options, 1979–present.https://danluu.c om/cli-complexity/(2020)

    1979

  26. [27]

    Online software (2006),https://code.videolan.org/videolan/x264

    Merritt, L., Vanam, R.: x264: A high performance H.264/AVC encoder. Online software (2006),https://code.videolan.org/videolan/x264

    2006

  27. [28]

    IEEE Access13, 213242–213259 (2025)

    Meske, C., Hermanns, T., von der Weiden, E., Loser, K., Berger, T.: Vibe cod- ing as a reconfiguration of intent mediation in software development: Defini- tion, implications, and research agenda. IEEE Access13, 213242–213259 (2025). https://doi.org/10.1109/ACCESS.2025.3645466

    work page doi:10.1109/access.2025.3645466 2025

  28. [29]

    In: Proc

    Mikkonen, T., Taivalsaari, A.: Software reuse in the generative AI era: From cargo cult towards systematic practices. In: Proc. Internetware. pp. 541–544. ACM (2025).https://doi.org/10.1145/3755881.3755981

    work page doi:10.1145/3755881.3755981 2025

  29. [30]

    SAILFISH: Vetting Smart Contract State-Inconsistency Bugs in Seconds,

    Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., Karri, R.: Asleep at the key- board? Assessing the security of GitHub Copilot’s code contributions. In: Proc. IEEE Symposium on Security and Privacy (SP). pp. 754–768. IEEE (2022). https://doi.org/10.1109/SP46214.2022.9833571

    work page doi:10.1109/sp46214.2022.9833571 2022

  30. [31]

    Pimenova, V., Fakhoury, S., Bird, C., Storey, M., Endres, M.: Good vibrations? A qualitative study of co-creation, communication, flow, and trust in vibe coding (2025).https://doi.org/10.48550/arXiv.2509.12491

    work page doi:10.48550/arxiv.2509.12491 2025

  31. [32]

    Springer (2005).https://doi.org/10.1 007/3-540-28901-1

    Pohl, K., Böckle, G., van der Linden, F.J.: Software Product Line Engineering: Foundations, Principles, and Techniques. Springer (2005).https://doi.org/10.1 007/3-540-28901-1

    2005

  32. [33]

    Computers14(3), 94 (2025).https: //doi.org/10.3390/computers14030094

    Sonkin, V., Tudose, C.: Beyond snippet assistance: A workflow-centric framework for end-to-end AI-driven code generation. Computers14(3), 94 (2025).https: //doi.org/10.3390/computers14030094

    work page doi:10.3390/computers14030094 2025

  33. [34]

    Online (2025),https://su rvey.stackoverflow.co/2025/, 49,000+ responses from 177 countries Where Did the Variability Go?

    Stack Overflow: Stack overflow developer survey 2025. Online (2025),https://su rvey.stackoverflow.co/2025/, 49,000+ responses from 177 countries Where Did the Variability Go?

    2025

  34. [35]

    In: Proc

    Stümpfle, C., Atray, S., Jazdi, N., Weyrich, M.: Large language model assisted transformation of software variants into a software product line. In: Proc. ICSR. pp. 12–20. IEEE (2025).https://doi.org/10.1109/ICSR66718.2025.00008

    work page doi:10.1109/icsr66718.2025.00008 2025

  35. [36]

    https://superwhisper.com/(2025)

    Superultra Inc.: Superwhisper: AI voice-to-text for macOS, Windows, and iOS. https://superwhisper.com/(2025)

    2025

  36. [37]

    Software—Practice and Experience35(8), 705–754 (2005).https: //doi.org/10.1002/spe.652

    Svahnberg, M., van Gurp, J., Bosch, J.: A taxonomy of variability realization techniques. Software—Practice and Experience35(8), 705–754 (2005).https: //doi.org/10.1002/spe.652

    work page doi:10.1002/spe.652 2005

  37. [38]

    In: SAC SE Symposium (2023), hAL Id: hal-03916459

    Tërnava, X., Acher, M., Combemale, B.: Specialization of run-time configuration space at compile-time: An exploratory study. In: SAC SE Symposium (2023), hAL Id: hal-03916459

    2023

  38. [39]

    In: Proc

    Tërnava, X., Collet, P.: On the diversity of capturing variability at the imple- mentation level. In: Proc. SPLC (Volume B). pp. 81–88. ACM (2017).https: //doi.org/10.1145/3109729.3109733

    work page doi:10.1145/3109729.3109733 2017

  39. [40]

    In: Proc

    Tërnava, X., Collet, P.: Tracing imperfectly modular variability in software product line implementation. In: Proc. ICSR. pp. 112–120. Springer (2017).https://doi. org/10.1007/978-3-319-56856-0_8

    work page doi:10.1007/978-3-319-56856-0_8 2017

  40. [41]

    HAL preprint (2025),https://hal.scienc e/hal-05097580, HAL Id: hal-05097580

    Tërnava, X., Randrianaina, G.A., Lesoil, L., Acher, M.: Small yet configurable: Unveiling null variability in software. HAL preprint (2025),https://hal.scienc e/hal-05097580, HAL Id: hal-05097580

    2025

  41. [42]

    Watanabe, K., Shirai, T., Kashiwa, Y., Iida, H.: What to cut? Predicting unnec- essary methods in agentic code generation (2026).https://doi.org/10.48550/a rXiv.2602.17091

    work page doi:10.48550/a 2026

  42. [43]

    X (Twitter) (Mar 2025),https://x.com/petergyang/status/18967931 72489155048

    Yang, P.: Vibe coding a zombie survival FPS with Cursor + Sonnet 3.7 + Super- whisper. X (Twitter) (Mar 2025),https://x.com/petergyang/status/18967931 72489155048

    arXiv 2025

  43. [44]

    In: Proc

    Zine,M.,Quinton,C.,Rouvoy,R.:LLM-basedco-evolutionofconfigurablesoftware systems. In: Proc. SPLC. pp. 27–38. ACM (2025).https://doi.org/10.1145/37 44915.3748460

    work page doi:10.1145/37 2025