Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

Reuse of AI agent skills is mostly a one-time copy: half stay untouched after adoption, and the behavioural contract almost never changes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 09:10 UTC pith:XPK23WBM

load-bearing objection First real empirical map of agent skills as reusable artefacts: one-time copy, additive local drift, and a mostly untouched behavioural contract—solid methods, disclosed limits, worth engaging. the 2 major comments →

arxiv 2607.00911 v2 pith:XPK23WBM submitted 2026-07-01 cs.SE

From Registry to Repository: How AI Agent Skills Are Written, Adapted, and Maintained

Haoyu Gao , Jai Lal Lulla , Hong Yi Lin , Sebastian Baltes , Christoph Treude , Mansooreh Zahedi This is my paper
classification cs.SE
keywords AI agent skillsSKILL.mdsoftware reusemaintenancecustomisationSWEBOKagent-based software engineeringdocumentation evolution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

AI coding agents load skills—SKILL.md files with YAML headers and Markdown bodies—as on-demand packages of domain knowledge, workflows, and scripts. Public registries now host tens of thousands of them, so skills have become a real unit of software reuse. This paper treats those skills as engineered artefacts rather than mere agent features. By mining registry skills, personal-use copies in GitHub repositories, and thousands of reuse links between them, the authors show that adoption is usually a near-verbatim one-time copy: most reused skills stay almost identical, more than half receive no later local edits, and when maintenance does happen it is overwhelmingly additive. Customisation mainly rebinds a skill to the host project’s tools and paths; later evolution mostly accretes more inline domain knowledge. Across both processes the behavioural contract—how the skill talks to users, watches runtime state, and recovers from failure—remains almost untouched. The practical upshot is that maintainers should concentrate effort on project-specific bindings, while registries should help consolidate the domain knowledge that individual skills keep re-authoring in isolation.

Core claim

Reuse of AI agent skills is largely a one-time copy operation: most reused skills remain near-verbatim, 53% are never modified after adoption, subsequent local maintenance is overwhelmingly additive, customisation primarily adapts skills to local environments while evolution accretes new inline domain knowledge, and across both a stable behavioural contract (user interaction, runtime monitoring, failure recovery) remains almost untouched.

What carries the argument

Recovered registry-to-repository reuse links (YAML name field plus longest-common-subsequence body similarity) that let the study separate customisation-at-adoption from later local evolution and measure how little the behavioural contract moves.

Load-bearing premise

The name-plus-similarity heuristic correctly recovers genuine reuse links at scale, so the one-time-copy story is not an artefact of missed renames, heavy rewrites, reverse reuse, or unlinked third-party sources.

What would settle it

A large audit of linked pairs that finds many heavy rewrites or reverse-direction copies, or a longitudinal check showing most local copies later re-sync with upstream registry revisions, would collapse the one-time-copy claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Maintainers should prioritise keeping project-specific paths, tools, and activation metadata current rather than rewriting interaction and failure-recovery rules.
  • Registries should offer agent-neutral templates and shared domain-knowledge packs so individual skills stop re-authoring the same facts in isolation.
  • Skill authors should invest most care in the behavioural contract at creation time, because downstream adopters almost never revise it.
  • Tooling that alerts adopters to upstream changes and traces when a skill actually fires would reduce silent drift and mis-activation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Skills may become a distinct dependency class that needs its own supply-chain hygiene (version pins, provenance, vulnerability scanning) beyond ordinary documentation or libraries.
  • The additive growth pattern implies skills will accumulate technical debt unless registries or editors supply pruning and consolidation tools.
  • The under-representation of planning and management knowledge areas suggests a market gap that registry designers or specialised skill authors could fill.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper presents the first large-scale empirical study of AI agent skills treated as engineered software artefacts rather than only agent capabilities. The authors mine 18,463 skills from the skills.sh registry and 23,199 personal-use skills from 5,876 GitHub repositories, recover 3,709 reuse links, classify skills into SWEBOK knowledge areas (Software Construction dominant, long tail elsewhere), induce a six-theme content taxonomy from 180 stratified SKILL.md documents, and code 444 customisation/evolution diffs into six change themes. The central claim is that registry reuse is largely a one-time copy: most recovered adoptions are near-verbatim, about 53% of reused skills receive no local post-adoption change, local maintenance is overwhelmingly additive, customisation mainly re-grounds skills to host projects, evolution accretes inline domain knowledge, and a behavioural contract (user interaction, runtime monitoring, failure recovery) is almost never revised.

Significance. If the results hold, the paper establishes skills as a first-class reuse and maintenance object in agent-based software engineering, with concrete implications for maintainers (focus on project-specific bindings and activation criteria), registries (agent-neutral formats and templates), and researchers (consolidating re-authored domain knowledge; tracing activation). Strengths include the dual-channel corpus at unusual scale, dual-LLM then dual-author gold-standard KA labelling (final human κ 0.85; model vs gold κ 0.82), dual-author thematic coding with stratified sampling of both content and change, Fisher tests with Benjamini–Hochberg correction for customisation vs evolution contrasts, an explicit threats section, and a public replication package. These are the right tools for an observational empirical SE study of a new artefact class.

major comments (2)
  1. Section III-A (Reuse-linkage Construction) and the reuse claims in §VI-A / abstract: the name-field + LCS body-similarity heuristic with threshold 0.1 is the load-bearing bridge from registry to repository. Validation is limited to two 30-pair audits (93% genuine in [0.1,0.3); 0/30 genuine below 0.1). The high-similarity majority (≥0.99 for 70.3% of pairs; 1,841/2,462 near-verbatim) carries the one-time-copy pattern and is robust to boundary error, but absolute reuse rates (16% of personal-use skills linked; 8.5% of centralised skills reused) and the claim that skills “rarely incorporate subsequent upstream updates” are sensitive to missed renames, heavy rewrites, reverse reuse, and other registries. Expand the audit (larger stratified samples across similarity bins and a sample of unlinked name-collisions) or report sensitivity under alternative thresholds/matchers so the reuse-rate fra
  2. Section III-D and Discussion VII-A: evolution is measured as net initial-to-final SKILL.md diffs (with churn fence), not full edit histories. The paper correctly discloses that this collapses intermediate add/remove/revert sequences, yet VII-A presents a strong directional claim that evolution is “overwhelmingly additive” (6.1:1 add:remove among coded key points; overall 2.7:1). Net endpoint growth is consistent with accretion, but the coded add/remove ratio on collapsed diffs can understate temporary churn and overstate pure additive maintenance behaviour. Either (a) re-frame the claim strictly as net accretion of content, or (b) quantify intermediate history on a subsample of the 256 evolution skills (e.g., commit-level add/remove balance) so the maintenance-behaviour interpretation is not stronger than the measurement design.
minor comments (6)
  1. Section III-B / IV / VIII: the gold standard is stratified by the LLM’s predicted KA labels, so reported KA proportions (Figure 2) are classifier-based estimates. The threats section already notes this; a short explicit caveat in the RQ1 results paragraph would prevent readers from treating the percentages as fully human-validated corpus prevalence.
  2. Section III-D: Fisher exact tests with BH correction are reported without accounting for repository clustering or multi-label dependence; the authors already call the comparison exploratory. State that limitation once in the RQ3 results text (not only in methodology) next to Table III so effect-size contrasts are not over-read.
  3. Table I and RQ2.1: conformance rates for optional fields and assets are useful; a one-sentence note on whether multi-file skills (scripts/references present) differ systematically in length or KA would help readers interpret packaging discipline.
  4. Figure 1 workflow diagram and the abstract both use “53%” never modified; body text gives 1,167/2,462 = 47.4% updated (hence ~52.6% never). Align the rounded figure and state the denominator (reused SE skills) wherever the percentage appears.
  5. Section II and implications: related work on AGENTS.md / Claude manifests is well cited; a brief contrast sentence on how SKILL.md progressive disclosure differs from always-on agent manifests would sharpen the novelty claim for readers new to the ecosystem.
  6. Minor polish: consistent hyphenation of “personal-use” / “personal use”; ensure SWEBOK KA names match the official list spelling throughout Figure 2 labels; check that all arXiv-linked related work have stable identifiers in the reference list.

Circularity Check

0 steps flagged

No significant circularity: observational empirical SE study whose quantitative and qualitative claims are measured from mined diffs, commit histories, and coded samples rather than derived by construction from fitted inputs or self-citation chains.

full rationale

The paper is a large-scale mining and qualitative study of AI agent skills (registry vs personal-use corpora, reuse-link recovery, SWEBOK KA classification, thematic content analysis, and open coding of customisation/evolution diffs). Its strongest claims (near-verbatim one-time copy adoption, 53% never modified after adoption, additive maintenance asymmetry, customisation re-grounding vs evolution accreting domain knowledge, stable behavioural contract) are descriptive statistics and coded frequencies computed on recovered pairs and commit histories, not predictions forced by fitted parameters or definitions. The name+LCS linkage heuristic and LLM KA labels are measurement instruments with disclosed validation (boundary audits, gold-standard Cohen’s κ); they do not make the reported rates true by construction. Self-citations are ordinary related-work pointers and are not load-bearing uniqueness or ansatz imports. No self-definitional equations, fitted-input-as-prediction steps, or renaming of known results as novel derivations appear. Score 0 is therefore the correct outcome.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The paper’s load-bearing claims rest on standard empirical-SE methods plus several operational thresholds and domain framing choices. There are no physical free parameters; free parameters are analysis cutoffs that shape which skills and diffs enter the reuse and maintenance stories. Axioms are mostly domain assumptions (SWEBOK as SE map; skills.sh as primary registry; name+similarity as reuse proxy). Invented entities are the induced taxonomies, which are analytical constructs with independent evidence only insofar as dual coding and sample frequencies support them.

free parameters (5)
  • reuse_similarity_threshold = 0.1 (link); 0.99 (verbatim)
    Personal-use skills are linked to registry skills only if name matches and LCS body similarity ≥ 0.1; pairs ≥ 0.99 treated as near-verbatim. Threshold chosen and boundary-validated on small audits, not derived.
  • registry_install_cutoff = 20 installs
    Centralised corpus restricted to skills with ≥20 installs as of 4 March 2026, excluding less-downloaded skills from reuse analysis.
  • personal_use_repo_skill_count_fence = 18 skills / Tukey fence
    Repositories with skill count above Tukey’s fence discarded as aggregators; fence set at 18 skills in the observed distribution.
  • evolution_churn_fence = 340 lines
    Unlinked skills with added+removed lines above Tukey upper fence of 340 lines discarded before evolution sampling.
  • qualitative_sample_sizes_and_strata = 180 content; 444 change samples
    Content analysis uses 10 skills per KA (180 total); change analysis uses quartile-stratified samples sized for 90% CI ±10% (188 customisation + 256 evolution). These design choices determine theme frequencies.
axioms (5)
  • domain assumption SWEBOK v4’s 18 knowledge areas plus a Non-SE bin are an adequate primary classification of what software-engineering knowledge skills encode.
    RQ1 forces each skill into a single primary KA from name/description via LLM prompts (§III-B, §IV).
  • domain assumption skills.sh is a sufficiently representative centralised registry for studying third-party skill reuse, better than star-based scrapers such as SkillMP.
    Stated in data collection and external validity (§III-A, §VIII).
  • ad hoc to paper YAML name plus body LCS similarity recovers genuine adoption from registry to personal-use, and reuse direction is registry→local rather than reverse or multi-hop.
    Core construct for the 3,709/2,462 reuse links and the one-time-copy claim (§III-A, Threats).
  • ad hoc to paper Net initial-to-final SKILL.md diffs adequately represent evolution behaviour for theme coding, even though intermediate add/remove/revert sequences are collapsed.
    Explicit construct-validity threat for RQ3 evolution arm (§III-D, §VIII).
  • domain assumption Dual-author open coding with consensus yields reliable content and change taxonomies for the stratified samples.
    Standard qualitative SE assumption; no Cohen’s κ reported for change content-type field (§III-C/D).
invented entities (2)
  • Six-theme SKILL.md content taxonomy no independent evidence
    purpose: Describe what skills contain (scoping/orchestrating, execution lifecycle, output quality, agent conduct, domain knowledge, user/agent coordination).
    Induced from 180 stratified documents; useful analytical construct, not an external physical entity. Independent evidence is limited to dual coding within this study.
  • Six-theme skill change taxonomy (customisation vs evolution) no independent evidence
    purpose: Categorize how skills are edited after adoption or over local history (metadata, operational specs, behavioural constraints, knowledge/resources, compatibility, presentation).
    Induced from 444 coded diffs; central to the maintenance claims. Evidence is internal qualitative coding plus descriptive/Fisher contrasts.

pith-pipeline@v1.1.0-grok45 · 24370 in / 3780 out tokens · 38857 ms · 2026-07-12T09:10:18.827500+00:00 · methodology

0 comments
read the original abstract

AI coding agents increasingly rely on skills: structured context bundles, typically a SKILL$.$md file with a YAML header and Markdown body, loaded on demand for domain knowledge, workflows, and scripts. Public registries such as skills$.$sh now host tens of thousands of skills, making them an emerging unit of reuse in agent-based software engineering. Yet skills have largely been viewed as agent capabilities rather than software artefacts whose content and evolution shape agent behaviour. We present the first empirical study of AI agent skills as engineered artefacts that are authored, reused, customised and maintained, across public registries and personal-use repositories. We mined 18,463 skills from skills$.$sh and 23,199 personal-use skills from 5,876 GitHub repositories, identifying 3,709 reuse links. LLM-based classification into SWEBOK knowledge areas (KAs) shows Software Construction dominates alongside a long tail of specialised areas. A thematic analysis of 180 skills identifies six content categories. Qualitative coding of 444 modifications reveals six themes, of which reworking operational specifications and adapting knowledge and resources are the primary target of change. Our findings show that reuse is largely a one-time copy operation: most reused skills remain near-verbatim, 53% are never modified after adoption, and subsequent local maintenance is overwhelmingly additive. Customisation primarily adapts skills to local environments, whereas evolution accretes new inline domain knowledge. Across both, a stable behavioural contract - how a skill interacts with users, monitors runtime state, and recovers from failures - remains almost untouched. These results suggest maintenance effort should focus on project-specific bindings, and that registries and tool support should enable consolidating the domain knowledge skills re-author in isolation.

Figures

Figures reproduced from arXiv: 2607.00911 by Christoph Treude, Haoyu Gao, Hong Yi Lin, Jai Lal Lulla, Mansooreh Zahedi, Sebastian Baltes.

Figure 1
Figure 1. Figure 1: Study design: we collect centralised and personal-use skills and recover their reuse links, then answer RQ1 (SWEBOK knowledge-area classification), [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: SWEBOK KA distribution of agent skills and 22.1% of personal-use skills. This is followed by Software Engineering Operations (≈ 5% to 7%), Software Testing (≈ 5% to 6%), and Software Configuration Management (≈ 4% to 9%). The remaining 14 KAs form a long tail. Overall, SE skills concentrate on concrete code authoring and execution, whereas domains concerning planning, management, and professional practices… view at source ↗
Figure 3
Figure 3. Figure 3: SKILL.md token length and header count TABLE I CONFORMANCE AND FEATURE PRESENCE Provision Cent. (%) Pers. (%) Mandatory provisions: conformance rate SKILL.md exists at skill root 100.00 100.00 File opens with valid YAML frontmatter 99.80 99.99 name present and non-empty 99.79 98.37 name matches required char. format 96.35 95.04 name equals parent directory name 97.65 91.23 description present, non-empty, s… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Skillware: A Software Ontology and Engineering Lifecycle for Persistent Behavioral Artifacts

    cs.SE 2026-07 conditional novelty 6.0

    Agent skills should be managed as persistent software units with identity, lifecycle, and engineering structure, not just prompt files.

Reference graph

Works this paper leans on

56 extracted references · 9 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Codereval: A benchmark of pragmatic code generation with generative pre-trained models,

    H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y . Ma, G. Liang, Y . Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” inProceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–12

    2024

  2. [2]

    Codere- viewqa: The code review comprehension assessment for large language models,

    H. Y . Lin, C. Liu, H. Gao, P. Thongtanunam, and C. Treude, “Codere- viewqa: The code review comprehension assessment for large language models,” inFindings of the Association for Computational Linguistics: ACL 2025, 2025, pp. 9138–9166

    2025

  3. [3]

    Repairagent: An autonomous, llm-based agent for program repair,

    I. Bouzenia, P. Devanbu, and M. Pradel, “Repairagent: An autonomous, llm-based agent for program repair,” in2025 IEEE/ACM 47th Interna- tional Conference on Software Engineering (ICSE). IEEE, 2025, pp. 2188–2200

    2025

  4. [4]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,”IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85–105, 2023

    2023

  5. [5]

    Does my readme file need to be updated? exploring llm-based readme maintenance,

    H. Gao, H. Y . Lin, C. Treude, G. Gay, and M. Zahedi, “Does my readme file need to be updated? exploring llm-based readme maintenance,”arXiv preprint arXiv:2603.00489, 2026

    arXiv 2026

  6. [6]

    The Agent Skills Directory — skills.sh,

    “The Agent Skills Directory — skills.sh,” https://www.skills.sh/, [Ac- cessed 28-06-2026]

    2026

  7. [7]

    Agent skills: A data-driven analysis of claude skills for extending large language model functionality,

    G. Ling, S. Zhong, and R. Huang, “Agent skills: A data-driven analysis of claude skills for extending large language model functionality,”arXiv preprint arXiv:2602.08004, 2026

    arXiv 2026

  8. [8]

    Agent skills in the wild: An empirical study of security vulnerabilities at scale,

    Y . Liu, W. Wang, R. Feng, Y . Zhang, G. Xu, G. Deng, Y . Li, and L. Zhang, “Agent skills in the wild: An empirical study of security vulnerabilities at scale,”arXiv preprint arXiv:2601.10338, 2026

    Pith/arXiv arXiv 2026

  9. [9]

    Credential leakage in llm agent skills: A large-scale empirical study,

    Z. Chen, Y . Zhang, Y . Liu, G. Deng, Y . Li, Y . Zhang, J. Ning, L. Y . Zhang, L. Ma, and Z. Li, “Credential leakage in llm agent skills: A large-scale empirical study,”arXiv preprint arXiv:2604.03070, 2026

    Pith/arXiv arXiv 2026

  10. [10]

    Skillsbench: Benchmarking how well agent skills work across diverse tasks,

    X. Li, W. Chen, Y . Liu, S. Zheng, X. Chen, Y . He, Y . Li, B. You, H. Shen, J. Sunet al., “Skillsbench: Benchmarking how well agent skills work across diverse tasks,”arXiv preprint arXiv:2602.12670, 2026

    Pith/arXiv arXiv 2026

  11. [11]

    On the untriviality of trivial packages: An empirical study of npm javascript packages,

    M. A. R. Chowdhury, R. Abdalkareem, E. Shihab, and B. Adams, “On the untriviality of trivial packages: An empirical study of npm javascript packages,”IEEE Transactions on Software Engineering, vol. 48, no. 8, pp. 2695–2708, 2021

    2021

  12. [12]

    Small world with high risks: A study of security threats in the npm ecosystem,

    M. Zimmermann, C.-A. Staicu, C. Tenny, and M. Pradel, “Small world with high risks: A study of security threats in the npm ecosystem,” in28th USENIX Security symposium (USENIX security 19), 2019, pp. 995–1010

    2019

  13. [13]

    An empirical comparison of dependency network evolution in seven software packaging ecosystems,

    A. Decan, T. Mens, and P. Grosjean, “An empirical comparison of dependency network evolution in seven software packaging ecosystems,” Empirical Software Engineering, vol. 24, no. 1, pp. 381–416, 2019

    2019

  14. [14]

    Adapting installation instructions in rapidly evolving software ecosystems,

    H. Gao, C. Treude, and M. Zahedi, “Adapting installation instructions in rapidly evolving software ecosystems,”IEEE Transactions on Software Engineering, 2025

    2025

  15. [15]

    Software documentation issues unveiled,

    E. Aghajani, C. Nagy, O. L. Vega-M ´arquez, M. Linares-V ´asquez, L. Moreno, G. Bavota, and M. Lanza, “Software documentation issues unveiled,” in2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 1199–1210

    2019

  16. [16]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  17. [17]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, 2024

    2024

  18. [18]

    A prompt pattern catalog to enhance prompt engineering with chatgpt,

    J. White, Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. El- nashar, J. Spencer-Smith, and D. C. Schmidt, “A prompt pattern catalog to enhance prompt engineering with chatgpt,”arXiv preprint arXiv:2302.11382, 2023

    Pith/arXiv arXiv 2023

  19. [19]

    ReAct: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR), 2023

    2023

  20. [20]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” inAdvances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  21. [21]

    SWE-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,” inAdvances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  22. [22]

    SWE-bench: Can language models resolve real-world GitHub issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world GitHub issues?” inInternational Conference on Learning Representa- tions (ICLR), 2024

    2024

  23. [23]

    Large language model-based agents for software engineering: A sur- vey,

    J. Liu, K. Wang, Y . Chen, X. Peng, Z. Chen, L. Zhang, and Y . Lou, “Large language model-based agents for software engineering: A sur- vey,”arXiv preprint arXiv:2409.02977, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Model context protocol (MCP): Landscape, security threats, and future research directions,

    X. Hou, Y . Zhao, S. Wang, and H. Wang, “Model context protocol (MCP): Landscape, security threats, and future research directions,” arXiv preprint arXiv:2503.23278, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Model context protocol (MCP) at first glance: Study- ing the security and maintainability of MCP servers,

    M. M. Hasan, H. Li, E. Fallahzadeh, G. K. Rajbahadur, B. Adams, and A. E. Hassan, “Model context protocol (MCP) at first glance: Study- ing the security and maintainability of MCP servers,”arXiv preprint arXiv:2506.13538, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Agent READMEs: An empirical study of context files for agentic coding,

    W. Chatlatanagulchai, H. Li, Y . Kashiwa, B. Reid, K. Thonglek, P. Lee- laprute, A. Rungsawang, B. Manaskasemsak, B. Adams, A. E. Hassan, and H. Iida, “Agent READMEs: An empirical study of context files for agentic coding,”arXiv preprint arXiv:2511.12884, 2025

    arXiv 2025

  27. [27]

    On the use of agentic coding manifests: An empirical study of Claude Code,

    W. Chatlatanagulchai, K. Thonglek, B. Reid, Y . Kashiwa, P. Leelaprute, A. Rungsawang, B. Manaskasemsak, and H. Iida, “On the use of agentic coding manifests: An empirical study of Claude Code,”arXiv preprint arXiv:2509.14744, 2025

    arXiv 2025

  28. [28]

    On the impact of agents.md files on the efficiency of ai coding agents,

    J. L. Lulla, S. Mohsenimofidi, M. Galster, J. M. Zhang, S. Baltes, and C. Treude, “On the impact of agents.md files on the efficiency of ai coding agents,” inProceedings of the 1st Journal Ahead Workshop (JAWs) at the International Conference on Software Engineering (ICSE), 2026, accepted. [Online]. Available: https://arxiv.org/abs/2601.20404

    arXiv 2026

  29. [29]

    An empirical study of usages, updates and risks of third-party libraries in Java projects,

    Y . Wang, B. Chen, K. Huang, B. Shi, C. Xu, X. Peng, Y . Wu, and Y . Liu, “An empirical study of usages, updates and risks of third-party libraries in Java projects,” in2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2020

    2020

  30. [30]

    Backstabber’s knife collection: A review of open source software supply chain attacks,

    M. Ohm, H. Plate, A. Sykosch, and M. Meier, “Backstabber’s knife collection: A review of open source software supply chain attacks,” in International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA). Springer, 2020

    2020

  31. [31]

    SoK: Taxonomy of attacks on open-source software supply chains,

    P. Ladisa, H. Plate, M. Martinez, and O. Barais, “SoK: Taxonomy of attacks on open-source software supply chains,” in2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023

    2023

  32. [32]

    An empirical study of pre-trained model reuse in the Hugging Face deep learning model registry,

    W. Jiang, N. Synovic, M. Hyatt, T. R. Schorlemmer, R. Sethi, Y .- H. Lu, G. K. Thiruvathukal, and J. C. Davis, “An empirical study of pre-trained model reuse in the Hugging Face deep learning model registry,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023

    2023

  33. [33]

    PeaTMOSS: A dataset and initial analysis of pre-trained models in open-source software,

    W. Jiang, J. Yasmin, J. Jones, N. Synovic, J. Kuo, N. Bielanski, Y . Tian, G. K. Thiruvathukal, and J. C. Davis, “PeaTMOSS: A dataset and initial analysis of pre-trained models in open-source software,” in 2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR). IEEE, 2024

    2024

  34. [34]

    Programs, life cycles, and laws of software evolution,

    M. M. Lehman, “Programs, life cycles, and laws of software evolution,” Proceedings of the IEEE, vol. 68, no. 9, pp. 1060–1076, 1980

    1980

  35. [35]

    A large-scale empirical study on code-comment inconsistencies,

    F. Wen, C. Nagy, G. Bavota, and M. Lanza, “A large-scale empirical study on code-comment inconsistencies,” in2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, 2019, pp. 53–64

    2019

  36. [36]

    Docable: Evaluating the executability of software tutorials,

    S. Mirhosseini and C. Parnin, “Docable: Evaluating the executability of software tutorials,” inProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 375–385

    2020

  37. [37]

    Cat- egorizing the content of GitHub README files,

    G. A. A. Prana, C. Treude, F. Thung, T. Atapattu, and D. Lo, “Cat- egorizing the content of GitHub README files,”Empirical Software Engineering, vol. 24, pp. 1296–1327, 2019

    2019

  38. [38]

    Augmenting api documentation with insights from stack overflow,

    C. Treude and M. P. Robillard, “Augmenting api documentation with insights from stack overflow,” inProceedings of the 38th International Conference on Software Engineering, 2016, pp. 392–403

    2016

  39. [39]

    Improving api caveats accessibility by mining api caveats knowledge graph,

    H. Li, S. Li, J. Sun, Z. Xing, X. Peng, M. Liu, and X. Zhao, “Improving api caveats accessibility by mining api caveats knowledge graph,” in 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2018, pp. 183–193

    2018

  40. [40]

    J. W. Tukey,Exploratory Data Analysis. Reading, MA: Addison- Wesley, 1977

    1977

  41. [41]

    Software Engineering Body of Knowledge (SWEBOK) — com- puter.org,

    “Software Engineering Body of Knowledge (SWEBOK) — com- puter.org,” https://www.computer.org/education/bodies-of-knowledge/ software-engineering#about, [Accessed 24-06-2026]

    2026

  42. [42]

    Can llms re- place manual annotation of software engineering artifacts?

    T. Ahmed, P. Devanbu, C. Treude, and M. Pradel, “Can llms re- place manual annotation of software engineering artifacts?” in2025 IEEE/ACM 22nd International Conference on Mining Software Reposi- tories (MSR). IEEE, 2025, pp. 526–538

    2025

  43. [43]

    Interrater reliability: the kappa statistic,

    M. L. McHugh, “Interrater reliability: the kappa statistic,”Biochemia medica, vol. 22, no. 3, pp. 276–282, 2012

    2012

  44. [44]

    Specification - Agent Skills — agentskills.io,

    “Specification - Agent Skills — agentskills.io,” https://agentskills.io/ specification, [Accessed 28-06-2026]

    2026

  45. [45]

    Sampling in software engineering research: A critical review and guidelines,

    S. Baltes and P. Ralph, “Sampling in software engineering research: A critical review and guidelines,”Empirical Software Engineering, vol. 27, no. 4, p. 94, 2022

    2022

  46. [46]

    Recommended steps for thematic synthesis in software engineering,

    D. S. Cruzes and T. Dyba, “Recommended steps for thematic synthesis in software engineering,” in2011 international symposium on empirical software engineering and measurement. IEEE, 2011, pp. 275–284

    2011

  47. [47]

    Self-organizing roles on agile soft- ware development teams,

    R. Hoda, J. Noble, and S. Marshall, “Self-organizing roles on agile soft- ware development teams,”IEEE Transactions on Software Engineering, vol. 39, no. 3, pp. 422–444, 2012

    2012

  48. [48]

    Fisher’s exact test,

    G. J. Upton, “Fisher’s exact test,”Journal of the Royal Statistical Society: Series A (Statistics in Society), vol. 155, no. 3, pp. 395–402, 1992

    1992

  49. [49]

    Controlling the false discovery rate: a practical and powerful approach to multiple testing,

    Y . Benjamini and Y . Hochberg, “Controlling the false discovery rate: a practical and powerful approach to multiple testing,”Journal of the Royal statistical society: series B (Methodological), vol. 57, no. 1, pp. 289–300, 1995

    1995

  50. [50]

    Cohen,Statistical Power Analysis for the Behavioral Sciences, 2nd ed

    J. Cohen,Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates, 1988

    1988

  51. [51]

    On a test of whether one of two random variables is stochastically larger than the other,

    H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,”The annals of mathematical statistics, pp. 50–60, 1947

    1947

  52. [52]

    Dominance statistics: Ordinal analyses to answer ordinal questions

    N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinal questions.”Psychological bulletin, vol. 114, no. 3, p. 494, 1993

    1993

  53. [53]

    EASYTOOL: Enhancing LLM-based agents with concise tool instruction,

    S. Yuan, K. Song, J. Chen, X. Tan, Y . Shen, K. Ren, D. Li, and D. Yang, “EASYTOOL: Enhancing LLM-based agents with concise tool instruction,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), L. Chiruzzo, A. Ritter, and L. Wang...

    2025

  54. [54]

    Skillops: Managing llm agent skill libraries as self-maintaining software ecosystems,

    H. Pu, X. Song, and L. Zhao, “Skillops: Managing llm agent skill libraries as self-maintaining software ecosystems,”arXiv preprint arXiv:2605.13716, 2026

    Pith/arXiv arXiv 2026

  55. [55]

    An empirical study of the non-determinism of chatgpt in code generation,

    S. Ouyang, J. M. Zhang, M. Harman, and M. Wang, “An empirical study of the non-determinism of chatgpt in code generation,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 2, pp. 1–28, 2025

    2025

  56. [56]

    Let me speak freely? a study on the impact of format restrictions on large language model performance

    Z. R. Tam, C.-K. Wu, Y .-L. Tsai, C.-Y . Lin, H.-y. Lee, and Y .-N. Chen, “Let me speak freely? a study on the impact of format restrictions on large language model performance.” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, 2024, pp. 1218–1236

    2024