Pith. sign in

REVIEW 5 major objections 8 minor 107 references

Extension Decisions in Open Source Software Ecosystem

T0 review · 5 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that approximately 65% of new Continuous Integration Actions on GitHub Marketplace replicate functionality that already exists, usually within six months, and that a few first-mover Actions account for most later copies.

desk verdict A useful dataset and a first-cut functional-relation analysis, undermined by an abstract headline statistic the body never derives. read the letter →

arxiv 2507.23168 v1 pith:G3R7IPBD submitted 2025-07-31 cs.SE

Elmira Onagh , Maleknaz Nayebi This is my paper · ORCID
classification cs.SE
keywords softwareecosystemGitHubMarketplaceActionscontinuousintegrationfeatureevolutionfunctionalredundancymigratorybehaviorreleaseplanning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that most new tools in the largest segment of GitHub Marketplace, Continuous Integration Actions, do not add new capabilities: about 65% of new CI Actions reproduce functionality that already exists, usually within six months, while a small set of first-mover Actions supplies the features that later tools copy. To show this, the authors turn each Action into a set of features extracted from its action.yml descriptions, link Actions sharing features in a graph, and timestamp every functionality's first appearance and adoption. If the claim holds, developers can use the published graph to pick the right time to launch a new Action and to find unmet needs, while maintainers can identify and eliminate redundant tools. The result matters because it quantifies, at ecosystem scale, how copying and competition shape open-source automation.

What carries the argument

The central object is a functional-relation graph: each Action is a set of features, Actions are linked when they share features according to four relation types (Independent, Subset, Identical, and Intersect), and features, Actions, and publishers are nodes in a single network. Features are extracted from the mandatory description fields of action.yml files by a fine-tuned large language model with human-verified exemplars, validated by self-consistency checks, and consolidated into unique features via embedding-based similarity at a 0.90 cosine threshold. This machinery does the work: it converts natural-language descriptions into set operations, makes redundancy measurable as set overlap, and lets the authors assign each feature's debut time and adoption trajectory, then classify feature movements using eight adapted migratory behaviors.

What would settle it

Take 100 pairs of Actions the model labels as 'identical' and diff their actual source code: if a large share of those pairs implements different commands or performs different side effects, then the 65% replication rate is an artifact of description wording rather than functional duplication.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the Continuous Integration segment of GitHub Marketplace is consolidating rather than diversifying. Measured across two snapshots, the number of unique features grew by 54.64% (from 10,694 to 16,537) while the number of independent Actions fell by 11.01% and intersecting Actions rose by 15.64%; the share of Actions with identical feature sets dropped sharply (71.49%), which the authors read as redundant tools being absorbed into broader offerings. Treating each Action as a set of features and linking Actions that share features, the paper finds that most new Actions land inside already-occupied feature space and that a small group of 85 early contributors, active before the Marketplace's official 2019 launch, supply the feature sets that later tools reproduce, with one early contributor's two Actions having 38 identical followers. The graph model timestamps every feature's first appearance and tracks its adoption, which is the basis for the abstract's headline numbers: roughly 65% of new CI Actions replicate existing capabilities, typically within six months, and first movers account for most later forks and extensions.

Load-bearing premise

The load-bearing premise is that the feature list a language model extracts from each Action's description, after merging near-identical phrasings at a 0.90 similarity threshold, faithfully represents what the Action actually does, and that comparing each Action's own launch date with a fixed January 2024 snapshot gives comparable evolution windows.

Editorial extensions

If this is right

  • A developer launching a new CI Action can consult the functionality graph to see whether the intended feature set already exists and to pick a launch window before the feature is copied.
  • Marketplace maintainers can use subset and identical relations to spot redundant Actions and either consolidate them into broader tools or retire them.
  • Innovation metrics for the Marketplace should count genuinely new features rather than new Actions, since the majority of new entries recombine existing capabilities.
  • Features that exhibit weak migration or birth are the ones most likely to drive future overlap, so tracking those features gives an early signal of where competition will intensify.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the 65% replication figure is likely sensitive to the 0.90 cosine similarity threshold; recomputing the set relations at 0.80 and 0.95 would show how much of the headline is a measurement choice.
  • My inference: because t0 is each Action's own release date while t1 is fixed in January 2024, the 'within six months' finding should be replicated on cohorts with comparable observation windows before being read as a universal lag.
  • My inference: applying the same functional-relation graph to other Marketplace categories, such as deployment, code review, or testing, would show whether the copy-heavy pattern is specific to CI or a general property of GitHub Marketplace.
  • My inference: the published graph could be mined for feature-saturation signals, so that when every feature in a niche already exists, the next launch is better aimed at an adjacent category, turning this descriptive dataset into a release-planning input.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper studies functional redundancy and evolution in the GitHub Marketplace's Continuous Integration category. It scrapes 6,983 CI Actions, retains 5,006 present at both a per-Action initial-release snapshot t0 and a fixed snapshot t1 (January 2024), extracts features from action.yml description fields using a fine-tuned llama3-8b-8192 model, consolidates features via cosine similarity and crowdsourcing, classifies pairwise relations (Independent, Subset, Identical, Intersect), builds a tripartite network of Actions, features, and providers, and adapts eight migratory behaviors from Sarro et al. The abstract claims that approximately 65% of new Actions replicate existing capabilities, typically within six months, and that a few first-mover Actions account for most later forks and extensions. The authors release the dataset and code.

Significance. If the central claims were properly supported, the paper would be a valuable empirical contribution to software-ecosystem research: it assembles a large public dataset, provides explicit graph-based definitions of functional relations, and offers practical guidance for release timing and platform governance. The strengths include the public data and code repository, the explicit formal definitions in Section 4.1.2, and the use of human refinement alongside LLM extraction. However, the headline quantitative claim (65% replication within six months) is not derived anywhere in the body, and the construct validity of the entire redundancy analysis rests on an LLM feature-extraction pipeline whose validation is currently far too weak to carry that claim.

major comments (5)
  1. [Abstract and §5.1.2] The headline claim that "approximately 65% of new CI Actions replicate existing capabilities, typically within six months" is never derived in the results. Section 5.1.2 reports counts of Independent, Subset, Identical, and Intersect Actions at t0 and t1 (e.g., at t1: 1,261 independent, 364 subset, 410 identical, 3,387 intersect for n=5,006), but these categories are not mutually exclusive and the paper itself notes that Actions can have co-occurring relations. No formula or table computes a per-Action replication rate, and no event-history or time-to-overlap analysis appears anywhere. Given that t0 varies per Action while t1 is fixed, the phrase "typically within six months" is unsupported by any reported statistic. This is load-bearing: the contribution advertised in the abstract is exactly this statistic, so it must either be derived explicitly from the released data or removed from the claims.
  2. [§4.1.1, Definitions 1–5] The central construct of the paper—whether one Action "replicates" another—is operationalized as set overlap of features extracted from developer-written action.yml descriptions and consolidated with a 0.90 cosine similarity threshold. The validation reported in §4.1.1 (manual review of 100 Actions with 17.38% corrections, and an NLI error rate of 4.96%) does not establish that the extracted feature sets are faithful representations of true functionality at both t0 and t1 for all 5,006 Actions. If the model tends to emit generic capabilities such as "checkout code" or "run tests," Definitions 2–5 will mechanically classify unrelated Actions as overlapping, inflating the Intersect and Identical counts. Section 7 acknowledges this threat qualitatively, but the impact on the reported rates is not quantified. Independent held-out validation, ideally against code-level behavior or human annotation at both snapshots, is needed before the redundancy claims can be accepted.
  3. [§4 (t0/t1 design) and §5.1.2] The temporal design compares each Action's initial release (t0, ranging from 2015 to late 2023) with a single fixed t1 (January 2024), so observation windows vary from the 12-week minimum to roughly nine years. The conclusions that "actions evolve to have more intricate interconnected functional relations" and that "developers are recreating existing functionalities" conflate aging effects with ecosystem-level trends: older Actions simply have more time to accumulate features and overlaps. A sensitivity analysis, age-binned comparisons, or a model that controls for Action age is required before the observed increases in features per Action and intersecting Actions can be attributed to ecosystem evolution rather than to the heterogeneous observation windows.
  4. [§5.1.2, Figure 7] The counts of Independent, Subset, Identical, and Intersect Actions are presented and discussed as if they partition the 5,006 Actions, but the relations are defined pairwise and are not exclusive (an Action can be both a subset of one Action and intersect another). The statement that "the number of subset Actions decreased by 27.63%, and identical Actions dropped sharply by 71.49%" is therefore ambiguous. The authors should report exclusive per-Action states (e.g., using a hierarchy: identical, subset, intersect, independent) or, at minimum, a binary independent/non-independent partition, so that aggregate percentages such as the 65% claim have a well-defined denominator.
  5. [§5.3, Figure 9] The migration analysis reports zero instances of Strong Migration and Strong Exodus. Since Strong Migration only requires a feature to persist in its original Action at t1 and also appear in a new Action at t1, the complete absence of this behavior is surprising given that Weak Migration is reported as the most common behavior. The paper should either provide a concrete example or count, or explain the definitional or implementation choices (e.g., the treatment of features in Actions that were created after t0) that make Strong Migration and Strong Exodus impossible under the current t0/t1 scheme.
minor comments (8)
  1. [§4, §5.2, §6.3] The number of publishers is reported inconsistently: 3,867 in Section 5.2, but 3,869 in the abstract, Section 6.3, and the introduction. These values should be reconciled.
  2. [§4] The text says t0 can be "any time between early 2015 to October 2024," but the data were collected as of January 2024; this appears to be a typo for October 2023 and should be corrected.
  3. [Figure 7 caption] The caption states that Independent Actions "have remained relatively stable with a slight increase," but the count decreases from 1,417 at t0 to 1,261 at t1; the caption contradicts the data.
  4. [Figure 10] The correlation matrix is difficult to read because the color scale and many numerical labels are illegible at the printed size; a larger figure or a table of the key correlations would improve readability.
  5. [References [63]] Reference [63] is titled "Extension decisions in open source software ecosystem" in the Journal of Systems and Software (2025), the same title as this manuscript; the relationship between this preprint and the cited journal article should be clarified.
  6. [Abstract and §1] The claim that GitHub Marketplace is expanding by approximately 41% annually is stated without a citation or a derivation from the data; a supporting reference or computation should be provided.
  7. [§5.1.1] The numbers 13,994 and 20,569 are described as the result of LLM-based refinement, but the relationship between these values and the subsequently reported 10,694 and 16,537 unique features is not fully explained; a brief step-by-step count would help.
  8. [Various] There are several typographical inconsistencies, including "Univerity" in the affiliation, "Azur" versus "Azure" in Section 5.2.1, and garbled subscript formatting in Definitions 2–7; these should be corrected in a final pass.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the headline 65% replication statistic is not derived in the body, but that is an unsubstantiated claim rather than a definitional reduction.

full rationale

The paper's derivation chain is self-contained: features are extracted from action.yml descriptions via a fine-tuned LLM (Section 4.1.1), consolidated by cosine similarity, and functional relations are then defined purely as set-theoretic relations over those feature sets (Definitions 2-5, Section 4.1.2). The reported counts (Section 5.1.2) are computed from those definitions, so no fitted parameter is renamed as a prediction. The abstract's claim that 'approximately 65% of new CI Actions replicate existing capabilities, typically within six months' does not appear as a computed result anywhere in the body: the relation counts are non-exclusive (e.g., Actions can be both subset and intersect), no 65% figure is derived, and the t0-to-t1 design uses heterogeneous per-Action intervals with no time-to-overlap analysis. This is a missing result and a reproducibility threat to the headline claim, not a circularity. Self-citations to Saroar and Nayebi [4, 16, 37, 63] support data-collection context and definitions but are not load-bearing for the central feature-overlap analysis, which rests on the paper's own formal definitions and external LLM/embedding tools.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central analysis rests on the LLM-derived feature sets; the 0.90 cosine threshold and the per-Action t0 window are uncalibrated choices that materially determine the redundancy counts. No independent ground truth is provided beyond a 100-Action human review with 17.38% corrections, so the axioms above carry the full weight of the conclusions.

free parameters (3)
  • cosine_similarity_threshold = 0.90
    Used to decide when two LLM-extracted features are the same feature (Section 4.1.1, citing [104]); the redundancy counts are directly sensitive to this threshold, but no sensitivity analysis is provided.
  • minimum_time_difference = 12 weeks
    Actions with less than 12 weeks between initial release and January 2024 were excluded (Section 4); this selection rule affects which Actions are in the 5,006-Action sample and therefore the replication rate.
  • llm_temperature = 0
    Set to zero for consistency during feature extraction (Section 4.1.1); a modeling choice that affects output determinism but is not fitted to data.
assumptions (4)
  • domain assumption action.yml description fields fully and accurately describe an Action's functionality
    Section 4.1.1 selects all description fields from action.yml as the feature source; Section 7 (Threats to Validity) acknowledges that developers may not document fully or correctly.
  • ad hoc to paper The LLM-extracted feature set is a faithful representation of the Action at both t0 and t1
    The whole network model (Definitions 1 to 5) operates on these extracted features; no independent behavioral validation, such as code inspection or execution tests, is provided.
  • ad hoc to paper The 0.90 cosine similarity threshold correctly merges paraphrases of the same feature
    Applied in Section 4.1.1 to consolidate features; taken as 'recommended' from [104] without calibration on this dataset or reporting of how the count changes with threshold.
  • domain assumption Comparing each Action's initial release (t0) with a fixed January 2024 snapshot (t1) yields comparable evolutionary observations
    Section 4 defines t0 individually per Action; windows range from 12 weeks to about nine years, so observed 'evolution' mixes age effects with true change and cannot support the 'six months' timing claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Extension Decisions in Open Source Software Ecosystem." pith.science (2026). https://pith.science/paper/G3R7IPBD

@misc{pith2026250723168,
  author       = {Pith},
  title        = {Pith review of: Extension Decisions in Open Source Software Ecosystem},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G3R7IPBD}},
  note         = {Machine review of arXiv:2507.23168}
}
read the original abstract

GitHub Marketplace is expanding by approximately 41% annually, with new tools; however, many additions replicate existing functionality. We study this phenomenon in the platform's largest segment, Continuous Integration (CI), by linking 6,983 CI Actions to 3,869 providers and mining their version histories. Our graph model timestamps every functionality's debut, tracks its adoption, and clusters redundant tools. We find that approximately 65% of new CI Actions replicate existing capabilities, typically within six months, and that a small set of first-mover Actions accounts for most subsequent forks and extensions. These insights enable developers to choose the optimal moment to launch, target unmet functionality, and help maintainers eliminate redundant tools. We publish the complete graph and dataset to encourage longitudinal research on innovation and competition in software ecosystems, and to provide practitioners with a data-driven roadmap for identifying emerging trends and guiding product strategy.

Figures

Figures reproduced from arXiv: 2507.23168 by the authors.

Figure 1
Figure 1. Actions are developed and shared on GitHub Mar￾ketplace, where they are integrated by others to automate workflows. efficiency and reducing duplication. The prevalence of over￾lapping tools raises critical questions about competition, re￾dundancy, and differentiation in open source ecosystems (4). Studying the GitHub Marketplace thus enables a deeper un￾derstanding of SECO evolution, feature emergence, and the dynam… view at source ↗
Figure 2
Figure 2. A template of action.yaml file encompassing all potential fields and sub-fields. For this project, our emphasis was specifically on collecting data from the description fields. the App Store using text-mining-based network analysis, is one such example. They visualized the associations among categories and applications, presenting both macro-level category networks and micro-level app networks. 3. Data collection De… view at source ↗
Figure 4
Figure 4. Feature extraction methodology using llama3-8b-8192 model fine-tuned using manually labeled sample dataset (1). Additionally, we used cosine similarity and crowd-sourcing to consolidate similar features (2). where 𝐴𝐴𝑖𝑖 𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 𝑖𝑖𝑖𝑖 is a set of all independent tools in the ecosystem. Definition 3 (Subset Relation). Consider tool 𝐴𝐴 with feature set 𝐹𝐹𝐴𝐴 and tool 𝐵𝐵 with feature set 𝐹𝐹𝐵𝐵. We define 𝐴𝐴 is subset of 𝐵𝐵… view at source ↗
Figures from the paper (2 more)
Figure 8
Figure 8. Figure 8: Number of publishers releasing the initial version of their Action over time. As time progressed, more new publish￾ers and Actions joined the marketplace, whereas 85 publishers have been producing Actions ahead of their competition since 2015 (RQ2). Count Count Cumulat…
Figure 12
Figure 12. Figure 12: An example of the value of features and their co￾occurrences and the concept of SuperActions. play a crucial role in shaping the ecosystem, with their fea￾ture sets serving as a foundation for future development. The migratory patterns of features further demonstrate …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

107 extracted references · 72 canonical work pages

  1. [1]

    J. F. Moore, Predators and prey: a new ecology of competition, Harvard business review 71 (3) (1993) 7586

    1993

  2. [2]

    Iansiti, R

    M. Iansiti, R. Levien, Strategy as ecology., Harvard business review 82 (3) (2004) 6878

    2004

  3. [3]

    Manikas, Revisiting software ecosystems research: A longitudi- nal literature study, Journal of Systems and Software 117 (2016) 84– 103

    K. Manikas, Revisiting software ecosystems research: A longitudi- nal literature study, Journal of Systems and Software 117 (2016) 84– 103

    2016

  4. [4]

    S. G. Saroar, W. Ahmed, E. Onagh, M. Nayebi, Github marketplace for automation and innovation in software production, Information and Software Technology 175 (2024) 107522

    2024

  5. [5]

    Santos, E

    R. Santos, E. Constantinou, P. Antonino, J. Bosch, Software engi - neering for systems-of-systems and software ecosystems (2023)

    2023

  6. [6]

    Antonino, D

    P. Antonino, D. -H. Bae, A. Bertolino, F. Lonetti, Report on the 12th acm/ieee international workshop on software engineering for systems-of-systems and software ecosystems -sesos@ icse 2024, ACM SIGSOFT Software Engineering Notes 49 (3) (2024) 53–56

    2024

  7. [7]

    Martin, F

    W. Martin, F. Sarro, Y. Jia, Y. Zhang, M. Harman, A survey of app store analysis for software engineering, IEEE transactions on software engineering 43 (9) (2016) 817–847

    2016

  8. [8]

    Nayebi, H

    M. Nayebi, H. Farahi, G. Ruhe, Which version should be released to app store?, in: 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, 2017, pp. 324–333

    2017

Show all 107 references
  1. [9]

    Rochet, J

    J.-C. Rochet, J. Tirole, Platform competition in two-sided markets, Journal of the European Economic Association 1 (4) (2003) 990– 1029

    2003

  2. [10]

    D. S. Evans, The antitrust economics of two -sided markets, Yale Journal on Regulation 20 (2003) 325

    2003

  3. [11]

    Barbosa12, C

    O. Barbosa12, C. Alves, A systematic mapping study on software ecosystems (2011)

    2011

  4. [12]

    Sutcu, Two-sided markets: Apple’s digital application platform, International Journal of Economics and Financial Issues 9 (1) (2019) 1–7

    C. Sutcu, Two-sided markets: Apple’s digital application platform, International Journal of Economics and Financial Issues 9 (1) (2019) 1–7

    2019

  5. [13]

    Cuadrado, J

    F. Cuadrado, J. C. Dueñas, Mobile application stores: success fac- tors, existing approaches, and future developments, IEEE Commu- nications Magazine 50 (11) (2012) 160–167

    2012

  6. [14]

    L. E. Hestres, App neutrality: Apple’s app store and freedom of expression online, International Journal of Communication 7 (2013) 15

    2013

  7. [15]

    URL https://docs.github.com/en/actions/creatingactions/ publishingactionsingithubmarketplace

    GitHub, Creating actions publishing actions in github marketplace, accessed on February 19, (n.d.). URL https://docs.github.com/en/actions/creatingactions/ publishingactionsingithubmarketplace

  8. [16]

    S. G. Saroar, M. Nayebi, Developers’ perception of github actions: A survey analysis, arXiv preprint arXiv:2303.04084 (2023)

    2023 arXiv

  9. [17]

    Golzadeh, A

    M. Golzadeh, A. Decan, T. Mens, On the rise and fall of ci services in github, in: SANER, IEEE, 2022, p. 662672

    2022

  10. [18]

    T. Mens, M. Claes, P. Grosjean, A. Serebrenik, Studying evolving software ecosystems based on ecological models, in: Evolving soft- ware systems, Springer, 2013, pp. 297–326

    2013

  11. [19]

    Joshua, D

    J. Joshua, D. Alao, S. Okolie, O. Awodele, Software ecosystem: Features, benefits and challenges, International Journal of Advanced Computer Science and Applications 4 (8) (2013)

    2013

  12. [20]

    X. Xie, H. Wang, How can open innovation ecosystem modes push product innovation forward? an fsqca analysis, Journal of Business Research 108 (2020) 29–41

    2020

  13. [21]

    Den Besten, C

    M. Den Besten, C. Amrit, A. Capiluppi, G. Robles, Collaboration and innovation dynamics in software ecosystems: A technology management research perspective, IEEE Transactions on Engineer- ing Management 68 (5) (2020) 1532–1537

    2020

  14. [22]

    Lamba, A

    H. Lamba, A. Trockman, D. Armanios, C. Kästner, H. Miller, B. Vasilescu, Heard it through the gitvine: an empirical study of tool diffusion across the npm ecosystem, in: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the ...

    2020

  15. [23]

    Dabbish, C

    L. Dabbish, C. Stuart, J. Tsay, J. Herbsleb, Leveraging transparency, IEEE software 30 (1) (2012) 37–43

    2012

  16. [24]

    URL https://networkx.org/

    NetworkX Developers, Networkx documentation , accessed on February 19 (n.d.). URL https://networkx.org/

  17. [25]

    E. Yu, S. Deng, Understanding software ecosystems: A strategic modeling approach, in: Iwseco -2011 software ecosystems 2011. proceedings of the third international workshop on software ecosys- tems. brussels, belgium, 2011, pp. 65–76

    2011

  18. [26]

    Sarro, A

    F. Sarro, A. A. AlSubaihin, M. Harman, Y. Jia, W. Martin, Y. Zhang, Feature lifecycles as they spread, migrate, remain, and die in app stores, in: 2015 IEEE 23rd International requirements engineering conference (RE), IEEE, 2015, p. 7685

    2015

  19. [27]

    U. A. Koana, Q. H. Le, S. Rahman, C. Carlson, F. Chew, M. Nayebi, Examining ownership models in software teams: A systematic litera- ture review and a replication study, Empirical Software Engineering 29 (6) (2024) 155. Extension Decisions in Open Source Software Ecosystem Aut...

    2024

  20. [28]

    Ahmed, S

    F. Ahmed, S. Datta, M. Nayebi, Negative results of image processing for identifying duplicate questions on stack overflow, arXiv preprint arXiv:2407.05523 (2024)

    2024 arXiv

  21. [29]

    U. A. Koana, F. Chew, C. Carlson, M. Nayebi, Ownership in the hands of accountability at brightsquid: A case study and a developer survey, in: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2...

    2023

  22. [30]

    Nayebi, B

    M. Nayebi, B. Adams, G. Ruhe, Release practices for mobile apps– what do users and developers think?, in: 2016 ieee 23rd interna - tional conference on software analysis, evolution, and reengineering (saner), Vol. 1, IEEE, 2016, pp. 552–562

    2016

  23. [31]

    M. Nayebi, Eye of the mind: Image processing for social coding, in: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results, 2020, pp. 49–52

    2020

  24. [32]

    Hashemi, M

    Y. Hashemi, M. Nayebi, G. Antoniol, Documentation of machine learning software, in: 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE, 2020, pp. 666–667

    2020

  25. [33]

    Nayebi, L

    M. Nayebi, L. Dicke, R. Ittyipe, C. Carlson, G. Ruhe, Essmart way to manage customer requests, Empirical Software Engineering 24 (2019) 3755–3789

    2019

  26. [34]

    Nayebi, G

    M. Nayebi, G. Ruhe, T. Zimmermann, Mining treatment- outcome constructs from sequential software engineering data, IEEE Trans- actions on Software Engineering 47 (2) (2019) 393–411

    2019

  27. [35]

    Nayebi, Analytical release management for mobile apps, Ph.D

    M. Nayebi, Analytical release management for mobile apps, Ph.D. thesis, PhD thesis, University of Calgary (2018)

    2018

  28. [36]

    Nayebi, Data driven requirements engineering: Implications for the community, in: 2018 IEEE 26th International Requirements Engineering Conference (RE), IEEE, 2018, pp

    M. Nayebi, Data driven requirements engineering: Implications for the community, in: 2018 IEEE 26th International Requirements Engineering Conference (RE), IEEE, 2018, pp. 439–441

    2018

  29. [37]

    S. G. Saroar, W. Ahmed, E. Onagh, M. Nayebi, Github marketplace: Driving automation and fostering innovation in software develop - ment, in: 2025 IEEE 32nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Journal First, 2025

    2025

  30. [38]

    F. Pepe, C. Farkas, M. Nayebi, G. Antoniol, M. Di Penta, How do papers make into machine learning frameworks: A preliminary study on tensorflow, in: 33rd IEEE/ACM International Conference on Program Comprehension (ICPC 2025), 2025

    2025

  31. [39]

    Ahmed, X

    F. Ahmed, X. Tan, O. Adewole, M. Nayebi, Inferring questions from programming screenshots, in: 22nd International Conference on Mining Software Repositories (MSR), 2025

    2025

  32. [40]

    Nayebi, K

    M. Nayebi, K. Kuznetsov, A. Zeller, G. Ruhe, Recommending and release planning of user -driven functionality deletion for mobile apps, Requirements Engineering 29 (4) (2024) 459–480

    2024

  33. [41]

    Nayebi, K

    M. Nayebi, K. Kuznetsov, A. Zeller, G. Ruhe, User driven func- tionality deletion for mobile apps, in: 2023 IEEE 31st International Requirements Engineering Conference (RE), IEEE, 2023, pp. 6–16

    2023

  34. [42]

    Nayebi, H

    M. Nayebi, H. Cho, G. Ruhe, App store mining is not enough for app improvement, Empirical Software Engineering 23 (2018) 2764– 2794

    2018

  35. [43]

    Nayebi, M

    M. Nayebi, M. Marbouti, R. Quapp, F. Maurer, G. Ruhe, Crowd - sourced exploration of mobile app features: A case study of the fort mcmurray wildfire, in: 2017 IEEE/ACM 39th International Con- ference on Software Engineering: Software Engineering in Society Track (ICSE-SEIS), I...

    2017

  36. [44]

    Nayebi, G

    M. Nayebi, G. Ruhe, Analytical product release planning, in: The art and science of analyzing software data, Elsevier, 2015, pp. 555–589

    2015

  37. [45]

    Nayebi, Y

    M. Nayebi, Y. Cai, R. Kazman, G. Ruhe, Q. Feng, C. Carlson, F. Chew, A longitudinal study of identifying and paying down architecture debt, in: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE- SEIP), IEEE, 2019, pp. 171–180

    2019

  38. [46]

    Nayebi, H

    M. Nayebi, H. Cho, H. Farrahi, G. Ruhe, App store mining is not enough, in: 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C), IEEE, 2017, pp. 152– 154

    2017

  39. [47]

    Nayebi, G

    M. Nayebi, G. Ruhe, Asymmetric release planning: Compromising satisfaction against dissatisfaction, IEEE Transactions on Software Engineering 45 (9) (2018) 839–857

    2018

  40. [48]

    Maalej, M

    W. Maalej, M. Nayebi, G. Ruhe, Data- driven requirements engineering-an update, in: 2019 IEEE/ACM 41st International Con- ference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), IEEE, 2019, pp. 289–290

    2019

  41. [49]

    Nayebi, K

    M. Nayebi, K. Kuznetsov, P. Chen, A. Zeller, G. Ruhe, Anatomy of functionality deletion: an exploratory study on mobile apps, in: Proceedings of the 15th International Conference on Mining Software Repositories, 2018, pp. 243–253

    2018

  42. [50]

    Nayebi, G

    M. Nayebi, G. Ruhe, Optimized functionality for super mobile apps, in: RE, IEEE, 2017, p. 388393

    2017

  43. [51]

    G. Ruhe, M. Nayebi, C. Ebert, The vision: Requirements engineering in society, in: 2017 IEEE 25th International Requirements Engineer- ing Conference (RE), IEEE, 2017, pp. 478–479

    2017

  44. [52]

    Nayebi, G

    M. Nayebi, G. Ruhe, R. C. Mota, M. Mufti, Analytics for software project management–where are we and where do we go?, in: 2015 30th IEEE/ACM International Conference on Automated Software Engineering Workshop (ASEW), IEEE, 2015, pp. 18–21

    2015

  45. [53]

    Nayebi, G

    M. Nayebi, G. Ruhe, An open innovation approach in support of product release decisions, in: Proceedings of the 7th International Workshop on Cooperative and Human Aspects of Software Engi - neering, 2014, pp. 64–71

    2014

  46. [54]

    Nayebi, H

    M. Nayebi, H. Farrahi, G. Ruhe, Analysis of marketed versus not- marketed mobile app releases, in: Proceedings of the 4th Interna- tional Workshop on Release Engineering, 2016, pp. 1–4

    2016

  47. [55]

    Nayebi, S

    M. Nayebi, S. J. Kabeer, G. Ruhe, C. Carlson, F. Chew, Hybrid labels are the new measure!, IEEE Software 35 (1) (2017) 54–57

    2017

  48. [56]

    G. Ruhe, M. Nayebi, What counts is decisions, not numbers—toward an analytics design sheet, in: Perspectives on Data Science for Software Engineering, Elsevier, 2016, pp. 111–114

    2016

  49. [57]

    Nayebi, G

    M. Nayebi, G. Ruhe, Analytical open innovation for value-optimized service portfolio planning, in: Software Business. Towards Contin- uous Value Delivery: 5th International Conference, ICSOB 2014, Paphos, Cyprus, June 16-18, 2014. Proceedings 5, Springer, 2014, pp. 273–288

    2014

  50. [58]

    Nayebi, H

    M. Nayebi, H. Farrahi, A. Lee, H. Cho, G. Ruhe, More insight from being more focused: analysis of clustered market apps, in: Proceed- ings of the International Workshop on App Market Analytics, 2016, pp. 30–36

    2016

  51. [59]

    S. J. Kabeer, M. Nayebi, G. Ruhe, C. Carlson, F. Chew, Predicting the vector impact of change-an industrial case study at brightsquid, in: 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, 2017, pp. 131–140

    2017

  52. [60]

    Nayebi, G

    M. Nayebi, G. Ruhe, Trade -off service portfolio planning –a case study on mining the android app market, Tech. rep., PeerJ PrePrints (2015)

    2015

  53. [61]

    X. Tan, D. Yadav, F. Ahmed, M. Nayebi, Imager: Enhancing bug re- port clarity by screenshots, arXiv preprint arXiv:2505.01925 (2025)

    2025 arXiv

  54. [62]

    S. P. Thirimanne, E. Y. Lemango, G. Antoniol, M. Nayebi, One doc- umentation does not fit all: Case study of tensorflow documentation, arXiv preprint arXiv:2505.01939 (2025)

    2025 arXiv

  55. [63]

    Onagh, M

    E. Onagh, M. Nayebi, Extension decisions in open source software ecosystem, Journal of Systems and Software (2025) 112552

    2025

  56. [64]

    Onagh, M

    E. Onagh, M. Nayebi, The impact of foundational models on patient- centric e-health systems

  57. [65]

    P. R. Campbell, F. Ahmed, A three -dimensional view of software ecosystems, in: Proceedings of the Fourth European Conference on Software Architecture: Companion Volume, 2010, pp. 81–84

    2010

  58. [66]

    M. H. Sadi, E. Yu, Analyzing the evolution of software development: From creative chaos to software ecosystems, in: 2014 IEEE eighth international conference on research challenges in information sci- ence (RCIS), IEEE, 2014, pp. 1–11

    2014

  59. [67]

    Bosch, From software product lines to software ecosystems., in: SPLC, Vol

    J. Bosch, From software product lines to software ecosystems., in: SPLC, Vol. 9, 2009, pp. 111–119

    2009

  60. [68]

    Malcher, D

    P. Malcher, D. Viana, P. O. Antonino, R. P. dos Santos, Investigating open innovation practices to support requirements management in Extension Decisions in Open Source Software Ecosystem Author’s preprint on arxiv Page 16 of 16 software ecosystems, in: International Conferenc...

    2023

  61. [69]

    T. Chen, Y. Zhang, S. Chen, T. Wang, Y. Wu, Let’s supercharge the workflows: An empirical study of github actions, in: QRSC, IEEE, 2021, p. 0110

    2021

  62. [70]

    ValenzuelaToledo, A

    P. ValenzuelaToledo, A. Bergel, Evolution of github action work - flows, in: SANER, IEEE, 2022, p. 123127

    2022

  63. [71]

    Decan, T

    A. Decan, T. Mens, P. R. Mazrae, M. Golzadeh, On the use of github actions in software development repositories, in: ICSME, IEEE, 2022, p. 235245

    2022

  64. [72]

    Kinsman, M

    T. Kinsman, M. Wessel, M. A. Gerosa, C. Treude, How do software developers use github actions to automate their workflows?, in: MSR, IEEE, 2021, p. 420431

    2021

  65. [73]

    Khatami, C

    A. Khatami, C. Willekens, A. Zaidman, Catching smells in the act: A github actions workflow investigation, in: 24th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 2024

    2024

  66. [74]

    Valenzuela- Toledo, A

    P. Valenzuela- Toledo, A. Bergel, T. Kehrer, O. Nierstrasz, The hidden costs of automation: An empirical study on github actions workflow maintenance, arXiv preprint arXiv:2409.02366 (2024)

    2024 arXiv

  67. [75]

    Mastropaolo, F

    A. Mastropaolo, F. Zampetti, G. Bavota, M. Di Penta, Toward automatically completing github workflows, in: ICSE, 2024, p. 112

    2024

  68. [76]

    Chandrasekara, P

    C. Chandrasekara, P. Herath, C. Chandrasekara, P. Herath, Introduc- tion to github actions, Hands-on GitHub actions: implement CI/CD with GitHub action workflows for your applications (2021) 1–8

    2021

  69. [77]

    Wessel, J

    M. Wessel, J. Vargovich, M. A. Gerosa, C. Treude, Github actions: the impact on the pull request process, Empirical Software Engineer- ing 28 (6) (2023) 131

    2023

  70. [78]

    Decan, T

    A. Decan, T. Mens, H. O. Delicheh, On the outdatedness of work - flows in the github actions ecosystem, Journal of Systems and Software 206 (2023) 111827

    2023

  71. [79]

    Harman, Y

    M. Harman, Y. Jia, Y. Zhang, App store mining and analysis: Msr for app stores, in: MSR, IEEE, 2012, p. 108111

    2012

  72. [80]

    Vogelsang, From specifications to prompts: On the future of generative large language models in requirements engineering, IEEE Software 41 (5) (2024) 913

    A. Vogelsang, From specifications to prompts: On the future of generative large language models in requirements engineering, IEEE Software 41 (5) (2024) 913

    2024

  73. [81]

    Motger, A

    Q. Motger, A. Miaschi, F. Dell’Orletta, X. Franch, J. Marco, Tfrex: A transformerbased feature extraction method from mobile app reviews, in: 2024 IEEE International Conference on Software Analy- sis, Evolution and Reengineering (SANER), IEEE, 2024, p. 227238

    2024

  74. [82]

    Ebert, P

    C. Ebert, P. Louridas, Generative ai for software practitioners, IEEE Software 40 (4) (2023) 30–38

    2023

  75. [83]

    S. S. Sengar, A. B. Hasan, S. Kumar, F. Carroll, Generative artificial intelligence: a systematic review and applications, Multimedia Tools and Applications (2024) 1–40

    2024

  76. [84]

    Russo, Navigating the complexity of generative ai adoption in software engineering, ACM Transactions on Software Engineering and Methodology 33 (5) (2024) 1–50

    D. Russo, Navigating the complexity of generative ai adoption in software engineering, ACM Transactions on Software Engineering and Methodology 33 (5) (2024) 1–50

    2024

  77. [85]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al., Judging llm-as-a-judge with mt- bench and chatbot arena, Advances in Neural Information Processing Systems 36 (2023) 46595–46623

    2023

  78. [86]

    Seidl, U

    C. Seidl, U. Aßmann, Towards modeling and analyzing variability in evolving software ecosystems, in: Proceedings of the 7th Inter - national Workshop on Variability Modelling of Software-intensive Systems, 2013, pp. 1–8

    2013

  79. [87]

    J. Kim, Y. Park, C. Kim, H. Lee, Mobile application service net- works: Apple’s app store, Service Business 8 (2014) 127

    2014

  80. [88]

    S. G. Saroar, W. Ahmed, M. Nayebi, Github marketplace for practi- tioners and researchers to date: A systematic analysis of the knowl- edge mobilization gap in open source software automation, arXiv preprint arXiv:2208.00332 (2022)

    2022 arXiv

  81. [89]

    URL https://pypi.org/project/beautifulsoup4/

    Python Software Foundation, Beautiful soup documentation, ac - cessed on February 19 (n.d.). URL https://pypi.org/project/beautifulsoup4/

  82. [90]

    URL https://github.com/SecretActionDeveloper/ Extension-Decisions-in-Open-Source-Software-Ecosystem/tree/ main

    SecretActionDeveloper, Extension decisions in open source software ecosystem, accessed: 2024-12-31 (2024). URL https://github.com/SecretActionDeveloper/ Extension-Decisions-in-Open-Source-Software-Ecosystem/tree/ main

    2024

  83. [91]

    Z. Zeng, J. Yu, T. Gao, Y. Meng, T. Goyal, D. Chen, Evaluating large language models at evaluating instruction following, arXiv preprint arXiv:2310.07641 (2023)

    2023 arXiv

  84. [92]

    L. Zhu, X. Wang, X. Wang, Judgelm: Fine-tuned large lan- guage models are scalable judges, arXiv preprint arXiv:2310.17631 (2023)

    2023 arXiv

  85. [93]

    Elangovan, L

    A. Elangovan, L. Liu, L. Xu, S. Bodapati, D. Roth, Considers - the-human evaluation framework: Rethinking human evaluation for generative large language models, arXiv preprint arXiv:2405.18638 (2024)

    2024 arXiv

  86. [94]

    URL https://help.openai.com/en/articles/ 6654000bestpracticesforpromptengineeringwiththeopenaiapi

    OpenAI, Best practices for prompt engineering with the openai api, accessed: 20240814 (2024). URL https://help.openai.com/en/articles/ 6654000bestpracticesforpromptengineeringwiththeopenaiapi

    2024

  87. [95]

    M. Wang, M. Wang, X. Xu, L. Yang, D. Cai, M. Yin, Unleashing chatgpt’s power: A case study on optimizing information retrieval in flipped classrooms via prompt engineering, IEEE Transactions on Learning Technologies (2023)

    2023

  88. [96]

    Zhong, Y

    R. Zhong, Y. Xu, C. Zhang, J. Yu, Leveraging large language model to generate a novel metaheuristic algorithm with crispe framework, Cluster Computing (2024) 135

    2024

  89. [97]

    URL https://groq.com/

    Groq, Groq, accessed: 20240905 (2024). URL https://groq.com/

    2024

  90. [98]

    X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, H. Wang, Large language models for software engineer- ing: A systematic literature review, ACM Transactions on Software Engineering and Methodology 33 (8) (2024) 1–79

    2024

  91. [99]

    Nashaat, J

    M. Nashaat, J. Miller, Towards efficient fine- tuning of language models with organizational data for automated software review, IEEE Transactions on Software Engineering (2024)

    2024

  92. [100]

    J. Shin, C. Tang, T. Mohati, M. Nayebi, S. Wang, H. Hemmati, Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks, arXiv preprint arXiv:2310.10508 (2023)

    2023 arXiv

  93. [101]

    Manakul, A

    P. Manakul, A. Liusie, M. J. Gales, Selfcheckgpt: Zeroresource blackbox hallucination detection for generative large language mod- els, arXiv preprint arXiv:2303.08896 (2023)

    2023 arXiv

  94. [102]

    Face, bertbasenlimeantokens, https://huggingface.co/ sentencetransformers/bertbasenlimeantokens, accessed: 20240905 (2024)

    H. Face, bertbasenlimeantokens, https://huggingface.co/ sentencetransformers/bertbasenlimeantokens, accessed: 20240905 (2024)

    2024

  95. [103]

    Scikitlearn, cosine similarity, https://scikitlearn.org, accessed: 20240905 (2024)

    2024

  96. [104]

    Abbas, A

    M. Abbas, A. Ferrari, A. Shatnawi, E. Enoiu, M. Saadatmand, D. Sundmark, On the relationship between similar requirements and similar software: A case study in the railway domain, Requirements Engineering 28 (1) (2023) 2347

    2023

  97. [105]

    A. A. Hagberg, D. A. Schult, P. J. Swart, Networkx , accessed: 2024-08-28 (2024). URL https://networkx.org/documentation/stable/reference/ algorithms/generated/networkx.algorithms.centrality.degree_ centrality.html

    2024

  98. [106]

    A. A. Hagberg, D. A. Schult, P. J. Swart, Networkx: Connected components, accessed: 2025-02-28 (2024). URL https://networkx.org/documentation/stable/reference/ algorithms/generated/networkx.algorithms.components.connected_ components.html

    2024

  99. [107]

    Pizard, D

    S. Pizard, D. Vallespir, B. Kitchenham, A longitudinal case study on the effects of an evidence- based software engineering training, in: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Software Engineering Education and Train- ing, 2022, pp. 1–13

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.