Pith. sign in

REVIEW 5 major objections 7 minor 251 references

Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report

T0 review · 5 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An experience report argues that the choice of LLM-based multi-agent framework barely affects task output quality, because five reimplementations of the same pipeline yield statistically equal ROUGE scores.

desk verdict A useful qualitative framework comparison whose quantitative headline is contradicted by its own significance tests. read the letter →

arxiv 2608.11965 v1 pith:YEA3SVXI submitted 2026-08-12 cs.SE

Mariama Celi Serafim De Oliveira , Motunrayo Osatohanmen Ibiyo , Marco Gianrusso , Claudio Di Sipio , Davide Di Ruscio , Phuong T. Nguyen This is my paper · ORCID
classification cs.SE
keywords multi-agentsystemslargelanguagemodelssoftwareengineeringframeworkcomparisonREADMEsummarizationROUGEmixed-methodsstudyexperiencereport
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Through a mixed-method study, the paper tries to establish that the choice among popular open-source frameworks for building LLM-based multi-agent systems has little effect on the quality of the system's output. The authors qualitatively analyze sixteen frameworks against a ten-feature taxonomy of multi-agent capabilities, then reimplement the same README summarization pipeline on four frameworks (plus a chat-based variant) and find the resulting ROUGE scores statistically indistinguishable. If this is right, practitioners can choose a framework based on documentation, orchestration style, and monitoring support rather than expected output quality. The study also finds that advanced capabilities such as agent telemetry and benchmarking remain largely missing, while execution time varies significantly across frameworks.

What carries the argument

The load-bearing machinery is the standardized reimplementation of a reference multi-agent pipeline, called Metagente in the paper, across the frameworks under comparison. The pipeline assigns four agent roles—an Extractor, a Summarizer, a Teacher, and a Prompt Creator—that run an iterative prompt-optimization loop capped at 15 iterations with a success threshold of 0.7 on ROUGE-L, followed by a two-agent evaluation stage. By keeping the same LLM models, temperatures, and dataset across all ports, the study isolates the framework as the experimental variable; the Wilcoxon signed-rank test, a non-parametric test for paired samples, then converts the ROUGE score distributions into pairwise significance judgments. Qualitatively, the analysis is carried by a ten-feature taxonomy of multi-agent capabilities—core architecture, role specification, tool support, monitoring, human feedback, and related dimensions—applied to each framework's documentation and feature set.

What would settle it

Replace the per-framework adapted prompts with one identical, carefully worded prompt set and rerun the optimization pipeline on the same datasets across all frameworks; if significant pairwise ROUGE differences appear under identical prompts, the paper's no-difference conclusion is an artifact of prompt adaptation rather than a fact about the frameworks. A simpler check is to inspect the published per-framework prompts: if they differ in output-format constraints, few-shot examples, or task instructions, the standardization premise is already violated.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the frameworks' effectiveness on a common multi-agent task is statistically indistinguishable. The authors reimplemented the same prompt-optimization pipeline for GitHub README summarization on four open-source frameworks (and a fifth chat-based variant), holding the underlying LLMs, temperatures, iteration limits, and the ROUGE-L threshold fixed. Pairwise Wilcoxon signed-rank tests over ROUGE-1, ROUGE-2, and ROUGE-L distributions mostly fail to reach significance, with two of the frameworks behaving comparably. The paper couples this with a qualitative finding that the frameworks cover foundational multi-agent capabilities well—role specification, orchestration, message handling—while advanced features such as telemetry, monitoring, and benchmarking remain missing.

Load-bearing premise

The quantitative comparison assumes that the five reimplementations are faithful, behaviorally equivalent ports of the same multi-agent logic, so the 'minor modifications' to prompts and the varying levels of framework support do not differentially affect the measured ROUGE scores; if the ports differ in implementation quality, the no-significant-difference result could be an artifact of the reimplementation rather than a property of the frameworks.

Editorial extensions

If this is right

  • Practitioners can select a multi-agent framework on documentation quality, low-code versus high-code fit, and orchestration style, because output quality on summarization-type tasks does not differentiate the frameworks.
  • Teams building production systems should budget for external telemetry and monitoring, since the study finds advanced observability features largely missing across the selected frameworks.
  • Execution-time variation means framework choice still affects rapid-prototyping speed and running cost, even when output quality is equal.
  • The ten-feature taxonomy provides a reusable checklist for evaluating future multi-agent frameworks as the tooling landscape evolves.
  • The no-difference result invites replication on other software-engineering tasks, such as code generation or test generation, to see whether it holds beyond summarization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the quality-indifference result generalizes beyond summarization, the practical bottleneck in building LLM-based multi-agent systems is likely the prompt design and orchestration logic, not the framework, which would shift developer attention toward composition patterns and away from framework selection.
  • A natural extension is a controlled ablation in which the same agent logic is ported by different development teams; robustness of the no-difference result across implementers would strengthen the interchangeability claim, while sensitivity would caution against over-generalizing.
  • ROUGE scores on short summaries may be too coarse to reveal framework differences; pairing ROUGE with semantic similarity metrics or human judgments could confirm the equivalence or expose differences that n-gram overlap misses.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. This paper is an experience report on LLM-based multi-agent system (MAS) frameworks for software engineering. It combines a qualitative analysis of open-source frameworks (feature and characteristic matrices, thematic analysis) with a quantitative evaluation in which the authors re-implement their previously proposed Metagente README-summarization pipeline in several frameworks (AutoGen, AutoGPT, Dify, Semantic Kernel, plus Semantic Kernel Chat) and compare ROUGE-1/2/L scores on training sets TS10 and TS50 and a test set ES. The abstract and conclusion state that there is no significant difference in ROUGE scores across frameworks, while execution time varies significantly. The paper also provides lessons learned and challenges for practitioners, and a replication package is referenced.

Significance. If the central claim were supported, the paper would be a useful practical guide: practitioners could choose a MAS framework based on qualitative features, documentation, and orchestration fit rather than expected summary quality, because README-summarization effectiveness would be indistinguishable across frameworks. The qualitative framework matrices and lessons learned are valuable resources, and the authors follow a transparent process (contamination audit, normality testing, Wilcoxon tests). However, the quantitative contribution is internally inconsistent: the abstract's 'no significant difference' claim is contradicted by the many significant pairwise p-values in Figure 5, the Metagente baseline promised in Section 4.3 is absent from the reported results, and the claimed efficiency finding is not reported anywhere. These load-bearing issues undermine the paper's main empirical conclusion.

major comments (5)
  1. [Abstract and §4.4.1 (Figure 5)] The abstract and conclusion state that 'there is no significant difference in terms of ROUGE scores' across frameworks, but Figure 5 reports many significant pairwise Wilcoxon p-values, e.g., TS10 ROUGE-1 AutoGen vs AutoGPT p=0.001, AutoGen vs Dify p=0.001, AutoGPT vs Dify p=0.001, and AutoGen vs Semantic Kernel p=0.014, with similar significant results in Figures 5(b)-(f). The claim is therefore contradicted by the paper's own reported statistics. Either the significance analysis is incorrect or the conclusion must be substantially qualified (e.g., 'no framework is consistently and significantly better on all metrics').
  2. [§4.3 vs Figure 5/Table 19] Section 4.3 says the Wilcoxon signed-rank test is used 'to compare the performance of each MAS framework with the Metagente framework,' but Figure 5 and Table 19 contain only the five reimplementations; no Metagente row or column appears. The promised baseline comparison is missing, so the quantitative evaluation cannot be checked against the reference system and the 'no significant difference' claim has no baseline anchor.
  3. [§4.4 and abstract] The abstract claims that 'time taken to complete the task varies significantly across the frameworks,' but no section presents the efficiency data (token usage, number of requests, usage time). Section 4.4 presents only Effectiveness, and the cross-reference to 'Section??' for efficiency is unresolved. A claimed finding of the paper is therefore unreported.
  4. [§3.2 and §3.3] The feature classification in Table 6 is attributed to 'a single author' (Section 3.2) with the classification distributed among authors; no inter-rater reliability measure is reported for the qualitative matrices. Additionally, §3.3 (Addressing RQ3) says the quantitative evaluation selected 'four tools,' but Table 19 and Figure 5 include five implementations (Semantic Kernel Chat is added). The paper should explain why there is a fifth implementation and how the selection criteria apply.
  5. [§4.1.1] The empirical evaluation re-implements the authors' own Metagente pipeline in each framework using 'minor modifications' to prompts, standardized models, 15 iterations, and a ROUGE-L threshold of 0.7 adopted from the original study. The comparison is therefore between the authors' ports of a single pipeline, not between idiomatic uses of the frameworks. The conclusion that framework choice does not affect summary quality should be scoped to this 'Metagente-style port' context, and the circularity risk—the benchmark infrastructure comes entirely from the authors' prior work—should be discussed explicitly.
minor comments (7)
  1. [§1, §4.3, §4.4] The manuscript contains unresolved 'Section ??' cross-references for the lessons-learned section and the efficiency section; these must be fixed.
  2. [§4.4.1] The text first says 'boxplots in the right side of Figure 4' and then refers to 'violin plots'; Figure 4 should be described consistently.
  3. [§3.2, Table 6] Table 6 uses unclear symbols (e.g., 'G #', '#') that are not defined; the legend should use unambiguous glyphs (e.g., full, partial, not supported) consistently.
  4. [§3.1] The search window is stated as January 2023 to January 2025, but the reference list includes works dated 2026 (e.g., Wang and Yin 2026; Zeshan et al. 2026); clarify the search and update the reported window.
  5. [§5] In the conclusion, 'recommendation accuracy' appears to be a typo for 'summarization accuracy' or similar; revise for clarity.
  6. [§3.3.6, Table 12] Table 12 lists 'PT' for AutoGPT's real-time logs without defining the value; use a defined symbol or spell out the partial support.
  7. [§4.4.1] The bullet list containing 'Impacts', 'Selection of frameworks', 'Limitations', and 'Future Extension' appears within §4.4.1 after the TS10 results; this material should be moved to a dedicated discussion section to avoid confusing results with discussion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports empirical measurements, and its self-citations supply a benchmark rather than proving the result.

full rationale

This paper is a mixed-method experience report, not a derivation of predictions from first principles. The qualitative analysis (RQ1/RQ2) evaluates frameworks against a feature template and external documentation; the categories and findings are not derived from the frameworks' outputs in a way that equates input with output. The quantitative study (RQ3) reimplements the authors' prior Metagente pipeline on four/five frameworks and measures ROUGE scores. Reusing the authors' own dataset, prompts, thresholds, and model configurations from Nguyen et al. (2025, 2026) is a methodological choice that raises validity and independence concerns, but it does not make the reported ROUGE comparison circular: the scores are observed outcomes of new reimplementations, not quantities defined to equal the inputs. The central claim of 'no significant difference' is an empirical summary, and the paper's own Figure 5 actually contains many significant pairwise Wilcoxon p-values (e.g., AutoGen vs AutoGPT p=0.001 for ROUGE-1 on TS10), which is an internal inconsistency or correctness risk, not a circular reduction. Similarly, Section 4.3 says the Wilcoxon test compares each framework with Metagente, but Figure 5 contains no Metagente row or column; this is a missing-baseline completeness problem, not circularity. The self-citations provide the benchmark scenario and dataset, but the conclusion about framework differences is not proven by those citations; the cited Metagente papers are published, externally checkable works, so they count as independent evidence under the stated rules. The limitations section appropriately acknowledges subjectivity, the single model provider, and task specificity, further showing that the authors do not present the framework comparison as a forced or definitional result. No passage exhibits the specific reduction required for a circularity finding: no equation is defined in terms of the claimed conclusion, no fitted parameter is renamed as a prediction, and no uniqueness theorem from the authors' prior work is invoked to forbid alternatives. Accordingly, the honest finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The study depends on a set of hand-chosen thresholds, dataset sizes, and faithfulness assumptions rather than on fitted model parameters. The free parameters are mostly inherited from the authors' prior work. The axioms reflect standard statistical practice and the domain-specific assumptions needed to treat the README summarization benchmark as a fair framework comparison. No new entities, forces, or mediators are introduced.

free parameters (5)
  • ROUGE-L threshold = 0.7
    Stops the optimization loop and selects candidate prompts; adopted from Nguyen et al. 2025 without independent sensitivity analysis.
  • Maximum iterations = 15
    Number of optimization loop iterations per README file; taken from the authors' prior Metagente study.
  • Cosine similarity cutoff = >0.6
    Filters README-About pairs in dataset curation; from Pham et al. 2023, applied before a manual verification step.
  • Temperature settings = 0.7 (Extractor and Summarizer), 0.0 (Teacher and Evaluator), 0.2 (Prompt Creator)
    Chosen by the authors; no sensitivity analysis is reported, and AutoGPT did not allow adjustment.
  • Training set sizes = TS10 with 10 samples and TS50 with 50 samples
    Randomly selected from 925 curated repositories; the sizes are not justified by a power analysis.
assumptions (6)
  • domain assumption ROUGE-1, ROUGE-2, and ROUGE-L scores are valid proxies for summary quality for this task.
    Section 4.3 defines the metrics, but Section 4.6 acknowledges that ROUGE 'might not be enough to evaluate the performance'.
  • standard math Wilcoxon signed-rank test and Shapiro-Wilk test are appropriate for the paired score distributions.
    Section 4.3 states normality was checked with Shapiro-Wilk and that Wilcoxon signed-rank tests were used; this is standard nonparametric practice.
  • domain assumption GitHub star count and forks are adequate popularity and quality proxies for selecting frameworks.
    Section 3.1, Table 3 uses at least 10,000 stars as an inclusion criterion, citing Borges and Tulio Valente (2018).
  • domain assumption The README.MD summarization task is representative of MAS usage in software engineering.
    Section 4 motivates the task, but the limitations in Section 4.6 concede that results may not generalize to other task types.
  • domain assumption The four or five framework implementations are faithful, equivalent ports of the Metagente pipeline.
    Section 4.1.1 says prompts were 'adapted' with 'minor modifications'; the equivalence of the ports is not validated or measured.
  • domain assumption OpenAI models represent LLM behavior across all frameworks in the comparison.
    Section 4.1.2 uses gpt-4o-mini and gpt-4o for all frameworks except AutoGPT, which does not allow temperature adjustment; Section 4.6 flags single-provider bias.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report." pith.science (2026). https://pith.science/paper/YEA3SVXI

@misc{pith2026260811965,
  author       = {Pith},
  title        = {Pith review of: Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YEA3SVXI}},
  note         = {Machine review of arXiv:2608.11965}
}
read the original abstract

The proliferation of Generative Artificial Intelligence (Gen AI) powered by large language models (LLMs) has transformed the software development process, introducing new paradigms for code generation, debugging, testing, and maintenance. While early applications focused on leveraging single, independent LLMs to assist developers with isolated tasks, recent advances have shifted toward multi-agent systems (MAS) that orchestrate multiple LLM-based agents working collaboratively toward common objectives. Despite their promising potential, using MAS encompasses a set of challenges for developers who have to carefully select the right technology, devise proper coordination rules, and design specific roles for the involved agents. In this paper, we provide a comprehensive overview of the existing tools and frameworks for implementing MAS in software engineering. First, we conducted a quantitative analysis of the most relevant open source MAS frameworks by evaluating their documentation, features, and capabilities from the developers' perspective. Second, we performed a qualitative evaluation of a subset of the selected frameworks by implementing a common use case: the summarization of README.MD files. The findings show that the selected frameworks provide a good coverage of fundamental components of MAS, though advanced features such as telemetry of agents are still missing. In addition, the empirical evaluation shows that there is no significant difference in terms of ROUGE scores considering the summarization task. Finally, we provide a set of lessons learned and challenges that can help researchers and practitioners to select a suitable MAS framework according to their needs.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

251 extracted references · 24 canonical work pages

  1. [1]

    https://github.com/Significant-Gravitas/Auto-GPT, gitHub repository, last accessed 11-02-2025

    (2024) Autonomous-gpt: Framework for autonomous gpt-4 agents. https://github.com/Significant-Gravitas/Auto-GPT, gitHub repository, last accessed 11-02-2025

    2024

  2. [2]

    https://github.com/run-llama/llama_index, gitHub repository, last accessed 11-02-2025

    (2024) Llama index: Data framework for llm applications. https://github.com/run-llama/llama_index, gitHub repository, last accessed 11-02-2025

    2024

  3. [3]

    https://github.com/microsoft/autogen, gitHub repository, last accessed 11-02-2025

    (2024 a ) Microsoft autogen: Framework for building multi-agent conversational systems. https://github.com/microsoft/autogen, gitHub repository, last accessed 11-02-2025

    2024

  4. [4]

    https://github.com/microsoft/semantic-kernel, gitHub repository, last accessed 11-02-2025

    (2024 b ) Microsoft semantic kernel: Integration framework for ai models. https://github.com/microsoft/semantic-kernel, gitHub repository, last accessed 11-02-2025

    2024

  5. [5]

    https://github.com/xlang-ai/OpenAgents, gitHub repository, last accessed 11-02-2025

    (2024) Openagents: Open platform for language agents. https://github.com/xlang-ai/OpenAgents, gitHub repository, last accessed 11-02-2025

    2024

  6. [12]

    Borges H, Hora A, Valente MT (2016) Understanding the Factors That Impact the Popularity of GitHub Repositories . In: 2016 IEEE International Conference on Software Maintenance and Evolution ( ICSME ), IEEE, Raleigh, NC, USA, pp 334--344, doi:10.1109/ICSME.2016.31, ://ieeexplore.ieee.org/document/7816479/

    arXiv 2016

  7. [17]

    https://github.com/camel-ai/camel, gitHub repository, last accessed 11-02-2025

    Contributors CA (2024 a ) Camel: Communicative agents for "mind" exploration. https://github.com/camel-ai/camel, gitHub repository, last accessed 11-02-2025

    2024

  8. [18]

    https://github.com/langgenius/dify, gitHub repository, last accessed 11-02-2025

    Contributors D (2024 b ) Dify: Open-source framework for llm applications. https://github.com/langgenius/dify, gitHub repository, last accessed 11-02-2025

    2024

Show all 251 references
  1. [19]

    https://github.com/FlowiseAI/Flowise, gitHub repository, last accessed 11-02-2025

    Contributors F (2024 c ) Flowise: Drag & drop ui framework for building llm flows. https://github.com/FlowiseAI/Flowise, gitHub repository, last accessed 11-02-2025

    2024

  2. [20]

    https://github.com/kreneskyp/ix, gitHub repository, last accessed 11-02-2025

    ix Contributors K (2024) ix: Autonomous agent framework. https://github.com/kreneskyp/ix, gitHub repository, last accessed 11-02-2025

    2024

  3. [21]

    Houghton Mifflin Boston

    Cook TD, Campbell DT, Day A (1979) Quasi-experimentation: Design & analysis issues for field settings, vol 351. Houghton Mifflin Boston

    1979

  4. [22]

    In: Bertolino A, Pascoal Faria J, Lago P, Semini L (eds) Quality of Information and Communications Technology, Springer Nature Switzerland, Cham, pp 161--176

    Cursaru VA, Duits L, Milligan J, Ural D, Sanchez BR, Stoico V, Malavolta I (2024) A controlled experiment on the energy efficiency of the source code generated by code llama. In: Bertolino A, Pascoal Faria J, Lago P, Semini L (eds) Quality of Information and Communications Tec...

    2024

  5. [23]

    Information and Software Technology 181:107678, doi:https://doi.org/10.1016/j.infsof.2025.107678, ://www.sciencedirect.com/science/article/pii/S0950584925000175

    De Martino V, Palomba F (2025) Classification and challenges of non-functional requirements in ml-enabled systems: A systematic literature review. Information and Software Technology 181:107678, doi:https://doi.org/10.1016/j.infsof.2025.107678, ://www.sciencedirect.com/science...

    2025

  6. [24]

    ://github.com/MDEGroup/LLMs-based-MAS-ReplicationPackage

    De Oliveira MCS, Ibiyo MO, Gianrusso M, Di Sipio C, Di Ruscio D, Nguyen PT (2025) Replication Package . ://github.com/MDEGroup/LLMs-based-MAS-ReplicationPackage

    2025

  7. [25]

    https://github.com/deepset-ai/haystack, gitHub repository, last accessed 11-02-2025

    Deepsetai (2024) Haystack: End-to-end nlp framework. https://github.com/deepset-ai/haystack, gitHub repository, last accessed 11-02-2025

    2024

  8. [29]

    https://github.com/agno-agi/agno, gitHub repository, last accessed 11-02-2025

    Github A (2025) Agno: Framework for ai-driven decision-making. https://github.com/agno-agi/agno, gitHub repository, last accessed 11-02-2025

    2025

  9. [30]

    In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, ://openreview.net/forum?id=2Rwq6c3tvr

    Golchin S, Surdeanu M (2024) Time travel in llms: Tracing data contamination in large language models. In: The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, ://openreview.net/forum?id=2Rwq6c3tvr

    2024

  10. [31]

    Lawrence Erlbaum Associates, ://books.google.it/books?id=4C49CGkNxLAC

    Grissom R, Kim J (2005) Effect Sizes for Research: A Broad Practical Approach. Lawrence Erlbaum Associates, ://books.google.it/books?id=4C49CGkNxLAC

    2005

  11. [40]

    In: Proceedings of the 40th International Conference on Machine Learning, JMLR.org, ICML'23

    Kandpal N, Lester B, Muqeeth M, Mascarenhas A, Evans M, Baskaran V, Huang T, Liu H, Raffel C (2023) Git-theta: a git extension for collaborative development of machine learning models. In: Proceedings of the 40th International Conference on Machine Learning, JMLR.org, ICML'23

    2023

  12. [43]

    IEEE Software 12(6):42--50, doi:10.1109/52.469759

    Kruchten P (1995) The 4+1 view model of architecture. IEEE Software 12(6):42--50, doi:10.1109/52.469759

    1995 doi

  13. [45]

    ://arxiv.org/abs/2303.17760, 2303.17760

    Li G, Hammoud HAAK, Itani H, Khizbullin D, Ghanem B (2023) Camel: Communicative agents for "mind" exploration of large language model society. ://arxiv.org/abs/2303.17760, 2303.17760

    2023 arXiv

  14. [47]

    Journal of Systems and Software 212:112002, doi:https://doi.org/10.1016/j.jss.2024.112002, ://www.sciencedirect.com/science/article/pii/S0164121224000451

    Li X, Yuan S, Gu X, Chen Y, Shen B (2024 b ) Few-shot code translation via task-adapted prompt learning. Journal of Systems and Software 212:112002, doi:https://doi.org/10.1016/j.jss.2024.112002, ://www.sciencedirect.com/science/article/pii/S0164121224000451

    2024

  15. [48]

    In: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain, pp 74--81, ://aclanthology.org/W04-1013/

    Lin CY (2004) ROUGE : A package for automatic evaluation of summaries. In: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain, pp 74--81, ://aclanthology.org/W04-1013/

    2004

  16. [53]

    doi:http://dx.doi.org/10.1016/j.redeen.2016.05.001

    Molina-Azorín JF (2016) Mixed methods research: An opportunity to improve our studies and our research skills. doi:http://dx.doi.org/10.1016/j.redeen.2016.05.001

    2016 doi

  17. [57]

    ://docs.softwareheritage.org/devel/swh-dataset/graph/dataset.html

    popular-3k python (2023) Dataset — Software Heritage documentation. ://docs.softwareheritage.org/devel/swh-dataset/graph/dataset.html

    2023

  18. [59]

    Rasheed Z, Waseem M, Sami MA, Kemell KK, Ahmad A, Duc AN, Systä K, Abrahamsson P (2025) Autonomous Agents in Software Development : A Vision Paper . In: Agile Processes in Software Engineering and Extreme Programming – Workshops , Springer, Cham, pp 15--23, doi:10.1007/978-3-0...

    2025 doi

  19. [60]

    In: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, JMLR.org, ICML'15, p 2152–2161

    Romera-Paredes B, Torr PHS (2015) An embarrassingly simple approach to zero-shot learning. In: Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, JMLR.org, ICML'15, p 2152–2161

    2015

  20. [66]

    Biometrika 52(3/4):591--611, ://www.jstor.org/stable/2333709

    Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3/4):591--611, ://www.jstor.org/stable/2333709

    1965

  21. [67]

    https://botpress.com, last accessed: Mar 19, 2025

    Team B (2024 a ) Botpress. https://botpress.com, last accessed: Mar 19, 2025

    2024

  22. [68]

    https://github.com/crewAIInc/crewAI, gitHub repository, last accessed 11-02-2025

    Team C (2024 b ) Crewai: Framework for orchestrating role-playing ai agents. https://github.com/crewAIInc/crewAI, gitHub repository, last accessed 11-02-2025

    2024

  23. [69]

    https://github.com/langchain-ai/langchain, gitHub repository, last accessed 11-02-2025

    Team L (2024 c ) Langchain: Building applications with llms through composability. https://github.com/langchain-ai/langchain, gitHub repository, last accessed 11-02-2025

    2024

  24. [70]

    https://github.com/geekan/MetaGPT, gitHub repository, last accessed 11-02-2025

    Team M (2024 d ) Metagpt: Multi-agent framework for software development. https://github.com/geekan/MetaGPT, gitHub repository, last accessed 11-02-2025

    2024

  25. [71]

    https://github.com/huggingface/smolagents, gitHub repository, last accessed 11-02-2025

    Team S (2024 e ) Smolagents: Minimalist framework for building powerful agents. https://github.com/huggingface/smolagents, gitHub repository, last accessed 11-02-2025

    2024

  26. [72]

    Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser u, Polosukhin I (2017) Attention is All you Need . In: Advances in Neural Information Processing Systems , Curran Associates, Inc., vol 30, ://proceedings.neurips.cc/paper_files/paper/2017/hash/3f5ee243547d...

    2017

  27. [75]

    Biometrics Bulletin 1(6):80--83, ://www.jstor.org/stable/3001968

    Wilcoxon F (1945) Individual comparisons by ranking methods. Biometrics Bulletin 1(6):80--83, ://www.jstor.org/stable/3001968

    1945

  28. [76]

    ://arxiv.org/abs/2308.08155, 2308.08155

    Wu Q, Bansal G, Zhang J, Wu Y, Li B, Zhu E, Jiang L, Zhang X, Zhang S, Liu J, Awadallah AH, White RW, Burger D, Wang C (2023) Autogen: Enabling next-gen llm applications via multi-agent conversation. ://arxiv.org/abs/2308.08155, 2308.08155

    2023 arXiv

  29. [77]

    Science China Information Sciences 68(2):121101, doi:10.1007/s11432-024-4222-0, ://doi.org/10.1007/s11432-024-4222-0, read\_Status: New Read\_Status\_Date: 2025-05-12T08:57:00.797Z

    Xi Z, Chen W, Guo X, He W, Ding Y, Hong B, Zhang M, Wang J, Jin S, Zhou E, Zheng R, Fan X, Wang X, Xiong L, Zhou Y, Wang W, Jiang C, Zou Y, Liu X, Yin Z, Dou S, Weng R, Qin W, Zheng Y, Qiu X, Huang X, Zhang Q, Gui T (2025) The rise and potential of large language model based a...

    2025 doi

  30. [78]

    ://arxiv.org/abs/2312.05562, arXiv:2312.05562 [cs]

    Yang G, Zhou Y, Chen X, Zhang X, Zhuo TY, Chen T (2023) Chain-of- Thought in Neural Code Generation : From and For Lightweight Language Models . ://arxiv.org/abs/2312.05562, arXiv:2312.05562 [cs]

    2023 arXiv

  31. [83]

    2005 , publisher=

    Effect Sizes for Research: A Broad Practical Approach , author=. 2005 , publisher=

    2005

  32. [84]

    The Twelfth International Conference on Learning Representations,

    Shahriar Golchin and Mihai Surdeanu , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =

    2024

  33. [85]

    2023 , eprint=

    AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation , author=. 2023 , eprint=

    2023

  34. [86]

    Select, Prompt, Filter: Distilling Large Language Models for Summarizing Conversations

    Pham, Minh-Quang and Indurthi, Sathish and Chollampatt, Shamil and Turchi, Marco. Select, Prompt, Filter: Distilling Large Language Models for Summarizing Conversations. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/...

    2023 doi

  35. [87]

    C hat D ev: Communicative Agents for Software Development

    Qian, Chen and Liu, Wei and Liu, Hongzhang and Chen, Nuo and Dang, Yufan and Li, Jiahao and Yang, Cheng and Chen, Weize and Su, Yusheng and Cong, Xin and Xu, Juyuan and Li, Dahai and Liu, Zhiyuan and Sun, Maosong. C hat D ev: Communicative Agents for Software Development. Proc...

    2024 doi

  36. [88]

    Many hands make light work: An LLM-based multi-agent system for detecting malicious PyPI packages , journal =

    Muhammad Umar. Many hands make light work: An LLM-based multi-agent system for detecting malicious PyPI packages , journal =. 2026 , issn =. doi:https://doi.org/10.1016/j.jss.2026.112792 , url =

    2026

  37. [89]

    ACM Trans

    Liu, Junwei and Wang, Kaixin and Chen, Yixuan and Peng, Xin and Chen, Zhenpeng and Zhang, Lingming and Lou, Yiling , title =. ACM Trans. Softw. Eng. Methodol. , month = mar, keywords =. 2026 , publisher =. doi:10.1145/3796507 , abstract =

    2026 doi

  38. [90]

    Guidelines for Empirical Studies in Software Engineering involving Large Language Models , journal =

    Sebastian Baltes and Florian Angermeir and Chetan Arora and Marvin Mu. Guidelines for Empirical Studies in Software Engineering involving Large Language Models , journal =. 2025 , url =. doi:10.48550/ARXIV.2508.15503 , eprinttype =. 2508.15503 , timestamp =

    2025 doi arXiv

  39. [91]

    Zhang, Ziyao and Wang, Chong and Wang, Yanlin and Shi, Ensheng and Ma, Yuchi and Zhong, Wanjun and Chen, Jiachi and Mao, Mingzhi and Zheng, Zibin , title =. Proc. ACM Softw. Eng. , month = jun, articleno =. 2025 , issue_date =. doi:10.1145/3728894 , abstract =

    2025 doi

  40. [92]

    Privacy issues in Large Language Models: A survey , journal =

    Hareem Kibriya and Wazir Zada Khan and Ayesha Siddiqa and Muhammad Khurram Khan , abstract =. Privacy issues in Large Language Models: A survey , journal =. 2024 , issn =. doi:https://doi.org/10.1016/j.compeleceng.2024.109698 , url =

    2024

  41. [93]

    Security and privacy in LLMs: A comprehensive survey of threats and mitigation strategies , journal =

    Aymen Dia Eddine Berini and Norziana Jamil and Ala-Eddine Benrazek and Abderrahmane Lakas and Leila Ismail and Mohamed Amine Ferrag and Kwok-Yan Lam , keywords =. Security and privacy in LLMs: A comprehensive survey of threats and mitigation strategies , journal =. 2026 , issn...

    2026

  42. [94]

    Nguyen , keywords =

    Alessio Bucaioni and Hampus Ekedahl and Vilma Helander and Phuong T. Nguyen , keywords =. Programming with ChatGPT: How far can we go? , journal =. 2024 , issn =. doi:https://doi.org/10.1016/j.mlwa.2024.100526 , url =

    2024

  43. [95]

    Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective , journal =

    Ernests Lavrinovics and Russa Biswas and Johannes Bjerva and Katja Hose , keywords =. Knowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective , journal =. 2025 , issn =. doi:https://doi.org/10.1016/j.websem.2024.100844 , url =

    2025

  44. [96]

    GCL: Group-shared continual learning fine-tuning for sparse LLMs , journal =

    Yanzhe Wang and Baoqun Yin , keywords =. GCL: Group-shared continual learning fine-tuning for sparse LLMs , journal =. 2026 , issn =. doi:https://doi.org/10.1016/j.neucom.2026.132918 , url =

    2026

  45. [97]

    doi:https://doi.org/10.4135/9781446280119 , year=

    Catherine Cassell and Gillian Symon , title=. doi:https://doi.org/10.4135/9781446280119 , year=

    doi

  46. [98]

    2024 , eprint=

    MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework , author=. 2024 , eprint=

    2024

  47. [99]

    2023 , eprint=

    CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society , author=. 2023 , eprint=

    2023

  48. [100]

    European Journal of Management and Business Economics , DOI =

    Mixed methods research: An opportunity to improve our studies and our research skills , author =. European Journal of Management and Business Economics , DOI =

  49. [101]

    , journal=

    Kruchten, P.B. , journal=. The 4+1 View Model of architecture , year=

  50. [102]

    Doan, Thu T. H. and Nguyen, Phuong T. and Di Rocco, Juri and Di Ruscio, Davide , title =. Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering , pages =. 2023 , isbn =. doi:10.1145/3593434.3593448 , abstract =

    2023

  51. [103]

    Automatically

    Zanartu, Francisco and Treude, Christoph and Cartaxo, Bruno and Borges, Hudson Silva and Moura, Pedro and Wagner, Markus and Pinto, Gustavo , year = 2022, month = jul, publisher =. Automatically. doi:10.48550/arXiv.2208.00269 , url =

    2022 doi arXiv

  52. [104]

    Dataset —

    2021-03-23-popular-3k-python , year = 2023, month = dec, url =. Dataset —

    2021

  53. [105]

    Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems , location =

    Exploration and Explanation in Computational Notebooks , author =. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems , location =. doi:10.1145/3173574.3173606 , isbn = 9781450356206, url =

    2018

  54. [106]

    Understanding the

    Borges, Hudson and Hora, Andre and Valente, Marco Tulio , year = 2016, month = oct, booktitle =. Understanding the. doi:10.1109/ICSME.2016.31 , isbn =

    2016 doi

  55. [107]

    Study the

    Wang, Tianlei and Wang, Shaowei and Chen, Tse-Hsun (Peter) , year = 2022, month = nov, address =. Study the. doi:10.2139/ssrn.4281782 , url =

    2022 doi

  56. [108]

    SIGSOFT Softw

    Grey Literature: A Safe Bridge Between Academy and Industry? , author =. SIGSOFT Softw. Eng. Notes , publisher =. doi:10.1145/3356773.3356776 , issn =

  57. [109]

    Proceedings of the 1st ACM International Conference on AI-Powered Software , location =

    Green AI in Action: Strategic Model Selection for Ensembles in Production , author =. Proceedings of the 1st ACM International Conference on AI-Powered Software , location =. doi:10.1145/3664646.3664763 , isbn = 9798400706851, url =

  58. [110]

    AI-ware: Bridging AI and Software Engineering for responsible and sustainable intelligent artefacts , author =

  59. [111]

    Nguyen, Duc S. H. and Truong, Bach G. and Nguyen, Phuong T. and Di Rocco, Juri and Di Ruscio, Davide , title =. Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering , pages =. 2025 , isbn =. doi:10.1145/3696630.3728511 , abstract =

    2025

  60. [112]

    High-Confidence Computing , volume = 4, number = 2, pages = 100211, doi =

    A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly , author =. High-Confidence Computing , volume = 4, number = 2, pages = 100211, doi =

  61. [113]

    ACM Trans

    A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions , author =. ACM Trans. Inf. Syst. , publisher =. doi:10.1145/3703155 , issn =

    doi

  62. [114]

    CoRR , volume =

    Overcoming catastrophic forgetting in neural networks , author =. CoRR , volume =. 1612.00796 , timestamp =

    arXiv

  63. [115]

    and Sethi, Rohan and Lu, Yung-Hsiang and Thiruvathukal, George K

    Jiang, Wenxin and Synovic, Nicholas and Hyatt, Matt and Schorlemmer, Taylor R. and Sethi, Rohan and Lu, Yung-Hsiang and Thiruvathukal, George K. and Davis, James C. , year = 2023, month =. An. Proceedings of the 45th. doi:10.1109/ICSE48619.2023.00206 , isbn =

    2023

  64. [116]

    Toolformer:

    Schick, Timo and Dwivedi-Yu, Jane and Dessì, Roberto and Raileanu, Roberta and Lomeli, Maria and Zettlemoyer, Luke and Cancedda, Nicola and Scialom, Thomas , year = 2023, month =. Toolformer:. doi:10.48550/arXiv.2302.04761 , url =

    2023 doi arXiv

  65. [117]

    doi:10.48550/arXiv.2307.16648 , url =

    Giglou, Hamed Babaei and D'Souza, Jennifer and Auer, Sören , year = 2023, month =. doi:10.48550/arXiv.2307.16648 , url =

    2023 doi arXiv

  66. [118]

    Knowledge

    Wei, Xiaokai and Wang, Shen and Zhang, Dejiao and Bhatia, Parminder and Arnold, Andrew , year = 2021, month =. Knowledge. doi:10.48550/arXiv.2110.08455 , url =

    2021 doi arXiv

  67. [119]

    Chain-of-

    Yang, Guang and Zhou, Yu and Chen, Xiang and Zhang, Xiangyu and Zhuo, Terry Yue and Chen, Taolue , year = 2023, month =. Chain-of-

    2023

  68. [120]

    IEEE Transactions on Software Engineering , volume = 49, number = 4, pages =

    Zhang, Jingxuan and Liu, Siyuan and Gong, Lina and Zhang, Haoxiang and Huang, Zhiqiu and Jiang, He , year = 2023, month =. IEEE Transactions on Software Engineering , volume = 49, number = 4, pages =. doi:10.1109/TSE.2022.3227559 , issn =

    2023

  69. [121]

    Automatic

    Zhao, James Xu and Xie, Yuxi and Kawaguchi, Kenji and He, Junxian and Xie, Michael Qizhe , year = 2023, month =. Automatic. doi:10.48550/arXiv.2305.14333 , url =

    2023 doi arXiv

  70. [123]

    Sharma, Tushar and Kechagia, Maria and Georgiou, Stefanos and Tiwari, Rohit and Vats, Indira and Moazen, Hadi and Sarro, Federica , year = 2022, month =. A. doi:10.48550/arXiv.2110.09610 , url =

    2022 doi arXiv

  71. [124]

    Dong, Yihong and Jiang, Xue and Jin, Zhi and Li, Ge , year = 2024, month =. Self-. ACM Trans. Softw. Eng. Methodol. , publisher =. doi:10.1145/3672459 , issn =

    2024 doi

  72. [125]

    doi:10.1007/978-3-031-70445-1_35 , isbn =

    Ziche, Clara and Apruzzese, Giovanni , year = 2024, booktitle =. doi:10.1007/978-3-031-70445-1_35 , isbn =

    2024 doi

  73. [126]

    doi:10.48550/arXiv.2403.12031 , url =

    Hu, Qitian Jason and Bieker, Jacob and Li, Xiuyu and Jiang, Nan and Keigwin, Benjamin and Ranganath, Gaurav and Keutzer, Kurt and Upadhyay, Shriyash Kaustubh , year = 2024, month =. doi:10.48550/arXiv.2403.12031 , url =

    2024 doi arXiv

  74. [127]

    Wang, Fali and Zhang, Zhiwei and Zhang, Xianren and Wu, Zongyu and Mo, Tzuhao and Lu, Qiuhao and Wang, Wanjing and Li, Rui and Xu, Junjie and Tang, Xianfeng and He, Qi and Ma, Yao and Huang, Ming and Wang, Suhang , year = 2024, month =. A. doi:10.48550/arXiv.2411.03350 , url =

    2024 doi arXiv

  75. [128]

    Morales, Sergio and Clarisó, Robert and Cabot, Jordi , year = 2024, month =. A. Proceedings of the. doi:10.1145/3640310.3674093 , isbn =

    2024

  76. [129]

    A survey on

    Li, Xinyi and Wang, Sai and Zeng, Siqi and Wu, Yu and Yang, Yi , year = 2024, month =. A survey on. Vicinagearth , volume = 1, number = 1, pages =. doi:10.1007/s44336-024-00009-2 , issn =

    2024 doi

  77. [130]

    Wu, Qingyun and Bansal, Gagan and Zhang, Jieyu and Wu, Yiran and Li, Beibin and Zhu, Erkang and Jiang, Li and Zhang, Xiaoyun and Zhang, Shaokun and Liu, Jiale and Awadallah, Ahmed and White, Ryen W and Burger, Doug and Wang, Chi , year = 2024, note =

    2024

  78. [131]

    Proceedings of the 37th

    Li, Guohao and Al Kader Hammoud, Hasan Abed and Itani, Hani and Khizbullin, Dmitrii and Ghanem, Bernard , year = 2023, month =. Proceedings of the 37th

    2023

  79. [132]

    Information and Software Technology , volume = 106, pages =

    Guidelines for including grey literature and conducting multivocal literature reviews in software engineering , author =. Information and Software Technology , volume = 106, pages =. doi:10.1016/j.infsof.2018.09.006 , issn =

    2018 doi

  80. [133]

    Proceedings of the 2023

    Li, Junyi and Cheng, Xiaoxue and Zhao, Xin and Nie, Jian-Yun and Wen, Ji-Rong , year = 2023, month =. Proceedings of the 2023. doi:10.18653/v1/2023.emnlp-main.397 , url =

    2023 doi

  81. [134]

    Impromptu: a framework for model-driven prompt engineering , shorttitle =

    Morales, Sergio and Clarisó, Robert and Cabot, Jordi , year = 2025, month =. Impromptu: a framework for model-driven prompt engineering , shorttitle =. Software and Systems Modeling , pages =. doi:10.1007/s10270-024-01235-4 , issn =

    2025 doi

  82. [135]

    ACM Trans

    Large Language Models for Software Engineering: A Systematic Literature Review , author =. ACM Trans. Softw. Eng. Methodol. , publisher =. doi:10.1145/3695988 , issn =

    doi

  83. [136]

    , year = 2023, month =

    Fan, Angela and Gokkaya, Beliz and Harman, Mark and Lyubarskiy, Mitya and Sengupta, Shubho and Yoo, Shin and Zhang, Jie M. , year = 2023, month =. Large

    2023

  84. [137]

    and Santos, Wylliams B

    Neto, Geraldo Torres G. and Santos, Wylliams B. and Endo, Patricia Takako and Fagundes, Roberta A.A. , year = 2019, month =. Multivocal literature reviews in software engineering:. 2019. doi:10.1109/ESEM.2019.8870142 , url =

    2019

  85. [138]

    and Lo, David , year = 2024, month =

    Liu, Yue and Le-Cong, Thanh and Widyasari, Ratnadira and Tantithamthavorn, Chakkrit and Li, Li and Le, Xuan-Bach D. and Lo, David , year = 2024, month =. Refining. ACM Trans. Softw. Eng. Methodol. , volume = 33, number = 5, pages =. doi:10.1145/3643674 , issn =

    2024 doi

  86. [139]

    Survey of

    Ji, Ziwei and Lee, Nayeon and Frieske, Rita and Yu, Tiezheng and Su, Dan and Xu, Yan and Ishii, Etsuko and Bang, Ye Jin and Madotto, Andrea and Fung, Pascale , year = 2023, month =. Survey of. ACM Comput. Surv. , volume = 55, number = 12, pages =. doi:10.1145/3571730 , issn =

    2023 doi

  87. [140]

    and Bui, Nghi D

    Nguyen, Minh Huynh and Chau, Thang Phan and Nguyen, Phong X. and Bui, Nghi D. Q. , year = 2024, month =. doi:10.48550/arXiv.2406.11912 , url =

    2024 doi arXiv

  88. [141]

    and Baxter, Daniel P

    Hepworth, Adam J. and Baxter, Daniel P. and Abbass, Hussein A. , year = 2022, journal =. doi:10.1109/ACCESS.2022.3180032 , issn =

    2022

  89. [142]

    Narasimha, V. B. and Sujatha, B. and Nagaprasad, S. , year = 2022, month =. An. The. doi:10.1201/9781003048367-33 , isbn =

    2022 doi

  90. [143]

    Nguyen, Minh Khoi and Tran, Hanh Nhi and Abualsaud, Razan and Ober, Ileana , year = 2023, month =. An. 2023 15th. doi:10.1109/KSE59128.2023.10299475 , url =

    2023

  91. [144]

    Introducing

    Giorgis, Stefano De and Gangemi, Aldo and Gromann, Dagmar , abstract =. Introducing

  92. [145]

    Naja, Iman and Markovic, Milan and Edwards, Peter and Pang, Wei and Cottrill, Caitlin and Williams, Rebecca , year = 2022, journal =. Using. doi:10.1109/ACCESS.2022.3188967 , issn =

    2022

  93. [146]

    Building and

    Muppasani, Bharath and Pallagani, Vishal and Srivastava, Biplav and Mutharaju, Raghava , abstract =. Building and

  94. [147]

    Aligning

    Ribeiro, Manuel de Sousa and Leite, João , year = 2021, month =. Aligning. Proceedings of the AAAI Conference on Artificial Intelligence , volume = 35, number = 6, pages =. doi:10.1609/aaai.v35i6.16626 , issn =

    2021 doi

  95. [148]

    Ferilli, Stefano and Redavid, Domenico and Pierro, Davide Di and Loop, Liza , abstract =. An

  96. [149]

    Behaviour & Information Technology , volume = 41, number = 9, pages =

    End-user engineering of ontology-based knowledge bases , author =. Behaviour & Information Technology , volume = 41, number = 9, pages =. doi:10.1080/0144929X.2022.2092032 , issn =

    2022

  97. [150]

    Journal of Web Semantics , volume = 32, pages =

    Using a suite of ontologies for preserving workflow-centric research objects , author =. Journal of Web Semantics , volume = 32, pages =. doi:10.1016/j.websem.2015.01.003 , issn =

    2015 doi

  98. [151]

    Generating

    Stinner, Florian and Yang, Yingying and Schreiber, Thomas and Bode, Gerrit and Baranski, Marc and Müller, Dirk , year = 2019, month =. Generating. doi:10.22260/ISARC2019/0031 , url =

    2019 doi

  99. [152]

    Sfayhi, Ahmed and Sahraoui, Houari , year = 2011, month =. What. 2011. doi:10.1109/SCAM.2011.6 , url =

    2011 doi

  100. [153]

    Manziuk, Eduard and Barmak, Olexander and Krak, Iurii and Mazurets, Olexander , abstract =. Formal

  101. [154]

    Generation of

    Qi, Ruihua and Li, Weilong and Lyu, Haobo , year = 2025, booktitle =. Generation of. doi:10.1007/978-981-97-9443-0_14 , isbn =

    2025 doi

  102. [155]

    ACM Trans

    LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead , author =. ACM Trans. Softw. Eng. Methodol. , publisher =. doi:10.1145/3712003 , issn =

    doi

  103. [156]

    and Wiest, Olaf and Zhang, Xiangliang , year = 2024, month =

    Guo, Taicheng and Chen, Xiuying and Wang, Yaqi and Chang, Ruidi and Pei, Shichao and Chawla, Nitesh V. and Wiest, Olaf and Zhang, Xiangliang , year = 2024, month =. Large. doi:10.24963/ijcai.2024/890 , url =

    2024 doi

  104. [157]

    Collective predictive coding hypothesis: symbol emergence as decentralized

    Taniguchi, Tadahiro , year = 2024, month =. Collective predictive coding hypothesis: symbol emergence as decentralized. Frontiers in Robotics and AI , volume = 11, doi =

    2024

  105. [158]

    Händler, Thorsten , year = 2023, booktitle =. A. doi:10.5220/0012239100003598 , isbn =

    2023 doi

  106. [159]

    Sahoo, Pranab and Singh, Ayush Kumar and Saha, Sriparna and Jain, Vinija and Mondal, Samrat and Chadha, Aman , year = 2024, month =. A. doi:10.48550/arXiv.2402.07927 , url =

    2024 doi arXiv

  107. [160]

    Jin, Weiqiang and Du, Hongyang and Zhao, Biao and Tian, Xingwu and Shi, Bohang and Yang, Guan , year = 2025, month =. A. doi:10.2139/ssrn.5106265 , url =

    2025 doi

  108. [161]

    Sun, Maojun and Han, Ruijian and Jiang, Binyan and Qi, Houduo and Sun, Defeng and Yuan, Yancheng and Huang, Jian , year = 2024, month =. A. doi:10.48550/arXiv.2412.14222 , url =

    2024 doi

  109. [162]

    doi:10.48550/arXiv.2411.04890 , url =

    Wang, Shuai and Liu, Weiwen and Chen, Jingxuan and Gan, Weinan and Zeng, Xingshan and Yu, Shuai and Hao, Xinlong and Shao, Kun and Wang, Yasheng and Tang, Ruiming , year = 2024, month =. doi:10.48550/arXiv.2411.04890 , url =

    2024 doi arXiv

  110. [163]

    doi:10.48550/arXiv.2405.11106 , url =

    Sun, Chuanneng and Huang, Songjun and Pompili, Dario , year = 2024, month =. doi:10.48550/arXiv.2405.11106 , url =

    2024 doi arXiv

  111. [164]

    Neurocomputing , volume = 610, pages = 128514, doi =

    Multi-agent, human–agent and beyond: A survey on cooperation in social dilemmas , author =. Neurocomputing , volume = 610, pages = 128514, doi =

  112. [165]

    Du, Hung and Thudumu, Srikanth and Vasa, Rajesh and Mouzakis, Kon , year = 2024, month =. A. doi:10.48550/arXiv.2402.01968 , url =

    2024 doi arXiv

  113. [166]

    Exploring

    Cheng, Yuheng and Zhang, Ceyao and Zhang, Zhengwen and Meng, Xiangrui and Hong, Sirui and Li, Wenhao and Wang, Zihao and Wang, Zekai and Yin, Feng and Zhao, Junhua and He, Xiuqiang , year = 2024, month =. Exploring. doi:10.48550/arXiv.2401.03428 , url =

    2024 doi arXiv

  114. [167]

    Dhamani, Dhruv and Maher, Mary Lou , year = 2023, month =. The. doi:10.48550/arXiv.2312.17601 , url =

    2023 doi arXiv

  115. [168]

    Science China Information Sciences , volume = 68, number = 2, pages = 121101, doi =

    The rise and potential of large language model based agents: a survey , author =. Science China Information Sciences , volume = 68, number = 2, pages = 121101, doi =

  116. [169]

    CrewAI: Framework for orchestrating role-playing AI agents , author =

  117. [170]

    Langchain: Building applications with LLMs through composability , author =

  118. [171]

    Microsoft AutoGen: Framework for building multi-agent conversational systems , year = 2024, note =

    2024

  119. [172]

    Llama Index: Data framework for LLM applications , year = 2024, note =

    2024

  120. [173]

    Microsoft Semantic Kernel: Integration framework for AI models , year = 2024, note =

    2024

  121. [174]

    Dify: Open-source framework for LLM applications , author =

  122. [175]

    Haystack: End-to-end NLP framework , author =

  123. [176]

    Embedchain: Framework for ChatGPT-like bots , year = 2024, note =

    2024

  124. [177]

    SuperAGI: Open-source autonomous AI agent framework , author =

  125. [178]

    AGiXT: Scalable framework for AI agents , year = 2024, note =

    2024

  126. [179]

    XAgent: Autonomous LLM-based agent framework , year = 2024, note =

    2024

  127. [180]

    OpenAgents: Open platform for language agents , year = 2024, note =

    2024

  128. [181]

    AI Legion: Swarm framework for autonomous agents , year = 2024, note =

    2024

  129. [182]

    Agent Protocol: Unified interface for AI agents , year = 2024, note =

    2024

  130. [183]

    Agents.js: JavaScript framework for building AI agents , year = 2024, note =

    2024

  131. [184]

    CAMEL: Communicative Agents for "Mind" Exploration , author =

  132. [185]

    BabyAGI: Lightweight framework for AI task management , year = 2024, note =

    2024

  133. [186]

    Autonomous-GPT: Framework for autonomous GPT-4 agents , year = 2024, note =

    2024

  134. [187]

    MetaGPT: Multi-agent framework for software development , author =

  135. [188]

    minions: Extensible framework for AI assistants , year = 2024, note =

    2024

  136. [189]

    ix: Autonomous agent framework , author =

  137. [190]

    saplings: Build smarter agents using tree search , year = 2024, note =

    2024

  138. [191]

    Amazon Bedrock , author =

  139. [192]

    Smolagents: Minimalist framework for building powerful agents , author =

  140. [193]

    Flowise: Drag & drop UI framework for building LLM flows , author =

  141. [194]

    Pydantic AI: Production-grade agent framework built on Pydantic , year = 2024, note =

    2024

  142. [195]

    and Katz, Daniel S

    Barker, Michelle and Chue Hong, Neil P. and Katz, Daniel S. and Lamprecht, Anna-Lena and Martinez-Ortiz, Carlos and Psomopoulos, Fotis and Harrow, Jennifer and Castro, Leyla Jael and Gruenpeter, Morane and Martinez, Paula Andrea and Honeyman, Tom , year = 2022, month =. Introd...

    2022

  143. [196]

    2308.00352 , archiveprefix =

    Hong, Sirui and Zhuge, Mingchen and Chen, Jonathan and Zheng, Xiawu and Cheng, Yuheng and Wang, Jinlin and Zhang, Ceyao and Wang, Zili and Yau, Steven Ka Shing and Lin, Zijuan and Zhou, Liyang and Ran, Chenyu and Xiao, Lingfeng and Wu, Chenglin and Schmidhuber, Jürgen , year =...

    2023 arXiv

  144. [197]

    Investigating

    Bhardwaj, Rishabh and Majumder, Navonil and Poria, Soujanya , year = 2021, month =. Investigating. Cognitive Computation , volume = 13, number = 4, pages =. doi:10.1007/s12559-021-09881-2 , issn =

    2021 doi

  145. [198]

    Proceedings of the 18th

    Guidelines for snowballing in systematic literature studies and a replication in software engineering , author =. Proceedings of the 18th. doi:10.1145/2601248.2601268 , isbn =

  146. [199]

    and Turner, Mark and Niazi, Mahmood and Linkman, Stephen , year = 2010, month =

    Kitchenham, Barbara and Pretorius, Rialette and Budgen, David and Pearl Brereton, O. and Turner, Mark and Niazi, Mahmood and Linkman, Stephen , year = 2010, month =. Systematic literature reviews in software engineering –. Information and Software Technology , volume = 52, num...

    2010 doi

  147. [201]

    Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering , location =

    Rethinking Software Engineering in the Era of Foundation Models: A Curated Catalogue of Challenges in the Development of Trustworthy FMware , author =. Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering , location =. doi:1...

  148. [202]

    What Is the Intended Usage Context of This Model? An Exploratory Study of Pre-Trained Models on Various Model Repositories , shorttitle =

    Gong, Lina and Zhang, Jingxuan and Wei, Mingqiang and Zhang, Haoxiang and Huang, Zhiqiu , year = 2023, month =. What Is the Intended Usage Context of This Model? An Exploratory Study of Pre-Trained Models on Various Model Repositories , shorttitle =. ACM Trans. Softw. Eng. Met...

    2023 doi

  149. [203]

    Hugging Face , note =

  150. [204]

    PyTorch Hub , note =

  151. [205]

    TensorFlow Hub , note =

  152. [206]

    Wolfram Neural Net Repository , note =

  153. [207]

    Nvidia NGC Catalog , note =

  154. [208]

    Kaggle Models , note =

  155. [209]

    Ollama Library , note =

  156. [210]

    SuperAGi Marketplace , note =

  157. [211]

    OpenAI GPT store , note =

  158. [212]

    Llama Index , author =

  159. [213]

    Vertex AI , author =

  160. [214]

    AI legion , author =

  161. [215]

    2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) , volume =

    Software Engineering for Machine Learning: A Case Study , author =. 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) , volume =. doi:10.1109/ICSE-SEIP.2019.00042 , keywords =

    2019

  162. [216]

    Proceedings of the 40th International Conference on Machine Learning , location =

    Git-Theta: a git extension for collaborative development of machine learning models , author =. Proceedings of the 40th International Conference on Machine Learning , location =

  163. [217]

    Proceedings of the 1st International Conference on AI Engineering: Software Engineering for AI , location =

    Code smells for machine learning applications , author =. Proceedings of the 1st International Conference on AI Engineering: Software Engineering for AI , location =. doi:10.1145/3522664.3528620 , isbn = 9781450392754, url =

  164. [218]

    Information and Software Technology , volume = 181, pages = 107678, doi =

    Classification and challenges of non-functional requirements in ML-enabled systems: A systematic literature review , author =. Information and Software Technology , volume = 181, pages = 107678, doi =

  165. [219]

    Studying the

    Mastropaolo, Antonio and Scalabrino, Simone and Cooper, Nathan and others , year = 2021, month = may, booktitle =. Studying the. doi:10.1109/ICSE43902.2021.00041 , isbn =

    2021

  166. [220]

    Exploring the

    Casta\. Exploring the. 2023. doi:10.1109/ESEM56168.2023.10304801 , url =

    2023

  167. [221]

    Quality of Information and Communications Technology , publisher =

    A Controlled Experiment on the Energy Efficiency of the Source Code Generated by Code Llama , author =. Quality of Information and Communications Technology , publisher =

  168. [222]

    Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 , location =

    An embarrassingly simple approach to zero-shot learning , author =. Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 , location =

  169. [223]

    Journal of Systems and Software , volume = 212, pages = 112002, doi =

    Few-shot code translation via task-adapted prompt learning , author =. Journal of Systems and Software , volume = 212, pages = 112002, doi =

  170. [224]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models , author =. Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 , url =

    2022

  171. [225]

    Attention is

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , year = 2017, booktitle =. Attention is

    2017

  172. [226]

    2024 IEEE 32nd International Requirements Engineering Conference (RE) , volume =

    Requirements are All You Need: From Requirements to Code with LLMs , author =. 2024 IEEE 32nd International Requirements Engineering Conference (RE) , volume =. doi:10.1109/RE59067.2024.00049 , keywords =

    2024

  173. [227]

    Sentiment

    Zhang, Ting and Xu, Bowen and Thung, Ferdian and Haryono, Stefanus Agus and Lo, David and others , year = 2020, month = sep, booktitle =. Sentiment. doi:10.1109/ICSME46990.2020.00017 , url =

    2020

  174. [228]

    Traceability

    Lin, Jinfeng and Liu, Yalin and Zeng, Qingkai and Jiang, Meng and Cleland-Huang, Jane , year = 2021, month = feb, publisher =. Traceability

    2021

  175. [229]

    Proceedings of the 44th

    Natural attack for pre-trained models of code , author =. Proceedings of the 44th. doi:10.1145/3510003.3510146 , isbn =

  176. [230]

    Proceedings of the 44th

    Bridging pre-trained models and downstream tasks for source code understanding , author =. Proceedings of the 44th. doi:10.1145/3510003.3510062 , isbn =

  177. [231]

    Proceedings of the 44th

    Using pre-trained models to boost code review automation , author =. Proceedings of the 44th. doi:10.1145/3510003.3510621 , isbn =

  178. [232]

    Proceedings of the 31st

    Using pre-trained language models to resolve textual and semantic merge conflicts (experience paper) , author =. Proceedings of the 31st. doi:10.1145/3533767.3534396 , isbn =

  179. [233]

    , year = 2022, month = mar, publisher =

    Gu, Jian and Salza, Pasquale and Gall, Harald C. , year = 2022, month = mar, publisher =. Assemble. doi:10.1109/SANER53432.2022.00112 , isbn =

    2022

  180. [234]

    Applying

    Mashhadi, Ehsan and Hemmati, Hadi , year = 2021, month = may, booktitle =. Applying. doi:10.1109/MSR52588.2021.00063 , isbn =

    2021

  181. [235]

    On the validity of pre-trained transformers for natural language processing in the software engineering domain , author =

  182. [236]

    Can pre-trained code embeddings improve model performance?

    Ding, Zishuo and Li, Heng and Shang, Weiyi and Chen, Tse-Hsun Peter , year = 2022, month = mar, journal =. Can pre-trained code embeddings improve model performance?. doi:10.1007/s10664-022-10118-5 , issn =

    2022 doi

  183. [239]

    Discrepancies among pre-trained deep neural networks: a new threat to model zoo reliability , shorttitle =

    Montes, Diego and Peerapatanapokin, Pongpatapee and Schultz, Jeff and Guo, Chengjun and Jiang, Wenxin and others , year = 2022, month = nov, booktitle =. Discrepancies among pre-trained deep neural networks: a new threat to model zoo reliability , shorttitle =. doi:10.1145/354...

    2022

  184. [241]

    Analyzing the

    Casta\. Analyzing the. doi:10.48550/arXiv.2311.13380 , url =

    doi arXiv

  185. [242]

    doi:10.48550/arXiv.2309.14393 , url =

    Faiz, Ahmad and Kaneda, Sotaro and Wang, Ruhan and Osi, Rita and Sharma, Prateek and others , year = 2024, month = jan, publisher =. doi:10.48550/arXiv.2309.14393 , url =

    2024 doi arXiv

  186. [243]

    IEEE Software , publisher =

    Green IT and green software , author =. IEEE Software , publisher =

  187. [244]

    2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) , pages =

    Recommending energy-efficient java collections , author =. 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR) , pages =

    2019

  188. [245]

    Proceedings of the 21st Pan-Hellenic Conference on Informatics , pages =

    Analyzing programming languages' energy consumption: An empirical study , author =. Proceedings of the 21st Pan-Hellenic Conference on Informatics , pages =

  189. [246]

    Identifying Relevant Studies in Software Engineering , author =. Inf. Softw. Technol. , publisher =. doi:10.1016/j.infsof.2010.12.010 , issn =

    2010 doi

  190. [247]

    and Kang, Kyo , year = 2005, month = jul, journal =

    Frakes, W.B. and Kang, Kyo , year = 2005, month = jul, journal =. Software reuse research: status and future , shorttitle =. doi:10.1109/TSE.2005.85 , issn =

    2005 doi

  191. [248]

    Integration of

    Sandhu, Amandeep Kaur and Batth, Ranbir Singh , year = 2021, month = jan, booktitle =. Integration of. doi:10.1109/ICCAKM50778.2021.9357738 , url =

    2021

  192. [249]

    Tackling the

    Rennie, Jason D M and Shih, Lawrence and Teevan, Jaime and Karger, David R , year = 2003, abstract =. Tackling the

    2003

  193. [250]

    doi:10.1145/1961189.1961199 , issn =

    Chang, Chih-Chung and Lin, Chih-Jen , year = 2011, month = apr, journal =. doi:10.1145/1961189.1961199 , issn =

    2011

  194. [251]

    IEEE Software , volume = 40, number = 3, pages =

    Application of Large Language Models to Software Engineering Tasks: Opportunities, Risks, and Implications , author =. IEEE Software , volume = 40, number = 3, pages =

  195. [252]

    AI Open , volume = 2, pages =

    Pre-trained models: Past, present and future , author =. AI Open , volume = 2, pages =. doi:https://doi.org/10.1016/j.aiopen.2021.08.002 , issn =

    2021 doi

  196. [253]

    1907.11692 , archiveprefix =

    RoBERTa: A Robustly Optimized BERT Pretraining Approach , author =. 1907.11692 , archiveprefix =

    1907 arXiv

  197. [254]

    1810.04805 , archiveprefix =

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , author =. 1810.04805 , archiveprefix =

    arXiv

  198. [255]

    Proceedings of the Conference on Fairness, Accountability, and Transparency , location =

    Model Cards for Model Reporting , author =. Proceedings of the Conference on Fairness, Accountability, and Transparency , location =. doi:10.1145/3287560.3287596 , isbn = 9781450361255, url =

  199. [256]

    ACM Computing Surveys , publisher =

    A guided tour to approximate string matching , author =. ACM Computing Surveys , publisher =

  200. [257]

    2304.07590 , archiveprefix =

    Self-collaboration Code Generation via ChatGPT , author =. 2304.07590 , archiveprefix =

    arXiv

  201. [258]

    Encyclopedia of Database Systems , publisher =

    Cross-Validation , author =. Encyclopedia of Database Systems , publisher =. doi:10.1007/978-0-387-39940-9\_565 , isbn =

    doi

  202. [259]

    and Rubei, Riccardo , year = 2023, month = apr, journal =

    Di Rocco, Juri and Di Ruscio, Davide and Di Sipio, Claudio and Nguyen, Phuong T. and Rubei, Riccardo , year = 2023, month = apr, journal =. doi:10.1007/s10489-022-03864-y , issn =

    2023 doi

  203. [260]

    , year = 2020, month = apr, booktitle =

    Di Sipio, Claudio and Rubei, Riccardo and Di Ruscio, Davide and Nguyen, Phuong T. , year = 2020, month = apr, booktitle =. A. doi:10.1145/3383219.3383227 , isbn =

    2020

  204. [261]

    and Rubei, Riccardo , year = 2021, month = jul, journal =

    Di Rocco, Juri and Di Ruscio, Davide and Di Sipio, Claudio and Nguyen, Phuong T. and Rubei, Riccardo , year = 2021, month = jul, journal =. Development of recommendation systems for software engineering: the. doi:10.1007/s10664-021-09963-7 , issn =

    2021 doi

  205. [262]

    doi:10.1145/3382494.3410690 , isbn =

    Di Rocco, Juri and Di Ruscio, Davide and Di Sipio, Claudio and Nguyen, Phuong and Rubei, Riccardo , year = 2020, month = oct, booktitle =. doi:10.1145/3382494.3410690 , isbn =

    2020

  206. [263]

    doi:10.1002/spe.3238 , issn =

    Sas, Cezar and Capiluppi, Andrea and Di Sipio, Claudio and Di Rocco, Juri and Di Ruscio, Davide , year = 2023, month = oct, journal =. doi:10.1002/spe.3238 , issn =

    2023 doi

  207. [264]

    Empirical Software Engineering , publisher =

    Topic recommendation for software repositories using multi-label classification algorithms , author =. Empirical Software Engineering , publisher =. doi:10.1007/s10664-021-09976-2 , url =

    doi

  208. [265]

    Empirical Software Engineering , volume = 28, number = 2, pages = 50, doi =

    Semantically-enhanced topic recommendation systems for software projects , author =. Empirical Software Engineering , volume = 28, number = 2, pages = 50, doi =

  209. [266]

    doi:10.1007/978-3-642-45135-5 , isbn =

    Recommendation. doi:10.1007/978-3-642-45135-5 , isbn =

    doi

  210. [267]

    Journal of the American society for information science , volume = 45, number = 1, pages =

    The relationship between recall and precision , author =. Journal of the American society for information science , volume = 45, number = 1, pages =

  211. [268]

    ACM Trans

    Understanding Software-2.0: A Study of Machine Learning Library Usage and Evolution , author =. ACM Trans. Softw. Eng. Methodol. , publisher =. doi:10.1145/3453478 , issn =

    doi

  212. [269]

    Widyasari, Ratnadira and Zhao, Zhipeng and Cong, Thanh Le and Jin Kang, Hong and Lo, David , year = 2023, month = mar, booktitle =. Topic. doi:10.1109/SANER56733.2023.00025 , isbn =

    2023

  213. [270]

    Mariama Celi Serafim

  214. [271]

    Yuqi Zhou and Jiawei Wu and Yanchun Sun , year = 2021, booktitle =. GHTRec:. doi:10.1109/ICWS53863.2021.00049 , url =

    2021

  215. [272]

    Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering , location =

    Automated categorization of pre-trained models in software engineering: A case study with a Hugging Face dataset , author =. Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering , location =. doi:10.1145/3661167.3661215 , isbn =...

  216. [273]

    Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension , location =

    How do Hugging Face Models Document Datasets, Bias, and Licenses? An Empirical Study , author =. Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension , location =. doi:10.1145/3643916.3644412 , isbn = 9798400705861, url =

  217. [274]

    Towards improving decision making and estimating the value of decisions in value-based software engineering: the

    Mendes, Emilia and Rodriguez, Pilar and Freitas, Vitor and Baker, Simon and Atoui, Mohamed Amine , year = 2018, month = jun, journal =. Towards improving decision making and estimating the value of decisions in value-based software engineering: the. doi:10.1007/s11219-017-9360...

    2018 doi

  218. [275]

    Agno: Framework for AI-driven decision-making , author =

  219. [276]

    Conversational

    Biancofiore, Giovanni Maria and Di Palma, Dario and Pomo, Claudio and Narducci, Fedelucio and Di Noia, Tommaso , year = 2025, booktitle =. Conversational. doi:10.1007/978-3-031-61375-3_4 , isbn =

    2025 doi

  220. [277]

    Onnx model zoo , author =

  221. [278]

    Mindspore , author =

  222. [279]

    Experimentation in

    Wohlin, Claes and Runeson, Per and H\". Experimentation in. doi:10.1007/978-3-642-29044-2 , isbn =

    doi

  223. [280]

    2502.16804 , archiveprefix =

    Multi-Agent Autonomous Driving Systems with Large Language Models: A Survey of Recent Advances , author =. 2502.16804 , archiveprefix =

  224. [281]

    2503.09648 , archiveprefix =

    A Survey on Trustworthy LLM Agents: Threats and Countermeasures , author =. 2503.09648 , archiveprefix =

    arXiv

  225. [282]

    2502.14743 , archiveprefix =

    Multi-Agent Coordination across Diverse Applications: A Survey , author =. 2502.14743 , archiveprefix =

    arXiv

  226. [283]

    2502.14321 , archiveprefix =

    Beyond Self-Talk: A Communication-Centric Survey of LLM-Based Multi-Agent Systems , author =. 2502.14321 , archiveprefix =

    arXiv

  227. [284]

    2501.09136 , archiveprefix =

    Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG , author =. 2501.09136 , archiveprefix =

    arXiv

  228. [285]

    Journal of Systems and Software , volume = 146, pages =

    What’s in a GitHub Star? Understanding Repository Starring Practices in a Social Coding Platform , author =. Journal of Systems and Software , volume = 146, pages =. doi:https://doi.org/10.1016/j.jss.2018.09.016 , issn =

    2018 doi

  229. [286]

    Pattern Recognition , publisher =

    Comparative Analysis of Pretrained Models for Text Classification, Generation and Summarization: A Detailed Analysis , author =. Pattern Recognition , publisher =

  230. [287]

    2504.01963 , archiveprefix =

    LLMs Working in Harmony: A Survey on the Technological Aspects of Building Effective LLM-Based Multi Agent Systems , author =. 2504.01963 , archiveprefix =

    arXiv

  231. [288]

    2501.06322 , archiveprefix =

    Multi-Agent Collaboration Mechanisms: A Survey of LLMs , author =. 2501.06322 , archiveprefix =

    arXiv

  232. [289]

    2210.03629 , archiveprefix =

    ReAct: Synergizing Reasoning and Acting in Language Models , author =. 2210.03629 , archiveprefix =

    arXiv

  233. [290]

    Lin, Chin-Yew , year = 2004, month = jul, booktitle =

    2004

  234. [291]

    Biometrics Bulletin , publisher =

    Individual Comparisons by Ranking Methods , author =. Biometrics Bulletin , publisher =

  235. [292]

    Journal of Systems and Software , volume = 170, pages = 110717, doi =

    Understanding coordination in global software engineering: A mixed-methods study on the use of meetings and Slack , author =. Journal of Systems and Software , volume = 170, pages = 110717, doi =

  236. [293]

    2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C) , volume =

    Combining Quantitative and Qualitative Studies in Empirical Software Engineering Research , author =. 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C) , volume =. doi:10.1109/ICSE-C.2017.163 , keywords =

    2017 doi

  237. [294]

    and Hoda, Rashina and Feldt, Robert , year = 2025, month = mar, journal =

    Seaman, Carolyn B. and Hoda, Rashina and Feldt, Robert , year = 2025, month = mar, journal =. doi:10.1109/TSE.2025.3538751 , issn =

    2025

  238. [295]

    Biometrika , publisher =

    An Analysis of Variance Test for Normality (Complete Samples) , author =. Biometrika , publisher =

  239. [296]

    IEEE Transactions on Software Engineering , volume = 50, number = 1, pages =

    An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation , author =. IEEE Transactions on Software Engineering , volume = 50, number = 1, pages =. doi:10.1109/TSE.2023.3334955 , keywords =

    2023

  240. [297]

    2501.05899 , archiveprefix =

    Prompt engineering and its implications on the energy consumption of Large Language Models , author =. 2501.05899 , archiveprefix =

  241. [298]

    Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , location =

    Optimizing the Utilization of Large Language Models via Schedule Optimization: An Exploratory Study , author =. Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , location =. doi:10.1145/3674805.3686671 , isbn = 9798400...

  242. [299]

    2020 46th

    Supporting the understanding and comparison of low-code development platforms , author =. 2020 46th. doi:10.1109/SEAA51224.2020.00036 , url =

    2020

  243. [300]

    Quasi-experimentation: Design & analysis issues for field settings , author =

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.