Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

Large Language Models for Code Generation: The Practitioners Perspective

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A practitioner survey of 60 developers ranks GPT-4o first for real-world code generation and GPT-3.5 Turbo last.

desk verdict The survey numbers don't add up: 14 valid responses in the methods, 60 practitioners in the results, and the headline ranking rests on the larger denominator. read the letter →

arxiv 2501.16998 v1 pith:ISI5RKBW submitted 2025-01-28 cs.SE

Zeeshan Rasheed , Muhammad Waseem , Kai Kristian Kemell , Aakash Ahmad , Malik Abdul Sami , Jussi Rasku , Kari Systä , Pekka Abrahamsson This is my paper · ORCID
classification cs.SE
keywords LargeLanguageModelsCodeGenerationPractitionerSurveyGPT-4oDevelopmentBotsAIforSoftwareEngineeringEmpiricalEvaluationBenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the usual way of evaluating code-generating language models—running them against benchmark tasks with predefined test cases—misses what working software developers care about. To test that, the authors built a single web platform that lets practitioners select among eight LLMs, describe a real project in natural language, and generate and run code. Sixty practitioners from eleven countries used the platform and answered a survey; on the basis of their responses, the paper claims GPT-4o is the most effective model overall, particularly for Python, with Llama 3.2 3B Instruct a cheap and fast second choice for smaller tasks and GPT-3.5 Turbo the least effective. The paper presents the survey results, preference counts, and practitioner quotes as evidence that model choice in practice turns on accuracy, syntax, context understanding, and debugging effort—aspects it says synthetic benchmarks underweight.

What carries the argument

The load-bearing mechanism is a purpose-built multi-model code-generation platform. It routes natural-language prompts through the OpenRouter API to eight LLMs, executes the generated code in a sandbox, and records which model produced each output. Practitioners then answered a 15-question survey whose closed-ended items produced preference counts and whose two open-ended items were analyzed with open and selective coding. The counts—31 of 60 choosing GPT-4o, 17 choosing Llama 3.2, and 7 choosing Mixtral—are the quantitative core that turns a collection of opinions into a claimed ranking.

What would settle it

A reader could go to the released survey data and recompute the preference counts under a consistent inclusion rule; if the denominator is 14 rather than 60, or if 31 practitioners did not actually name GPT-4o as best, the claimed ranking would not hold. Replicating the survey with a new 60-person sample and finding a different top model would also settle the question.

Watch

Extended reading notes

Core claim

The paper's central claim is that a practitioner-centered evaluation of eight LLMs, carried out through a custom unified platform, produces a clear ranking that existing benchmarks do not. In the survey, 31 of 60 practitioners rated GPT-4o the best model for code generation, citing accurate syntax, compliance with best practices, effective library use, and reliable handling of both front-end and back-end tasks; 17 preferred Llama 3.2 3B Instruct for its concise, fast, and cost-effective output on smaller projects; 7 selected Mixtral 8x7B Instruct for consistency in data-analysis and machine-learning workflows; and GPT-3.5 Turbo was rated the worst model, with practitioners reporting outdated methods, frequent errors, and outputs that required more debugging than writing the code from scratch. The paper also reports that 92 percent of participants found the platform easy to use and that Python dominated usage at 56 percent.

Load-bearing premise

The entire ranking rests on the claim that exactly 60 valid practitioners answered the survey; the paper also reports retaining only 14 valid responses in the methods section, and every percentage depends on which denominator is correct.

Editorial extensions

If this is right

  • Teams choosing an LLM for general-purpose code generation should expect GPT-4o to require the least post-generation debugging, especially on Python tasks.
  • Llama 3.2 3B Instruct is a viable low-cost, low-latency default for small or straightforward tasks, where its concise output offsets its limited complexity.
  • Models that score well on benchmark correctness may still disappoint in practice if their code is outdated, verbose, or context-blind; GPT-3.5 Turbo is the paper's example.
  • Benchmarks such as HumanEval and MBPP should be supplemented with practitioner studies that capture usability and maintenance effort, since those attributes drive real-world model choice.
  • Replication of the ranking by other teams can reuse the released platform and data to check whether the GPT-4o preference generalizes beyond this sample.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The platform-plus-survey design implies a template for evaluating future LLMs: generate, execute, and ask practitioners to judge debugging effort, without waiting for benchmark suites to catch up.
  • The 56 percent Python share suggests the ranking may transfer most directly to Python-centric teams; a language-balanced replication would show whether GPT-4o's margin holds for Java, C#, or PHP.
  • The emphasis on code that required less debugging hints that effort-based metrics, such as time to make generated code compile and pass tests, could quantify what the survey captures qualitatively.
  • Since the platform logs project descriptions, the released data could be mined to connect model preference with task type, such as API, front-end, or data processing, a connection the paper reports only anecdotally.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents a multi-model code-generation platform that integrates several large language models and a practitioner survey meant to compare their effectiveness for real-world code generation. The authors report that GPT-4o is the most effective model (31/60 practitioners), Llama 3.2 3B Instruct is second (17/60), and GPT-3.5 Turbo is the least effective. The study also claims to provide a publicly available platform and dataset for replication. The main contributions are empirical practitioner feedback on model usability, accuracy, and limitations, framed as complementing existing benchmarks such as HumanEval and MBPP.

Significance. If the reported survey were reliably conducted and consistently reported, the paper would provide a useful practitioner-oriented complement to benchmark-based LLM code-generation evaluations, along with a reusable platform and open data. The claim that practitioners prefer GPT-4o over Llama 3.2, Mixtral, and GPT-3.5 Turbo is concrete and falsifiable, and the public repository is a positive step for replication. However, the manuscript currently contains severe internal inconsistencies in the sample size, response rates, model counts, demographic percentages, and per-model counts. Since every preference ranking and percentage in the results is a simple aggregate over the survey responses, these inconsistencies are load-bearing: the central claim is not supported by a consistent evidence base as written.

major comments (5)
  1. [§3.2, §4.1, §5.3] The sample size is irreconcilable across the paper. Section 3.2 states: 'We received 27 responses with an approximate response rate of 14.65%... we retained 14 valid responses for data analysis.' Section 4.1 states: '60 practitioners participated and completed the survey.' Section 5.3 states: 'we received only 60 valid responses, i.e, response rate was 10.3%.' If 157 invitations were sent, then 60/157 = 38.2%, not 10.3%, while 14/157 = 8.9% is close to 10.3%. Every reported count and percentage (31/60, 17/60, 52%, 28%, 12%, demographic shares) uses 60 as the denominator. The authors must reconcile these numbers and state exactly how many valid responses were analyzed. If the true valid sample is 14, the reported counts and percentages are impossible as printed; if it is 60, Section 3.2 and the response-rate statement are incorrect and the survey process is not transparently reported.
  2. [§1, §3.1, Figure 4] The paper repeatedly claims that eight LLMs were evaluated, but the actual lists are inconsistent. The Introduction lists six models ('Llama 3.2 3B Instruct, Mixtral 8×7B Instruct, GPT-4o, GPT-4 Turbo, GPT-3.5 Turbo, and Gemini 1.5 Flask-8B'), Section 3.1 lists seven models ('Llama 3.2 3B Instruct, Mixtral 8*7B Instruct, GPT-3.5, GPT-4, GPT-4 Turbo, GPT-4o, and Google Gemini'), and Figure 4 shows seven model columns. The authors must specify exactly which models were integrated into the platform and evaluated, and ensure the count and the reported results align with that list.
  3. [§4.3, Figure 4] The counts for Mixtral 8*7B Instruct are self-contradictory. The text first says '9 considered Mixtral 8*7B Instruct the best-performing model,' then later says '7 out of 60 practitioners selected Mixtral 8*7B Instruct as the best model,' and the takeaways state that 12% of practitioners selected it. Since 9/60 = 15% and 7/60 ≈ 12%, these statements cannot all be correct. The per-model counts and the corresponding percentages in Figure 4 need to be made mutually consistent for every model, not just Mixtral.
  4. [§4.2] The usability statistics are numerically impossible. The text reports that '53 practitioners (92%)' found the system easy to use and '8 practitioners (8%)' found it challenging. These two counts sum to 61, not 60, and 8/60 = 13.3%, not 8%. The same issue appears in the following paragraph, where 'approximately 92%' and 'about 8%' do not correspond to any consistent partition of 60 respondents. The counts and percentages need to be corrected against the actual data.
  5. [§4.1, Figure 3] The demographic text and Figure 3 do not agree. The text says 25% of respondents were from Europe, 24% from Asia, and 7% from North America, but Figure 3(a) displays Europe (25, 42%), Asia (24, 40%), and North America (7, 12%). Similar mismatches appear for experience and qualification: the text cites '33.55%' and '20.33%' while the figure shows counts and percentages such as 33, 55% and 20, 33%. The authors should reconcile the figure and the prose and use one consistent set of demographic statistics.
minor comments (5)
  1. [§3.2] The participant-selection paragraph contains an internal non sequitur: after stating 27 responses were received, it says 'Among the 23 responses, 9 invalid responses were excluded.' Please clarify whether 23 or 27 responses were received before exclusions.
  2. [§5.3] The word 'presentative' should be 'representative' in the sentence 'This makes us confident that the results are presentative and generalizable to some extent.'
  3. [§3.1, Figure 4] The model names are spelled inconsistently: 'Gemini 1.5 Flask-8B' appears in the Introduction while Section 3.1 and Figure 4 use 'Gemini 1.5 Flash' or 'Flasks'; 'Mixteral' appears in Figure 4 instead of 'Mixtral.' Please use consistent, correct model names throughout.
  4. [§4.1] The citation to Guest et al. [17] for '12 to 15 participants' being sufficient for data saturation refers to interview studies, not survey research; the relevance of this citation to the survey sample size should be justified or removed.
  5. [Figure 2] The workflow figure states 'Survey Invitations = 157, Final Responses = 60, Excluded = 9,' which conflicts with the numbers in Section 3.2. This figure should be updated to match the reconciled sample-size numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the GPT-4o preference ranking is a direct survey tabulation, not a fitted or self-derived result; the sample-size inconsistency is a soundness issue, not circularity.

full rationale

This paper contains no derivation chain that could be circular in the sense of this review. Its central claim—that practitioners prefer GPT-4o for code generation—is a direct tabulation of survey responses (31 out of 60 in Figure 4 and Section 4.3), not the output of a fitted model, an assumed ansatz, or a theorem imported from the authors' prior work. The model ranking is therefore a descriptive summary of the input data rather than a prediction derived from assumptions that already encode the result. Self-citations appear in the related work and in references to the authors' own platform and prior papers (e.g., CodePori [32] and the GitHub repository [21]), but none is load-bearing: the evaluation platform is described in Section 3.1 independently of those citations, and the survey instrument is the actual evidence base. The paper's validity discussion contains a serious internal inconsistency—Section 3.2 reports 27 responses and 14 retained valid responses for analysis, while Section 4 and Section 5.3 use 60 respondents and a claimed 10.3% response rate that does not match 60 out of 157 invitations—but that is a correctness, transparency, and reproducibility threat, not a circularity step, because the reported ranking is not obtained through an equation or argument that presupposes its own conclusion. No circular step can be exhibited, so the score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted because the paper is an observational survey rather than a quantitative model. The central claim rests on the validity and correct tallying of the survey sample, on self-report as a proxy for code quality, and on the platform invoking models as intended. These assumptions are flagged in Section 3.2 and Section 5.3. No new entities are introduced.

assumptions (4)
  • domain assumption The 60 respondents are distinct valid practitioners whose answers are accurately tallied.
    Section 4.1 reports 60 valid participants, but Section 3.2 says 27 responses were received and 14 valid responses were retained. The ranking in Section 4.3 depends on the 60-person denominator.
  • domain assumption Survey self-reports are a valid proxy for code generation functionality, syntax, and accuracy.
    Section 4.3 derives preferences from survey questions Q10 to Q15, but no execution-based quality metrics are reported despite the abstract claiming assessment of functionality and accuracy.
  • domain assumption Recruitment via social networks, GitHub, Stack Overflow, and snowball sampling yields a representative practitioner sample.
    Section 3.2 describes a non-probability sampling strategy, and Section 5.3 concedes external validity risk from the small response rate and unbalanced demographics.
  • domain assumption The OpenRouter API and the proposed platform invoke each LLM correctly and do not bias outputs.
    Section 3.1 relies on OpenRouter endpoints and error handling, but the paper reports no independent checks of generated code or model responses beyond practitioner opinion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Models for Code Generation: The Practitioners Perspective." pith.science (2026). https://pith.science/paper/ISI5RKBW

@misc{pith2026250116998,
  author       = {Pith},
  title        = {Pith review of: Large Language Models for Code Generation: The Practitioners Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ISI5RKBW}},
  note         = {Machine review of arXiv:2501.16998}
}
read the original abstract

Large Language Models (LLMs) have emerged as coding assistants, capable of generating source code from natural language prompts. With the increasing adoption of LLMs in software development, academic research and industry based projects are developing various tools, benchmarks, and metrics to evaluate the effectiveness of LLM-generated code. However, there is a lack of solutions evaluated through empirically grounded methods that incorporate practitioners perspectives to assess functionality, syntax, and accuracy in real world applications. To address this gap, we propose and develop a multi-model unified platform to generate and execute code based on natural language prompts. We conducted a survey with 60 software practitioners from 11 countries across four continents working in diverse professional roles and domains to evaluate the usability, performance, strengths, and limitations of each model. The results present practitioners feedback and insights into the use of LLMs in software development, including their strengths and weaknesses, key aspects overlooked by benchmarks and metrics, and a broader understanding of their practical applicability. These findings can help researchers and practitioners make informed decisions for systematically selecting and using LLMs in software development projects. Future research will focus on integrating more diverse models into the proposed system, incorporating additional case studies, and conducting developer interviews for deeper empirical insights into LLM-driven software development.

Figures

Figures reproduced from arXiv: 2501.16998 by the authors.

Figure 1
Figure 1. An overview of the proposed study Gemini 1.5 Flask-8B. To achieve this objective, as shown in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Workflow for solution execution and evaluation gateway between the application and the above-mentioned LLMs. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Practitioners’ feedback on the performance of models for code generation (7 out of 60) practitioners selected Mixtral 8*7B Instruct as the best model for code generation. Those who preferred it highlighted several unique advan￾tages offered by the model. For example, (…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Pilot Study on LLM-Based Agentic Translation from Android to iOS: Pitfalls and Insights

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A five-project pilot study found that a GPT-4o agent pipeline translates 70.7% of Android files to valid Swift after automated validation, with failures concentrated in internal references and platform-specific APIs.

  2. LLM-Generated Microservice Implementations from RESTful API Definitions

    cs.SE 2025-02 conditional novelty 6.0 of 10

    A GPT-4-based multi-agent pipeline can generate OpenAPI specs, Express.js server code, and iterative log-driven fixes for CRUD microservices, and six surveyed practitioners found it useful.

  3. Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Research

    cs.RO 2025-06 accept novelty 1.0 of 10

    A perspective article reviews the state of using foundation models for laboratory automation and proposes a roadmap for fully autonomous experiments.

Reference graph

Works this paper leans on

47 extracted references · 19 canonical work pages · cited by 3 Pith papers

  1. [1]

    In: Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering

    Ahmad, A., Waseem, M., Liang, P., Fahmideh, M., Aktar, M.S., Mikkonen, T.: To- wards human-bot collaborative software architecting with chatgpt. In: Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering. pp. 279–285 (2023)

    work page 2023

  2. [2]

    arXiv preprint arXiv:2312.11444 (2023)

    Akter, S.N., Yu, Z., Muhamed, A., Ou, T., Bäuerle, A., Cabrera, Á.A., Dholakia, K., Xiong, C., Neubig, G.: An in-depth look at gemini’s language abilities. arXiv preprint arXiv:2312.11444 (2023)

    arXiv 2023

  3. [3]

    arXiv preprint arXiv:2108.07732 (2021)

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al.: Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  4. [4]

    If you use this software, please cite it using these metadata58 (2021)

    Black, S., Gao, L., Wang, P., Leahy, C., Biderman, S.: Gpt-neo: Large scale autore- gressive language modeling with mesh-tensorflow. If you use this software, please cite it using these metadata58 (2021)

    work page 2021

  5. [5]

    Journal of Methods and Measurement in the Social Sciences6(1), 14–29 (2015)

    Blair, E.: A reflexive exploration of two qualitative data coding techniques. Journal of Methods and Measurement in the Social Sciences6(1), 14–29 (2015)

    2015

  6. [6]

    IEEE Transactions on Software Engineering 49(7), 3675–3691 (2023)

    Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.H., Zi, Y., Anderson, C.J., Feldman, M.Q., et al.: Multipl-e: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering 49(7), 3675–3691 (2023)

    work page 2023

  7. [7]

    In: Proceedings of the 12th ACM/IEEE international symposium on empirical software engineering and mea- surement

    Chakraborty, P., Shahriyar, R., Iqbal, A., Bosu, A.: Understanding the software development practices of blockchain projects: a survey. In: Proceedings of the 12th ACM/IEEE international symposium on empirical software engineering and mea- surement. pp. 1–10 (2018)

    work page 2018

  8. [8]

    arXiv preprint arXiv:2207.10397 (2022)

    Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.G., Chen, W.: Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397 (2022)

    arXiv 2022

Show all 47 references
  1. [9]

    arXiv preprint arXiv:2408.16498 (2024)

    Chen, L., Guo, Q., Jia, H., Zeng, Z., Wang, X., Xu, Y., Wu, J., Wang, Y., Gao, Q., Wang, J., et al.: A survey on evaluating large language models in code generation tasks. arXiv preprint arXiv:2408.16498 (2024)

    2024 arXiv

  2. [10]

    arXiv preprint arXiv:2107.03374 (2021)

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.: Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    2021 arXiv

  3. [11]

    arXiv preprint arXiv:2407.06153 (2024) Large Language Models for Code Generation: The Practitioners’ Perspective 19

    Dou, S., Jia, H., Wu, S., Zheng, H., Zhou, W., Wu, M., Chai, M., Fan, J., Huang, C., Tao, Y., et al.: What’s wrong with your code generated by large language models? an extensive study. arXiv preprint arXiv:2407.06153 (2024) Large Language Models for Code Generation: The Pract...

    2024

  4. [12]

    arXiv preprint arXiv:2308.01861 (2023)

    Du, X., Liu, M., Wang, K., Wang, H., Liu, J., Chen, Y., Feng, J., Sha, C., Peng, X., Lou, Y.: Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861 (2023)

    2023 arXiv

  5. [13]

    arXiv preprint arXiv:2402.00689 (2024)

    Elgedawy, R., Sadik, J., Dutta, S., Gautam, A., Georgiou, K., Gholamrezae, F., Ji, F., Lim, K., Liu, Q., Ruoti, S.: Ocassionally secure: A comparative analysis of code generation assistants. arXiv preprint arXiv:2402.00689 (2024)

    2024

  6. [14]

    In: 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE)

    Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., Zhang, J.M.: Large language models for software engineering: Survey and open problems. In: 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). pp. 3...

    2023

  7. [15]

    arXiv preprint arXiv:2204.05999 (2022)

    Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.t., Zettlemoyer, L., Lewis, M.: Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999 (2022)

    2022 arXiv

  8. [16]

    arXiv e-prints pp

    Ghosh Paul, D., Zhu, H., Bayley, I.: Benchmarks and metrics for evaluations of code generation: A critical review. arXiv e-prints pp. arXiv–2406 (2024)

    2024

  9. [17]

    Field methods18(1), 59–82 (2006)

    Guest, G., Bunce, A., Johnson, L.: How many interviews are enough? an experi- ment with data saturation and variability. Field methods18(1), 59–82 (2006)

    2006

  10. [18]

    arXiv preprint arXiv:2105.09938 (2021)

    Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., et al.: Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 (2021)

    2021 arXiv

  11. [19]

    arXiv preprint arXiv:2308.00352 (2023)

    Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S.K.S., Lin, Z., Zhou, L., et al.: Metagpt: Meta programming for multi-agent col- laborative framework. arXiv preprint arXiv:2308.00352 (2023)

    2023 arXiv

  12. [20]

    In: Guide to advanced empirical software engineering, pp

    Kitchenham, B.A., Pfleeger, S.L.: Personal opinion surveys. In: Guide to advanced empirical software engineering, pp. 63–92. Springer (2008)

    2008

  13. [21]

    Laboratory, G.: Llm evaluation platform.https://github.com/GPT-Laboratory/ LLM-Evaluation (2025), accessed: January 26, 2025

    2025

  14. [22]

    Knowledge Manage- ment & E-Learning13(4), 408 (2021)

    Li, Y., Shyr, C., Borycki, E.M., Kushniruk, A.W.: Automated thematic analysis of health information technology (hit) related incident reports. Knowledge Manage- ment & E-Learning13(4), 408 (2021)

    2021

  15. [23]

    Science378(6624), 1092–1097 (2022)

    Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al.: Competition-level code generation with alphacode. Science378(6624), 1092–1097 (2022)

    2022

  16. [24]

    ACM Transactions on Software Engineering and Methodology (2024)

    Lyu, M.R., Ray, B., Roychoudhury, A., Tan, S.H., Thongtanunam, P.: Automatic programming: Large language models and beyond. ACM Transactions on Software Engineering and Methodology (2024)

    2024

  17. [25]

    arXiv preprint arXiv:2310.18648 (2023)

    Nguyen-Duc, A., Cabrero-Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.K., et al.: Generative artificial intelligence for software engineering–a research agenda. arXiv preprint arXiv:2310.18648 (2023)

    2023 arXiv

  18. [26]

    arXiv preprint arXiv:2203.13474 (2022)

    Nijkamp,E.,Pang,B.,Hayashi,H.,Tu,L.,Wang,H.,Zhou,Y.,Savarese,S.,Xiong, C.: Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474 (2022)

    2022 arXiv

  19. [27]

    arXiv preprint arXiv:2402.16694 (2024)

    Peng, Q., Chai, Y., Li, X.: Humaneval-xl: A multilingual code generation benchmark for cross-lingual natural language generalization. arXiv preprint arXiv:2402.16694 (2024)

    2024 arXiv

  20. [28]

    arXiv preprint arXiv:2105.12655 (2021) 20 Z

    Puri, R., Kung, D.S., Janssen, G., Zhang, W., Domeniconi, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M., Decker, L., et al.: Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks. arXiv preprint arXiv:2105.12655 (2021) 20 Z. Rasheed et al

    2021 arXiv

  21. [29]

    arXiv preprint arXiv:2307.07924 (2023)

    Qian, C., Cong, X., Yang, C., Chen, W., Su, Y., Xu, J., Liu, Z., Sun, M.: Commu- nicative agents for software development. arXiv preprint arXiv:2307.07924 (2023)

    2023 arXiv

  22. [30]

    arXiv preprint arXiv:2411.08507 (2024)

    Rasheed, Z., Sami, M.A., Rasku, J., Kemell, K.K., Zhang, Z., Harjamaki, J., Sid- deeq, S., Lahti, S., Herda, T., Nurminen, M., et al.: Timeless: A vision for the next generation of software development. arXiv preprint arXiv:2411.08507 (2024)

    2024 arXiv

  23. [31]

    arXiv preprint arXiv:2311.18440 (2023)

    Rasheed, Z., Waseem, M., Kemell, K.K., Xiaofeng, W., Duc, A.N., Systä, K., Abra- hamsson, P.: Autonomous agents in software development: A vision paper. arXiv preprint arXiv:2311.18440 (2023)

    2023 arXiv

  24. [32]

    arXiv preprint arXiv:2402.01411 (2024)

    Rasheed, Z., Waseem, M., Saari, M., Systä, K., Abrahamsson, P.: Codepori: Large scale model for autonomous software development by using multi-agents. arXiv preprint arXiv:2402.01411 (2024)

    2024 arXiv

  25. [33]

    In: ICLR 2024 Workshop on Large Language Model (LLM) Agents

    Rasheed, Z., Waseem, M., Systä, K., Abrahamsson, P.: Large language model eval- uation via multi ai agents: Preliminary results. In: ICLR 2024 Workshop on Large Language Model (LLM) Agents

    2024

  26. [34]

    arXiv preprint arXiv:2308.12950 (2023)

    Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X.E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al.: Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950 (2023)

    2023 arXiv

  27. [35]

    Empirical software engineering14, 131–164 (2009)

    Runeson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empirical software engineering14, 131–164 (2009)

    2009

  28. [36]

    In: International Conference on Product-Focused Software Process Improvement

    Sami, M.A., Waseem, M., Zhang, Z., Rasheed, Z., Systä, K., Abrahamsson, P.: Early results of an ai multiagent system for requirements elicitation and analysis. In: International Conference on Product-Focused Software Process Improvement. pp. 307–316. Springer (2024)

    2024

  29. [37]

    ACM Transactions on Design Automation of Electronic Systems29(3), 1–31 (2024)

    Thakur, S., Ahmad, B., Pearce, H., Tan, B., Dolan-Gavitt, B., Karri, R., Garg, S.: Verigen: A large language model for verilog code generation. ACM Transactions on Design Automation of Electronic Systems29(3), 1–31 (2024)

    2024

  30. [38]

    O’Reilly Media, Inc

    Tunstall, L., Von Werra, L., Wolf, T.: Natural language processing with transform- ers. " O’Reilly Media, Inc." (2022)

    2022

  31. [39]

    Wang, B., Komatsuzaki, A.: Gpt-j-6b: A 6 billion parameter autoregressive lan- guage model (2021)

    2021

  32. [40]

    arXiv preprint arXiv:2203.11171 (2022)

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

    2022 arXiv

  33. [41]

    arXiv preprint arXiv:2109.00859 (2021)

    Wang, Y., Wang, W., Joty, S., Hoi, S.C.: Codet5: Identifier-aware unified pre- trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859 (2021)

    2021 arXiv

  34. [42]

    Web engineering pp

    Wohlin, C., Höst, M., Henningsson, K.: Empirical research methods in web and software engineering. Web engineering pp. 409–430 (2006)

    2006

  35. [43]

    In: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming

    Xu, F.F., Alon, U., Neubig, G., Hellendoorn, V.J.: A systematic evaluation of large language models of code. In: Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming. pp. 1–10 (2022)

    2022

  36. [44]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    Xu, P., Shao, W., Zhang, K., Gao, P., Liu, S., Lei, M., Meng, F., Huang, S., Qiao, Y., Luo, P.: Lvlm-ehub: A comprehensive evaluation benchmark for large vision- language models. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  37. [45]

    arXiv preprint arXiv:1809.08887 (2018) Large Language Models for Code Generation: The Practitioners’ Perspective 21

    Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al.: Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887 (2018) Large Language Models...

    2018 arXiv

  38. [46]

    arXiv preprint arXiv:2303.17568 (2023)

    Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Wang, Z., Shen, L., Wang, A., Li, Y., et al.: Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568 (2023)

    2023 arXiv

  39. [47]

    arXiv preprint arXiv:2311.10372 (2023)

    Zheng, Z., Ning, K., Wang, Y., Zhang, J., Zheng, D., Ye, M., Chen, J.: A survey of large language models for code: Evolution, benchmarking, and future trends. arXiv preprint arXiv:2311.10372 (2023)

    2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.