The reviewed record of science sign in
Pith

arxiv: 2401.06401 · v4 · pith:DRT7NPV7 · submitted 2024-01-12 · cs.SE · cs.AI· cs.CL

DevEval: Evaluating Code Generation in Practical Software Projects

Jia Li , Ge Li , Yunfei Zhao , Yongmin Li , Zhi Jin , Hao Zhu , Huanyu Liu , Kaibo Liu
show 9 more authors
Lecheng Wang Zheng Fang Lanshen Wang Jiazheng Ding Xuanming Zhang Yihong Dong Yuqi Zhu Bin Gu Mengfei Yang
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:DRT7NPV7record.jsonopen to challenge →

classification cs.SE cs.AIcs.CL
keywords practicalprojectsdevevalcodegenerationllmsbenchmarkscontexts
0
0 comments X
read the original abstract

How to evaluate Large Language Models (LLMs) in code generation is an open question. Many benchmarks have been proposed but are inconsistent with practical software projects, e.g., unreal program distributions, insufficient dependencies, and small-scale project contexts. Thus, the capabilities of LLMs in practical projects are still unclear. In this paper, we propose a new benchmark named DevEval, aligned with Developers' experiences in practical projects. DevEval is collected through a rigorous pipeline, containing 2,690 samples from 119 practical projects and covering 10 domains. Compared to previous benchmarks, DevEval aligns to practical projects in multiple dimensions, e.g., real program distributions, sufficient dependencies, and enough-scale project contexts. We assess five popular LLMs on DevEval (e.g., gpt-4, gpt-3.5-turbo, CodeLLaMa, and StarCoder) and reveal their actual abilities in code generation. For instance, the highest Pass@1 of gpt-3.5-turbo only is 42 in our experiments. We also discuss the challenges and future directions of code generation in practical projects. We open-source DevEval and hope it can facilitate the development of code generation in practical projects.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PlayCoder: Making LLM-Generated GUI Code Playable

    cs.SE 2026-04 conditional novelty 7.0

    PlayCoder raises the rate of LLM-generated GUI apps that can be played end-to-end without logic errors from near zero to 20.3% Play@3 by adding repository-aware generation, agent-driven testing, and iterative repair.

  2. RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices

    cs.SE 2026-04 unverdicted novelty 6.0

    RealBench is a new repo-level code generation benchmark that adds UML diagrams to natural language specs, showing LLMs struggle more at full repositories, create modules with errors, and perform best with whole-repo g...

  3. CityRAG: Stepping Into a City via Spatially-Grounded Video Generation

    cs.CV 2026-04 unverdicted novelty 6.0

    CityRAG generates minutes-long 3D-consistent videos of real-world cities by grounding outputs in geo-registered data and using temporally unaligned training to disentangle fixed scenes from transient elements like weather.

  4. CityRAG: Stepping Into a City via Spatially-Grounded Video Generation

    cs.CV 2026-04 conditional novelty 6.0

    PlayCoder combines a repository-aware coding agent with a vision-based GUI testing agent and an automated program repair loop to detect and fix silent logic errors in LLM-generated interactive application code.

  5. RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices

    cs.SE 2026-04 conditional novelty 5.0

    RealBench is a repo-level code generation benchmark pairing UML diagrams with natural language requirements, revealing that LLMs perform significantly worse on realistic repo-level tasks than existing benchmarks suggest.