REVIEW 7 cited by
TESTEVAL: Benchmarking Large Language Models for Test Case Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Testing plays a crucial role in the software development cycle, enabling the detection of bugs, vulnerabilities, and other undesirable behaviors. To perform software testing, testers need to write code snippets that execute the program under test. Recently, researchers have recognized the potential of large language models (LLMs) in software testing. However, there remains a lack of fair comparisons between different LLMs in terms of test case generation capabilities. In this paper, we propose TESTEVAL, a novel benchmark for test case generation with LLMs. We collect 210 Python programs from an online programming platform, LeetCode, and design three different tasks: overall coverage, targeted line/branch coverage, and targeted path coverage. We further evaluate sixteen popular LLMs, including both commercial and open-source ones, on TESTEVAL. We find that generating test cases to cover specific program lines/branches/paths is still challenging for current LLMs, indicating a lack of ability to comprehend program logic and execution paths. We have open-sourced our dataset and benchmark pipelines at https://github.com/LLM4SoftwareTesting/TestEval.
Forward citations
Cited by 7 Pith papers
-
SCATE: Learning to Supervise Coding Agents for Cost-Effective Test Generation
A contextual-bandit supervisor learns to route coding agents among default generation, analysis-augmented generation, and stop, yielding large coverage gains over unsupervised agents and non-agentic LLM test generators.
-
Benchmarking LLMs for Unit Test Generation from Real-World Functions
A new decontaminated benchmark of complex Python functions shows LLMs generate far fewer correct, covering, and bug-killing unit tests than on older benchmarks.
-
Rethinking Verification for LLM Code Generation: From Generation to Testing
SAGA, a human-LLM workflow, generates tests that catch 90.62% of known human bugs on a new benchmark, though those bugs also helped create the tests.
-
Can LLMs Generate High-Quality Test Cases for Algorithm Problems? TestCase-Eval: A Systematic Evaluation of Fault Coverage and Exposure
TestCase-Eval introduces fault coverage and fault exposure tasks over 500 Codeforces problems and finds that 19 LLMs, led by Qwen3-32B, remain far below human expert performance.
-
Trade Policy and Structural Change
A dynamic multi-country model with nonhomothetic preferences and complementary sectors implies a 20 percentage point U.S. manufacturing tariff raises the manufacturing value-added share by about one percentage point a...
-
HyClone: Bridging LLM Understanding and Dynamic Execution for Semantic Code Clone Detection
A two-stage LLM-plus-execution framework is claimed to improve semantic code clone detection, but the submitted text does not contain the claimed framework.
-
Comparative Evaluation of Large Language Models for Test-Skeleton Generation
In a single-prompt study on one Ruby class, DeepSeek-Chat produced the most maintainable RSpec test skeletons, while GPT-4's high method coverage was undercut by incorrect RSpec conventions.
Discussion (0). Sign in to comment.