REVIEW 5 cited by
Large Language Models as Test Case Generators: Performance Evaluation and Enhancement
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Code generation with Large Language Models (LLMs) has been extensively studied and achieved remarkable progress. As a complementary aspect to code generation, test case generation is of crucial importance in ensuring the quality and reliability of code. However, using LLMs as test case generators has been much less explored. Current research along this line primarily focuses on enhancing code generation with assistance from test cases generated by LLMs, while the performance of LLMs in test case generation alone has not been comprehensively examined. To bridge this gap, we conduct extensive experiments to study how well LLMs can generate high-quality test cases. We find that as the problem difficulty increases, state-of-the-art LLMs struggle to generate correct test cases, largely due to their inherent limitations in computation and reasoning. To mitigate this issue, we further propose a multi-agent framework called \emph{TestChain} that decouples the generation of test inputs and test outputs. Notably, TestChain uses a ReAct format conversation chain for LLMs to interact with a Python interpreter in order to provide more accurate test outputs. Our results indicate that TestChain outperforms the baseline by a large margin. Particularly, in terms of the accuracy of test cases, TestChain using GPT-4 as the backbone achieves a 13.84\% improvement over the baseline on the LeetCode-hard dataset.
Forward citations
Cited by 5 Pith papers
-
PROGRESS: Property-Guided Regression Search for Semantic Falsification
PROGRESS embeds LLM-generated jqwik properties as search objectives in EvoSuite's DynaMOSA, detecting 58% of injected Java bugs that regression assertions miss and satisfying hard preconditions 4x more often than stan...
-
Rethinking Verification for LLM Code Generation: From Generation to Testing
SAGA, a human-LLM workflow, generates tests that catch 90.62% of known human bugs on a new benchmark, though those bugs also helped create the tests.
-
MineValiCoder: Reliable Code Generation with Test Case Quality Mining and Bipartite Graph-Based Mutual Validation
A closed-loop test-driven framework - filter tests by LLM self-consistency, refine code in parallel, and select the winner by bipartite code-test mutual scoring - reaches 96.34% Pass@1 on HumanEval without human-writt...
-
Trade Policy and Structural Change
A dynamic multi-country model with nonhomothetic preferences and complementary sectors implies a 20 percentage point U.S. manufacturing tariff raises the manufacturing value-added share by about one percentage point a...
-
HyClone: Bridging LLM Understanding and Dynamic Execution for Semantic Code Clone Detection
A two-stage LLM-plus-execution framework is claimed to improve semantic code clone detection, but the submitted text does not contain the claimed framework.
Discussion (0). Continue with ORCID to comment.