Pith. sign in

REVIEW 20 cited by

Unit Test Case Generation with Transformers and Focal Context

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2009.05617 v2 pith:2PKGYHEB submitted 2020-09-11 cs.SE cs.CLcs.LG

Michele Tufano , Dawn Drain , Alexey Svyatkovskiy , Shao Kun Deng , Neel Sundaresan This is my paper · ORCID
classification cs.SEcs.CLcs.LG
keywords testfocalcasesunitcasemethodsathenatestcontext
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Automated unit test case generation tools facilitate test-driven development and support developers by suggesting tests intended to identify flaws in their code. Existing approaches are usually guided by the test coverage criteria, generating synthetic test cases that are often difficult for developers to read or understand. In this paper we propose AthenaTest, an approach that aims to generate unit test cases by learning from real-world focal methods and developer-written testcases. We formulate unit test case generation as a sequence-to-sequence learning task, adopting a two-step training procedure consisting of denoising pretraining on a large unsupervised Java corpus, and supervised finetuning for a downstream translation task of generating unit tests. We investigate the impact of natural language and source code pretraining, as well as the focal context information surrounding the focal method. Both techniques provide improvements in terms of validation loss, with pretraining yielding 25% relative improvement and focal context providing additional 11.1% improvement. We also introduce Methods2Test, the largest publicly available supervised parallel corpus of unit test case methods and corresponding focal methods in Java, which comprises 780K test cases mined from 91K open-source repositories from GitHub. We evaluate AthenaTest on five defects4j projects, generating 25K passing test cases covering 43.7% of the focal methods with only 30 attempts. We execute the test cases, collect test coverage information, and compare them with test cases generated by EvoSuite and GPT-3, finding that our approach outperforms GPT-3 and has comparable coverage w.r.t. EvoSuite. Finally, we survey professional developers on their preference in terms of readability, understandability, and testing effectiveness of the generated tests, showing overwhelmingly preference towards AthenaTest.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sakura: An Approach for Generating Complex Tests from Natural Language Test Descriptions

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    Sakura is a multi-agent system that generates structurally complex tests from NL descriptions, achieving 50-78% higher compilability and 38-66% higher coverage overlap than baselines on 1,464 scenarios from 20 Apache ...

  2. An Iterative Test-and-Repair Framework for Competitive Code Generation

    cs.SE 2026-04 unverdicted novelty 7.0 of 10

    FixAudit improves LLM code generation on competitive programming benchmarks by training a shared model for iterative code-aware test generation and repair, achieving 35%+ gains in Pass@1 over baselines on the same 7B model.

  3. SCATE: Learning to Supervise Coding Agents for Cost-Effective Test Generation

    cs.SE 2026-07 conditional novelty 6.5 of 10

    A contextual-bandit supervisor learns to route coding agents among default generation, analysis-augmented generation, and stop, yielding large coverage gains over unsupervised agents and non-agentic LLM test generators.

  4. Agent-Based Test Assertion Generation via Diverse Perspective Aggregation

    cs.SE 2026-08 conditional novelty 6.0 of 10

    AssertMate decomposes assertion generation into actual value construction, multi-perspective expected value prediction, and judge-based selection, and reports higher compilation, pass, and bug-detection rates on Defec...

  5. Context Matters: Improving the Practical Reliability of LLM-Based Unit Test Generation

    cs.SE 2026-07 conditional novelty 6.0 of 10

    CATGen improves LLM unit-test reliability by combining structured project-context retrieval, deterministic test-class skeletons, and static-analysis repair, beating six baselines on compilation success, coverage, and cost.

  6. Multi-Agent LLM Collaboration for Unit Test Generation via Human-Testing-Inspired Workflows

    cs.SE 2026-07 conditional novelty 6.0 of 10

    A three-agent LLM workflow plus a test-specialized knowledge graph generates unit tests that beat prior LLM and search-based tools on coverage and mutation score.

  7. Teaching Code LLMs to Reason with Intermediate Formal Specifications

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Verification-guided fine-tuning with mutants and refinement traces teaches CodeLLMs to emit intermediate executable assertions that improve checking and repair on HumanExec.

  8. TATG: Tracking-Aware Testing Objective for LLM-based Test Generation

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Tracking individual static and feedback testing objectives across a coverage-then-mutation two-stage LLM workflow raises line/branch coverage and mutation score on complex Java methods versus KTester and PANTA.

  9. SpecMind: Cognitively Inspired, Interactive Multi-Turn Framework for Postcondition Inference

    cs.SE 2026-02 conditional novelty 6.0 of 10

    A feedback-driven multi-turn prompting framework outperforms single-pass LLM postcondition generation in correctness and completeness on EvalPlus and FixEval.

  10. Compiling Large Multi-Modal Requirement Documents into Runnable Software Systems: From an Agentic Test-Driven Perspective

    cs.SE 2026-02 conditional novelty 6.0 of 10

    ARC compiles DSL-based requirement documents into runnable web apps by generating interface tests top-down, then using those tests to gate bottom-up code generation, achieving higher GUI pass rates than baseline codin...

  11. SAGE: Semantic-Aware Gray-Box Game Regression Testing with Large Language Models

    cs.SE 2025-11 conditional novelty 6.0 of 10

    SAGE generates gray-box game regression tests via LLM-guided RL, compresses them with Pareto optimization, and prioritizes them via LLM-analyzed update logs, beating automated baselines on unique-bug detection at lowe...

  12. Hamster: A Large-Scale Study and Characterization of Developer-Written Tests

    cs.SE 2025-09 conditional novelty 6.0 of 10

    A large-scale study of 1.7M developer-written Java tests shows they commonly use fixtures, mocking, structured inputs, and multi-class scope, features largely missing from automated test generation tools.

  13. Leveraging GPT-4 for Vulnerability-Witnessing Unit Test Generation

    cs.SE 2025-06 conditional novelty 6.0 of 10

    GPT-4 generated syntactically valid vulnerability-witnessing unit tests in 66.5% of runs, semantically valid tests in 7.5%, and useful templates in 68.5%, suggesting a semi-automated role.

  14. CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification

    cs.SE 2025-02 conditional novelty 6.0 of 10

    CLOVER is a new Python unit-test-generation benchmark with 845 problems, contexts up to 128k tokens, and coverage-based retrieval, where all 14 evaluated models score below 35% on the hardest task.

  15. ReProAgent: Tool-Augmented Multi-Stage Agentic Generation of Bug Reproduction Tests from Issue Reports

    cs.SE 2026-07 conditional novelty 5.5 of 10

    A tool-augmented multi-stage agent reproduces 58–70% of real GitHub issues as fail-to-pass tests, outperforming prior prompt and agent baselines at about $0.14 per issue.

  16. Quantize with Confidence? An Empirical Study of Quantization for Code Generation

    cs.SE 2026-07 conditional novelty 5.0 of 10

    4-bit quantization of code LLMs mostly preserves pass@1, but AQLM matches or beats full precision while QuIP# degrades most on complex prompts; sensitivity is model-dependent.

  17. Knowledge Matters: Injecting Project and Testing Knowledge into LLM-based Unit Test Generation

    cs.SE 2025-11 conditional novelty 5.0 of 10

    KTester improves LLM-generated Java unit tests on a 110-method benchmark by injecting static project knowledge and decoupling test-case design from test-method generation, beating four baselines.

  18. Seed&Steer: Guiding Large Language Models with Compilable Prefix and Branch Signals for Unit Test Generation

    cs.SE 2025-07 conditional novelty 5.0 of 10

    Combining EvoSuite invocation examples with branch-intent prompts lets LLMs generate Java unit tests that compile more often and cover more branches.

  19. Quality Assessment of Python Tests Generated by Large Language Models

    cs.SE 2025-06 conditional novelty 4.0 of 10

    A comparative study of Python test suites generated by GPT-4o, Amazon Q, and LLama 3.3 found 151 execution errors and 512 test smells, with assertion failures and low-cohesion tests most common.

  20. A Tool for Generating Exceptional Behavior Tests With Large Language Models

    cs.SE 2025-05 conditional novelty 4.0 of 10

    exLong generates exceptional behavior tests with a fine-tuned LLM using stack traces, guard expressions, and existing tests as context, and claims gains over prior tools.

Pith tools