Pith. sign in

REVIEW 2 cited by

Automated Test Case Repair Using Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.06765 v4 pith:7U7L7VSZ submitted 2024-01-12 cs.SE

Ahmadreza Saboor Yaraghi , Darren Holden , Nafiseh Kahani , Lionel Briand This is my paper · ORCID
classification cs.SE
keywords testrepairlanguagetargetapproachcasesacrossautomated
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Ensuring the quality of software systems through testing is essential, yet maintaining test cases poses significant challenges and costs. The need for frequent updates to align with the evolving system under test often entails high complexity and cost for maintaining these test cases. Further, unrepaired broken test cases can degrade test suite quality and disrupt the software development process, wasting developers' time. To address this challenge, we present TaRGET (Test Repair GEneraTor), a novel approach leveraging pre-trained code language models for automated test case repair. TaRGET treats test repair as a language translation task, employing a two-step process to fine-tune a language model based on essential context data characterizing the test breakage. To evaluate our approach, we introduce TaRBench, a comprehensive benchmark we developed covering 45,373 broken test repairs across 59 open-source projects. Our results demonstrate TaRGET's effectiveness, achieving a 66.1% exact match accuracy. Furthermore, our study examines the effectiveness of TaRGET across different test repair scenarios. We provide a practical guide to predict situations where the generated test repairs might be less reliable. We also explore whether project-specific data is always necessary for fine-tuning and if our approach can be effective on new projects.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. REACCEPT: Automated Co-evolution of Production and Test Code Based on Dynamic Validation and Large Language Models

    cs.SE 2024-11 reject novelty 6.0 of 10

    REACCEPT uses GPT-4 with retrieval and compile/test/coverage feedback to identify and rewrite outdated Java test code, reporting higher success than prior methods.

  2. A Large-scale Empirical Study on Fine-tuning Large Language Models for Unit Testing

    cs.SE 2024-12 reject novelty 5.0 of 10

    A broad benchmark of fine-tuned LLMs across three unit-testing tasks shows strong gains on assertion and test evolution, but not on test generation.

Pith tools