Pith. sign in

REVIEW 4 major objections 8 minor 62 references

REACCEPT: Automated Co-evolution of Production and Test Code Based on Dynamic Validation and Large Language Models

T0 review · 4 major / 8 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read REACCEPT claims an LLM feedback loop updates 60.16% of obsolete Java tests, doubling the prior best.

desk verdict Useful LLM-driven test co-evolution pipeline and a large public dataset, but the headline update accuracy is circular and needs an independent semantic check before it can be believed. read the letter →

arxiv 2411.11033 v1 pith:IB2PA4CS submitted 2024-11-17 cs.SE

Jianlei Chi , Xiaotian Wang , Yuhan Huang , Lechen Yu , Di Cui , Jianguo Sun , Jun Sun This is my paper · ORCID
classification cs.SE
keywords product-testco-evolutionobsoletetestidentificationcoderepairlargelanguagemodelsdynamicvalidationretrieval-augmentedgenerationReActmechanismJavaunittesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the full production-test co-evolution loop, deciding which tests are obsolete after a code change and rewriting those tests, can be automated well enough for routine use. It proposes REACCEPT, an LLM-based pipeline that identifies obsolete Java tests, generates replacements, then runs each candidate through compile, execution, and coverage checks, feeding failures back into the model for up to eight rounds. On a benchmark built from 537 Java projects, REACCEPT updated 60.16% of correctly identified obsolete tests, roughly double the 31.62% of the previous best method. The intended payoff is a tool that keeps test suites synchronized with evolving code with limited human review.

What carries the argument

The central mechanism is the dynamic-validation feedback loop. REACCEPT decomposes test quality into four levels, compilation failure, test failure, coverage failure, and success, and uses the failure category to select a specialized prompt template; retrieval-augmented generation supplies a similar historical production/test diff pair as a few-shot example, and the ReAct-style agent interleaves reasoning with tool access. The identity carrying the argument is the valid-update predicate: a rewritten test counts as correct only if javac compiles it, JUnit runs it without failure, and JaCoCo shows it covers the changed statements in the production code.

What would settle it

Take a random sample of REACCEPT's successfully updated tests, mutate the changed production code so that the covered statements remain present but the behavior changes, for example, flip a comparison or return a wrong value, and rerun the updated tests: if most still pass, the tests are not validating the new behavior and the accuracy claim is an artifact of the validity predicate. A second check is to compare each updated test's assertions with the original test's assertions and count how many were deleted, weakened, or adjusted to match the new output.

Watch

Extended reading notes

Core claim

On its own terms, REACCEPT's central discovery is that dynamic feedback closes the gap that defeated earlier learning-based updaters. Earlier methods generated test code in one shot and scored it with textual similarity; REACCEPT instead treats update generation as an iterative repair loop. The LLM proposes an updated test, the Java compiler, the JUnit runner, and the JaCoCo coverage tool check it, the error, failure, or uncovered-statement messages are turned into new prompts, and the cycle repeats until the test compiles, passes, and covers the changed production statements or a cutoff is reached. With that loop, 71.84% of update tasks succeeded end-to-end, and the paper attributes the improvement to the dynamic-validation feedback, which reduced LLM hallucination and raised update effectiveness in ablation.

Load-bearing premise

The claim rests on treating compiles, passes, and covers the changed statements as a correct test update; if those three checks do not guarantee that the test meaningfully checks the new behavior, the reported 60 to 72 percent accuracy overstates how well REACCEPT preserves test intent.

Editorial extensions

If this is right

  • REACCEPT's dynamic validation raises the share of updates that compile, pass, and cover changes to 71.84%, versus 35.92% for the best earlier method.
  • The identification phase is separate and can be swapped: the paper says CEPROT identifies obsolete tests slightly better, while REACCEPT wins overall through its update phase, so combining a stronger identifier with REACCEPT's updater should improve end-to-end accuracy.
  • Ablation shows both RAG and dynamic validation contribute; removing dynamic validation drops the coverage-based success rate to 52.43%, and removing RAG drops it to 50.49%.
  • Most successful updates, 70.27%, pass on the first iteration, and 22 of the remaining 29 are fixed through feedback, so the loop's value concentrates on hard cases.
  • The evaluation is on Java, but the paper states that the underlying pipeline, which relies on a compiler, a test runner, and a coverage tool, can transfer to other programming languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported accuracy may overstate semantic correctness, because a test that compiles, passes, and covers changed statements can still pass after an LLM weakens or deletes an assertion; a stronger metric would compare assertion strength or run mutation testing on the changed code.
  • The 60.16% figure is conditional on correct identification and was measured on 103 dynamically evaluable samples from six projects, so end-to-end usefulness in the wild depends on pairing the updater with an identifier at least as accurate as CEPROT's 97.5% identification accuracy.
  • Because 70% of updates succeed on the first generation, a cheaper two-stage design, one-shot LLM update followed by targeted repair only for failures, could capture most of the benefit at lower token cost.
  • The paper itself notes that retrieval failures on dddlib trace to similar edit structures with different identifiers, suggesting that replacing textual-similarity retrieval with structure-aware retrieval over AST changes is a testable extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes REACCEPT, an LLM-based approach for automating product-test co-evolution. The method has three phases: preprocessing (mining commit diffs and building a vector knowledge base), identification (LLM classification with experience-based prompts to decide whether a test must change), and updating (LLM test generation with RAG, ReAct-style reasoning, and an iterative dynamic-validation loop that feeds compiler, JUnit, and JaCoCo messages back to the LLM until the test compiles, passes, and covers the changed production statements). The evaluation uses a dataset of 537 Java projects (23,403 samples) for identification, and 103 runnable samples from 6 projects for dynamic update evaluation. The paper reports identification F1 of about 96%, an update accuracy (UCR) of 71.84%, and an end-to-end accuracy of 60.16%, surpassing CEPROT's 31.62% by about 90%.

Significance. If the reported accuracy reflects genuinely correct test updates, this is a practically significant contribution: it would be one of the first fully automated PT co-evolution pipelines with a majority success rate, and the proposed combination of LLM prompting, RAG, and dynamic validation is a natural and reusable design. The paper ships a replication package, uses real compiler/JUnit/JaCoCo feedback rather than text-only evaluation, and reports CodeBLEU alongside dynamic metrics. However, the headline numbers currently rest on a small, environment-filtered dynamic evaluation and on a success criterion that is exactly what the iterative loop optimizes, so the significance hinges on additional evidence that the updated tests are semantically meaningful, not merely that the loop terminates.

major comments (4)
  1. [Section 4.2.3 and Section 3.4.4] The update success criterion (UCR) is the same compile-pass-cover condition that the dynamic-validation loop in Figure 4 is explicitly designed to satisfy. Consequently, the reported 71.84% update accuracy and the 60.16% end-to-end accuracy primarily measure how often the loop reaches its stopping condition, not whether the updated test meaningfully validates the new production behavior. A generated test can satisfy the criterion by deleting or relaxing assertions, aligning expected values with observed output, or invoking the changed method with arguments that touch the modified statements while asserting nothing. The paper reports no assertion-preservation check, no mutation analysis, and no manual semantic review of successful updates, yet Section 1 claims the updated tests 'confirm correctness in both syntax and semantics.' I recommend adding an assertion-preservation analysis, mutation testing on the changed statements, and/or manual inspection of a random sample of successful updates to substantiate the central claim.
  2. [Table 2, Table 5, and Figure 6] The central update and end-to-end results are based on only 103 runnable samples from 6 Maven projects (74 of 94 commits; 103 of 130 collected samples), not on the 537-project, 23,403-sample dataset used for identification. This small, environment-filtered sample seriously limits the generality of the 71.84% and 60.16% headline figures. The paper should report the exact filtering criteria, provide per-project and per-commit breakdowns with confidence intervals, and be explicit that the update evaluation is a pilot-scale study rather than a full benchmark.
  3. [Section 3.3.2] The identification prompt and 'experience' are described as resulting from 'multiple rounds of iterative optimization' with prompt content adjusted 'based on the identification results and the actual performance of the LLM.' If any of this tuning used the 10% test split, the reported identification F1 of 96.01% and the downstream end-to-end accuracy would be overfit estimates. The paper must state explicitly that all prompt and experience tuning was done only on the training split, or re-run the identification with a fresh, frozen-prompt test set. This is load-bearing for RQ1 and RQ2, not a presentation issue.
  4. [Section 4.3.3 and Figure 7d] The comparison with CEPROT is asymmetric: REACCEPT's update loop receives compiler, JUnit, and JaCoCo feedback for up to 8 iterations, while the baselines receive no dynamic feedback. The reported 90% relative improvement over CEPROT therefore conflates the benefit of the dynamic-validation mechanism with the benefit of the LLM and RAG components. A fairer comparison would give baselines the same oracle feedback, or present the dynamic-validation gain only as an ablation (as in Figure 7d). This does not invalidate the method, but it weakens the headline 'surpassing CEPROT by 90%' claim as a statement about the whole approach versus a prior method.
minor comments (8)
  1. [Abstract] The sentence 'we extensive experiments' should read 'we conducted extensive experiments.'
  2. [Figure 6] The caption uses 'CodeBLUE' instead of 'CodeBLEU.'
  3. [Figure 3] The prompt template has an unmatched closing tag '<test_prod>' that should likely be '</old_test>', and the placeholder 'The answer is yes/no' is awkwardly worded.
  4. [Section 3.4.2] The parameter name 'prod_samlpe' is a typo for 'prod_sample.'
  5. [Equation 4 and surrounding text] The phrase 'remaining chord similarity' should be 'cosine similarity.'
  6. [Section 4.3.4] The iteration analysis says 74 samples were successfully updated, while Table 5 is based on 103 samples; clarify the denominator used in Figure 7e.
  7. [Table 1] The 'Data Set' header is malformed; the columns should be labeled clearly as Train/Test and Positive/Negative.
  8. [Section 4.2.1] The text says 'The first two columns record each project's name and the number of collected commits,' but Table 2 has three columns (Project, Commits, Samples); please align the description.

Circularity Check

1 steps flagged · score 6.0 of 10

REACCEPT's headline 'update accuracy' (71.84% and 60.16%) is measured with the same compile–pass–cover criterion that its dynamic-validation loop is explicitly engineered to satisfy, making the central success claim partly self-fulfilling.

  1. self definitional [Section 3.4.4 (Test Update Validation) and Section 4.2.3 (Evaluation Metrics)]
    "The dynamic validation process will keep executing the process shown in Figure 4 until generating a valid test code or reaching a pre-defined cutoff ... When the test code passes the checks of compilation, re-execution, and test coverage, ReAccept will stop further interaction with the LLM and output the updated test code. ... Update Coverage Rate (UCR): the percentage of updated test code that successfully runs, passes, and covers the changes in the product code."

    Section 3.4.1 defines a 'valid update' as test code that 'must cover the modified statements in the production code while not arising any runtime errors,' and Section 4.2.3 defines UCR as the percentage that 'successfully runs, passes, and covers the changes.' These are the same three conditions (compile, JUnit pass, JaCoCo coverage of changed statements) that the update loop iteratively enforces before it stops and emits the updated test. Therefore the headline 71.84% update accuracy and the end-to-end 60.16% count how often the system reaches its own acceptance condition, rather than independently verifying that the updated test meaningfully validates the new production behavior.

full rationale

The central update-accuracy claim is partially circular: UCR is the same compile–pass–cover predicate that REACCEPT's dynamic-validation loop is designed to optimize as its stopping condition. This is a genuine reduction-by-construction of the main reported success metric. However, the circularity is not total: the loop does not always converge (UCR is 71.84%, not 100%), so the rate still measures something empirical about convergence; the identification phase (RQ2) is evaluated against independent human-provided labels, with a reported 98.13% accuracy; and CodeBLEU (82.03%) is an external, non-optimized metric that also favors REACCEPT over the baselines. No load-bearing self-citation chain or imported uniqueness theorem was found. Accordingly, the score is 6: one key 'prediction' (update accuracy) reduces by construction to the system's own acceptance criterion, while other parts of the evaluation retain independent content.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

This is an empirical software engineering paper rather than a mathematical derivation. The free parameters are hyperparameters and prompt content that were chosen or tuned in the study. The axioms are domain assumptions about the validity of commit-history labels, the sufficiency of dynamic validation as a correctness criterion, and the representativeness of the six-project evaluation.

free parameters (8)
  • Experience prompt content = Text prompt, no fixed numeric value
    The identification and update prompts, including the AL/PI/CP experience list, were optimized using the LLM's performance on the evaluation data (Section 3.3.2).
  • Maximum dynamic validation iterations = 8
    Paper states 'currently, we empirically set the maximum rounds of iteration to 8' (Section 3.4.4); affects update success rate.
  • LLM model = gpt-4-0125-preview
    Selected after ablation over four GPT models (Section 4.3.4); all headline results depend on this choice.
  • Temperature = 0
    Default and ablation-validated; lower temperature improves CSR and TPS (Figure 7b).
  • Top P = 1
    Default and ablation-validated; higher Top P improves update performance (Figure 7c).
  • Tokenization block size = 50
    Chosen in Section 3.2.2 for diff tokenization; affects retrieval and prompt quality.
  • RAG retrieved sample count = 1 (most similar)
    The update prompt uses 'the retrieved most similar historical records' (Section 3.4.1); the number of samples is not ablated.
  • Context window size = 3
    Default setting for the conversation chain (Section 4); affects how much feedback the LLM sees.
assumptions (6)
  • domain assumption Co-change of production and test code in a commit is a valid ground-truth label for obsolete tests.
    Positive samples are mined from commits where production and test changed together (Section 4.2.1); the authors admit in Section 4.5.2 that some pairs are uncorrelated with mismatched names.
  • domain assumption A test that compiles, passes, and covers the changed statements is a correct update.
    This is the definition of UCR, the headline metric (Section 4.2.3); it fails if assertions are weakened or removed.
  • domain assumption The six selected Maven projects are representative of Java projects for evaluating updates.
    Only 6 of 537 projects could be built and run for dynamic validation (Table 2); the paper acknowledges this selection in Section 4.2.1.
  • domain assumption GPT-4 can follow the summarized experience and repair code from compiler/test/coverage error messages.
    The whole method relies on proprietary LLM behavior; this is tested empirically but not guaranteed and not reproducible in closed form.
  • domain assumption Cosine similarity on ada-002 embeddings of tokenized diffs retrieves useful exemplars.
    The RAG component assumes semantic similarity of code diffs aligns with update relevance; Section 4.4 shows it can retrieve suboptimal samples.
  • domain assumption Maven naming conventions allow reliable pairing of production and test code.
    Projects were chosen for 'well-organized repository structures and clear naming conventions' (Section 3.2.1), which limits external validity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of REACCEPT: Automated Co-evolution of Production and Test Code Based on Dynamic Validation and Large Language Models." pith.science (2026). https://pith.science/paper/IB2PA4CS

@misc{pith2026241111033,
  author       = {Pith},
  title        = {Pith review of: REACCEPT: Automated Co-evolution of Production and Test Code Based on Dynamic Validation and Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IB2PA4CS}},
  note         = {Machine review of arXiv:2411.11033}
}
read the original abstract

Synchronizing production and test code, known as PT co-evolution, is critical for software quality in the software development lifecycle. Existing methods for automatic PT co-evolution either utilize predefined heuristic rules or rely on simple application of machine learning techniques. Due to the limitations of underlying techniques, existing methods either only partially automate PT co-evolution (e.g., only automate obsolete test code identification) or result in low accuracy. In this paper, we propose REACCEPT, a novel approach that leverages large language models and dynamic validation to fully automate PT co-evolution (i.e., capable of both identifying and updating obsolete test cases). REACCEPT relies on experience-based prompt template generation, dynamic validation, and retrieval-augmented generation techniques to accomplish automated PT co-evolution. To evaluate REACCEPT's effectiveness, we extensive experiments with a dataset of 537 Java projects and compared REACCEPT's performance with several state-of-the-art methods. Results show that REACCEPT achieved an update accuracy of 60.16% on correctly identified obsolete test code, surpassing the state-of-the-art technique CEPROT by 90%. This confirms that REACCEPT can effectively assist developers in maintaining test code, improving overall software quality and reducing maintenance effort.

Figures

Figures reproduced from arXiv: 2411.11033 by the authors.

Figure 1
Figure 1. Example of obsolete test code • Maintaining a test suite is expensive. Given the limited time and resources, developers often deprioritize the co-evolution of test code. • Developers may not be cognizant of all relevant test code related to a specific feature, resulting in the negligence of obsolete test code. • Even experienced developers may not understand the whole project due to the complexity of modern software… view at source ↗
Figure 2
Figure 2. Overview of ReAccept’s workflow 3 Our Method In this section, we introduce the workflow of ReAccept and describe the details of its key components. 3.1 Overall Workflow The ultimate of ReAccept is to automate the PT co-evolution completely, i.e., automatically identify and update obsolete test code. While numerous novel approaches have been proposed in recent years [13, 21, 27, 31–33, 47, 50, 51, 53], most focus pri… view at source ↗
Figure 3
Figure 3. The prompt template of the identification task [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Test process decomposition 3.3.3 Identification Result Generation. After multiple rounds of optimization and experience accu￾mulation, our prompt design can now accurately guide the LLM in identifying the obsolete test code. During the process of generating identificat…
Figure 5
Figure 5. Figure 5: The prompt template of the updating task [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: The overall effectiveness of different approaches [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Ablation study results 4.4 Discussion It was observed that ReAccept’s performance on dddlib is not as strong as expected. After an in-depth investigation on dddlib, it turns out that the root cause of such performance downgrade lies in the unsuccessful updated samples.…
Figure 8
Figure 8. Figure 8: Impact of dynamic validation on updating obsolete test code [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 43 canonical work pages

  1. [1]

    REACCEPT: REasoning-Action mechanism and Code dynamic validation assisted Co-Evolution of Production and Test code

    Augest 2, 2024. REACCEPT: REasoning-Action mechanism and Code dynamic validation assisted Co-Evolution of Production and Test code. https://github.com/Timiyang-ai/REACCEPT.git

    work page 2024

  2. [2]

    Chroma - the open-source embedding database

    July 30, 2024. Chroma - the open-source embedding database. https://github.com/chroma-core/chroma

    work page 2024

  3. [3]

    JaCoCo Java Code Coverage Library

    July 30, 2024. JaCoCo Java Code Coverage Library. https://www.jacoco.org/jacoco/

    work page 2024

  4. [4]

    javac - Java programming language compiler

    July 30, 2024. javac - Java programming language compiler. https://docs.oracle.com/javase/7/docs/technotes/tools/ solaris/javac.html

    work page 2024

  5. [5]

    July 30, 2024. JUnit 4. https://junit.org/junit4/

    work page 2024

  6. [6]

    Langchain Framework

    July 30, 2024. Langchain Framework. https://www.langchain.com/

    work page 2024

  7. [7]

    Machine Learning Platform - Text Analysis Service | Datumbox

    July 30, 2024. Machine Learning Platform - Text Analysis Service | Datumbox. https://www.datumbox.com/

    work page 2024

  8. [8]

    OpenAI Embeddings

    July 30, 2024. OpenAI Embeddings. https://platform.openai.com/docs/guides/embeddings

    work page 2024

Show all 62 references
  1. [9]

    Georg Buchgeher, Christian Ernstbrunner, Rudolf Ramler, and Michael Lusser. 2013. Towards tool-support for test case selection in manual regression testing. In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops. IEEE, 74–79

    2013

  2. [10]

    Yufan Cai, Yun Lin, Chenyan Liu, Jinglian Wu, Yifan Zhang, Yiming Liu, Yeyun Gong, and Jin Song Dong. 2024. On-the-fly adapting code summarization on trainable cost-effective language models. Advances in Neural Information Processing Systems 36 (2024)

    2024

  3. [11]

    Saikat Chakraborty, Shuvendu K Lahiri, Sarah Fakhoury, Madanlal Musuvathi, Akash Lal, Aseem Rastogi, Aditya Senthilnathan, Rahul Sharma, and Nikhil Swamy. 2023. Ranking llm-generated loop invariants for program verification. arXiv preprint arXiv:2310.09342 (2023)

    2023 arXiv

  4. [12]

    Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guo- liang Dong, Artem Aliev, et al . 2024. CodeR: Issue Resolving with Multi-Agent and Task Graphs. arXiv preprint arXiv:2406.01304 (2024)

    2024 arXiv

  5. [13]

    Jianlei Chi, Yu Qu, Ting Liu, Qinghua Zheng, and Heng Yin. 2022. Seqtrans: automatic vulnerability fix via sequence to sequence learning. IEEE Transactions on Software Engineering 49, 2 (2022), 564–585

    2022

  6. [14]

    Emelie Engström and Per Runeson. 2010. A qualitative survey of regression testing practices. In Product-Focused Software Process Improvement: 11th International Conference, PROFES 2010, Limerick, Ireland, June 21-23, 2010. Proceedings

    2010

  7. [15]

    Jean-Rémy Falleri, Floréal Morandat, Xavier Blanc, Matias Martinez, and Martin Monperrus. 2014. Fine-grained and accurate source code differencing. In Proceedings of the 29th ACM/IEEE international conference on Automated software engineering. 313–324

    2014

  8. [16]

    Michael Fu, Chakkrit Tantithamthavorn, Trung Le, Van Nguyen, and Dinh Phung. 2022. VulRepair: a T5-based automated software vulnerability repair. In Proceedings of the 30th ACM joint european software engineering conference and symposium on the foundations of software engineer...

    2022

  9. [17]

    Mats Grindal, Jeff Offutt, and Jonas Mellin. 2006. On the testing maturity of software producing organizations. In Testing: Academic & Industrial Conference-Practice And Research Techniques (TAIC PART’06) . IEEE, 171–180

    2006

  10. [18]

    Qiuhan Gu. 2023. Llm-based code generation method for golang compiler testing. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 2201–2203

    2023

  11. [19]

    Qi Guo, Xiaohong Li, Xiaofei Xie, Shangqing Liu, Ze Tang, Ruitao Feng, Junjie Wang, Jidong Ge, and Lei Bu. 2024. FT2Ra: A Fine-Tuning-Inspired Approach to Retrieval-Augmented Code Completion. arXiv preprint arXiv:2404.01554 (2024)

    2024 arXiv

  12. [20]

    Yuejun Guo, Qiang Hu, Xiaofei Xie, Maxime Cordy, Mike Papadakis, and Yves Le Traon. 2023. KAPE: k NN-based performance testing for deep code search. ACM Transactions on Software Engineering and Methodology 33, 2 (2023), 1–24

    2023

  13. [21]

    Xing Hu, Zhuang Liu, Xin Xia, Zhongxin Liu, Tongtong Xu, and Xiaohu Yang. 2023. Identify and Update Test Cases When Production Code Changes: A Transformer-Based Approach. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1111–1122. ,...

    2023

  14. [22]

    Yuan Huang, Zhicao Tang, Xiangping Chen, and Xiaocong Zhou. 2024. Towards automatically identifying the co-change of production and test code. Software Testing, Verification and Reliability 34, 3 (2024), e1870

    2024

  15. [23]

    Victor Hurdugaci and Andy Zaidman. 2012. Aiding software developers to maintain developer tests. In 2012 16th European Conference on Software Maintenance and Reengineering . IEEE, 11–20

    2012

  16. [24]

    Marko Ivankovic, Goran Petrovic, Yana Kulizhskaya, Mateusz Lewko, Luka Kalinovcic, René Just, and Gordon Fraser

  17. [25]

    Nan Jiang, Thibaud Lutellier, and Lin Tan. 2021. Cure: Code-aware neural machine translation for automatic program repair. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . IEEE, 1161–1173

    2021

  18. [26]

    Kailun Jin, Chung-Yu Wang, Hung Viet Pham, and Hadi Hemmati. 2024. Can ChatGPT Support Developers? An Empirical Evaluation of Large Language Models for Code Generation. In 2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR) . IEEE, 167–171

    2024

  19. [27]

    Tenma Kitai, Hirohisa Aman, Sousuke Amasaki, Tomoyuki Yokogawa, and Minoru Kawahara. 2022. Have Java Production Methods Co-Evolved With Test Methods Properly?: A Fine-Grained Repository-Based Co-Evolution Analysis. In 2022 48th Euromicro Conference on Software Engineering and ...

    2022

  20. [28]

    Bonan Kou, Shengmai Chen, Zhijie Wang, Lei Ma, and Tianyi Zhang. 2023. Is model attention aligned with human attention? an empirical study on large language models for code generation. arXiv preprint arXiv:2306.01220 (2023)

    2023 arXiv

  21. [29]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing S...

    2020

  22. [30]

    Xiangyu Li, Marcelo d’Amorim, and Alessandro Orso. 2019. Intent-preserving test repair. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). IEEE, 217–227

    2019

  23. [31]

    Xiaoli Lian, Shuaisong Wang, Jieping Ma, Xin Tan, Fang Liu, Lin Shi, Cuiyun Gao, and Li Zhang. 2024. Imperfect Code Generation: Uncovering Weaknesses in Automatic Code Generation by Large Language Models. In Proceedings of the 2024 IEEE/ACM 46th International Conference on Sof...

    2024

  24. [32]

    Chao Liu, Xuanlin Bao, Hongyu Zhang, Neng Zhang, Haibo Hu, Xiaohong Zhang, and Meng Yan. 2024. Guiding ChatGPT for Better Code Generation: An Empirical Study. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 102–113

    2024

  25. [33]

    Lei Liu, Sinan Wang, Yepang Liu, Jinliang Deng, and Sicen Liu. 2023. Drift: Fine-Grained Prediction of the Co-Evolution of Production and Test Code via Machine Learning. In Proceedings of the 14th Asia-Pacific Symposium on Internetware . 227–237

    2023

  26. [34]

    Zeeger Lubsen, Andy Zaidman, and Martin Pinzger. 2009. Using association rules to study the co-evolution of production & test code. In 2009 6th IEEE International Working Conference on Mining Software Repositories . IEEE, 151–154

    2009

  27. [35]

    Cosmin Marsavina, Daniele Romano, and Andy Zaidman. 2014. Studying fine-grained co-evolution patterns of production and test code. In 2014 IEEE 14th International Working Conference on Source Code Analysis and Manipulation . IEEE, 195–204

    2014

  28. [36]

    Simone Mezzaro, Alessio Gambi, and Gordon Fraser. 2024. An Empirical Study on How Large Language Models Impact Software Testing Learning. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 555–564

    2024

  29. [37]

    Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2332–2354

    2024

  30. [38]

    Wendkûuni C Ouédraogo, Kader Kaboré, Haoye Tian, Yewei Song, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F Bissyandé. 2024. Large-scale, Independent and Comprehensive study of the power of LLMs for test case generation. arXiv preprint arXiv:2407.00225 (2024)

    2024

  31. [39]

    Fabio Palomba, Annibale Panichella, Andy Zaidman, Rocco Oliveto, and Andrea De Lucia. 2016. Automatic test case generation: What if test code quality matters?. In Proceedings of the 25th International Symposium on Software Testing and Analysis. 130–141

    2016

  32. [40]

    Jiantao Pan. 1999. Software testing. Dependable Embedded Systems 5, 2006 (1999), 1

    1999

  33. [41]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

    2002

  34. [42]

    Yun Peng, Shuzheng Gao, Cuiyun Gao, Yintong Huo, and Michael Lyu. 2024. Domain knowledge matters: Improving prompts with fix templates for repairing python type errors. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering . 1–13

    2024

  35. [43]

    Leandro Sales Pinto, Saurabh Sinha, and Alessandro Orso. 2012. Understanding myths and realities of test-suite evolution. In Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering . , Vol. 1, No. 1, Article . Publication date: No...

    2012

  36. [44]

    Sanyogita Piya and Allison Sullivan. 2023. LLM4TDD: Best Practices for Test Driven Development Using Large Language Models. arXiv preprint arXiv:2312.04687 (2023)

    2023 arXiv

  37. [45]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297 (2020)

    2020 arXiv

  38. [46]

    Per Runeson. 2006. A survey of unit testing practices. IEEE software 23, 4 (2006), 22–29

    2006

  39. [47]

    Samiha Shimmi and Mona Rahimi. 2022. Leveraging code-test co-evolution patterns for automated test case recom- mendation. In Proceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test . 65–76

    2022

  40. [48]

    Jiho Shin, Sepehr Hashtroudi, Hadi Hemmati, and Song Wang. 2024. Domain Adaptation for Code Model-based Unit Test Case Generation. arXiv:2308.08033 [cs.SE] https://arxiv.org/abs/2308.08033

    2024 arXiv

  41. [49]

    Mats Skoglund and Per Runeson. 2004. A case study on regression test suite maintenance in system evolution. In 20th IEEE International Conference on Software Maintenance, 2004. Proceedings. IEEE, 438–442

    2004

  42. [50]

    Jeongju Sohn and Mike Papadakis. 2022. Using Evolutionary Coupling to Establish Relevance Links Between Tests and Code Units. A case study on fault localization. arXiv preprint arXiv:2203.11343 (2022)

    2022 arXiv

  43. [51]

    Weifeng Sun, Meng Yan, Zhongxin Liu, Xin Xia, Yan Lei, and David Lo. 2023. Revisiting the Identification of the Co-evolution of Production and Test Code. ACM Transactions on Software Engineering and Methodology 32, 6 (2023), 1–37

    2023

  44. [52]

    Bart Van Rompaey and Serge Demeyer. 2009. Establishing traceability links between unit test cases and units under test. In 2009 13th European Conference on Software Maintenance and Reengineering . IEEE, 209–218

    2009

  45. [53]

    Sinan Wang, Ming Wen, Yepang Liu, Ying Wang, and Rongxin Wu. 2021. Understanding and facilitating the co-evolution of production and test code. In 2021 IEEE International conference on software analysis, evolution and reengineering (SANER). IEEE, 272–283

    2021

  46. [54]

    Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. 2024. TESTEVAL: Benchmarking Large Language Models for Test Case Generation. arXiv preprint arXiv:2406.04531 (2024)

    2024 arXiv

  47. [55]

    Robert White, Jens Krinke, and Raymond Tan. 2020. Establishing multilevel test-to-code traceability links. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering . 861–872

    2020

  48. [56]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022)

    2022 arXiv

  49. [57]

    Ahmadreza Saboor Yaraghi, Darren Holden, Nafiseh Kahani, and Lionel Briand. 2024. Automated Test Case Repair Using Language Models. arXiv preprint arXiv:2401.06765 (2024)

    2024 arXiv

  50. [58]

    Zhiqiang Yuan, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng, and Yiling Lou. 2024. Evaluating and Improving ChatGPT for Unit Test Generation. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1703–1726

    2024

  51. [59]

    Andy Zaidman, Bart Van Rompaey, Serge Demeyer, and Arie Van Deursen. 2008. Mining software repositories to study co-evolution of production & test code. In 2008 1st international conference on software testing, verification, and validation. IEEE, 220–229

    2008

  52. [60]

    Andy Zaidman, Bart Van Rompaey, Arie Van Deursen, and Serge Demeyer. 2011. Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining. Empirical Software Engineering 16 (2011), 325–364

    2011

  53. [61]

    Daniel Zimmermann and Anne Koziolek. 2023. Automating gui-based software testing with gpt-3. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) . IEEE, 62–65. , Vol. 1, No. 1, Article . Publication date: November 2024

    2023

  54. [2024]

    In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice

    Productive Coverage: Improving the Actionability of Code Coverage. In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice . 58–68

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.