{"id":"894ffb7d-2d50-46ec-bf45-86b8f07711c5","arxiv_id":"2411.11033","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"REACCEPT uses GPT-4 with retrieval and compile/test/coverage feedback to identify and rewrite outdated Java test code, reporting higher success than prior methods.","lead":"This paper presents REACCEPT, a system that uses a large language model to spot test code made obsolete by production code changes and to rewrite those tests. The authors report that the rewritten tests compile, pass, and cover the changed code in about 60% of correctly identified cases, roughly double the rate of the previous best method.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The update accuracy is measured with the same compile–pass–cover criterion that the loop optimizes; nothing prevents assertion deletion or vacuous tests, so 71.84%/60.16% do not yet establish meaningful test updates.","rationale":"The reader's REJECT verdict is appropriately supported, and the weakest assumption I identify is the same one the reader flagged: defining 'correct update' as compile, pass, and cover. This is load-bearing because every headline number (71.84% UCR, 60.16% end-to-end, 'surpassing CEPROT by 90%') is computed against that definition, and because the definition is precisely the target that the dynamic-validation loop optimizes. The paper gives no evidence that the generated tests preserve or strengthen the original assertions or would fail on behavior-changing mutants of the new production code. This is not a critique of the engineering: the ReAct loop, RAG, and ablation studies are real contributions, and the dynamic-validation feedback likely does help fix compilation and runtime errors. But the core claim is about assisting developers to keep tests meaningful, and the reported metric cannot distinguish a genuine update from a vacuous one. A mutation-testing and assertion-preservation study on the 74 successful updates would settle the question. Secondary concerns—the small 103-sample dynamic evaluation, lack of significance testing, and prompt tuning—remain, but the semantic-validity confound is the most serious. Since the current evidence does not support the central claim as stated, I do not recommend changing the reader's verdict.","tokens_in":21453,"tokens_out":9400,"duration_ms":185139,"concrete_test":"Take the 74 updated tests that passed REACCEPT's validator on the dynamic benchmark (Table 5). (1) Run PIT mutation testing on the changed production methods using each updated test and compute the mutation score; compare it with the mutation score of the original developer-written tests on the same methods. If the updated tests kill substantially fewer mutants than the original tests, or score near zero, then pass-plus-coverage is insufficient evidence of correct updates. (2) Automatically diff the assertion statements between test_old and test_new, counting deleted or weakened assertions (e.g., removed assertEquals, changed tolerance, replaced expected values); manually inspect every changed assertion. If a meaningful fraction of passing updates delete or weaken assertions, the 71.84%/60.16% figures overstate the quality of co-evolution. Passing both checks would resolve the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"REACCEPT's central claim is that it can automatically update obsolete tests. The paper defines a valid update (Section 3.4.4, Section 4.2.3) as test code that compiles, passes, and covers the changed production statements. The update loop (Figure 4, Section 3.4.4) iteratively feeds compiler, JUnit, and JaCoCo feedback to the LLM until exactly that criterion is met. Consequently, UCR (71.84% in Table 5) and the end-to-end 60.16% (Figure 6) count how often the system satisfies its own stopping condition, not whether the updated test meaningfully validates the new behavior. A generated test can satisfy the criterion by deleting or relaxing assertions, aligning expected values to observed output, or invoking the changed method with arguments that touch modified statements while asserting nothing. The paper reports no assertion-preservation check, no mutation analysis, and no manual semantic review, and the comparison to CEPROT partly contrasts a loop with dynamic oracle feedback against a baseline without it. The reported accuracy therefore does not establish that co-evolved tests correctly validate the new production behavior, which is the basis of the claim that REACCEPT can assist developers.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes REACCEPT, an LLM-based approach for automating product-test co-evolution. The method has three phases: preprocessing (mining commit diffs and building a vector knowledge base), identification (LLM classification with experience-based prompts to decide whether a test must change), and updating (LLM test generation with RAG, ReAct-style reasoning, and an iterative dynamic-validation loop that feeds compiler, JUnit, and JaCoCo messages back to the LLM until the test compiles, passes, and covers the changed production statements). The evaluation uses a dataset of 537 Java projects (23,403 samples) for identification, and 103 runnable samples from 6 projects for dynamic update evaluation. The paper reports identification F1 of about 96%, an update accuracy (UCR) of 71.84%, and an end-to-end accuracy of 60.16%, surpassing CEPROT's 31.62% by about 90%.","tokens_in":21658,"tokens_out":4953,"duration_ms":53835,"significance":"If the reported accuracy reflects genuinely correct test updates, this is a practically significant contribution: it would be one of the first fully automated PT co-evolution pipelines with a majority success rate, and the proposed combination of LLM prompting, RAG, and dynamic validation is a natural and reusable design. The paper ships a replication package, uses real compiler/JUnit/JaCoCo feedback rather than text-only evaluation, and reports CodeBLEU alongside dynamic metrics. However, the headline numbers currently rest on a small, environment-filtered dynamic evaluation and on a success criterion that is exactly what the iterative loop optimizes, so the significance hinges on additional evidence that the updated tests are semantically meaningful, not merely that the loop terminates.","major_comments":[{"comment":"The update success criterion (UCR) is the same compile-pass-cover condition that the dynamic-validation loop in Figure 4 is explicitly designed to satisfy. Consequently, the reported 71.84% update accuracy and the 60.16% end-to-end accuracy primarily measure how often the loop reaches its stopping condition, not whether the updated test meaningfully validates the new production behavior. A generated test can satisfy the criterion by deleting or relaxing assertions, aligning expected values with observed output, or invoking the changed method with arguments that touch the modified statements while asserting nothing. The paper reports no assertion-preservation check, no mutation analysis, and no manual semantic review of successful updates, yet Section 1 claims the updated tests 'confirm correctness in both syntax and semantics.' I recommend adding an assertion-preservation analysis, mutation testing on the changed statements, and/or manual inspection of a random sample of successful updates to substantiate the central claim.","section":"Section 4.2.3 and Section 3.4.4"},{"comment":"The central update and end-to-end results are based on only 103 runnable samples from 6 Maven projects (74 of 94 commits; 103 of 130 collected samples), not on the 537-project, 23,403-sample dataset used for identification. This small, environment-filtered sample seriously limits the generality of the 71.84% and 60.16% headline figures. The paper should report the exact filtering criteria, provide per-project and per-commit breakdowns with confidence intervals, and be explicit that the update evaluation is a pilot-scale study rather than a full benchmark.","section":"Table 2, Table 5, and Figure 6"},{"comment":"The identification prompt and 'experience' are described as resulting from 'multiple rounds of iterative optimization' with prompt content adjusted 'based on the identification results and the actual performance of the LLM.' If any of this tuning used the 10% test split, the reported identification F1 of 96.01% and the downstream end-to-end accuracy would be overfit estimates. The paper must state explicitly that all prompt and experience tuning was done only on the training split, or re-run the identification with a fresh, frozen-prompt test set. This is load-bearing for RQ1 and RQ2, not a presentation issue.","section":"Section 3.3.2"},{"comment":"The comparison with CEPROT is asymmetric: REACCEPT's update loop receives compiler, JUnit, and JaCoCo feedback for up to 8 iterations, while the baselines receive no dynamic feedback. The reported 90% relative improvement over CEPROT therefore conflates the benefit of the dynamic-validation mechanism with the benefit of the LLM and RAG components. A fairer comparison would give baselines the same oracle feedback, or present the dynamic-validation gain only as an ablation (as in Figure 7d). This does not invalidate the method, but it weakens the headline 'surpassing CEPROT by 90%' claim as a statement about the whole approach versus a prior method.","section":"Section 4.3.3 and Figure 7d"}],"minor_comments":[{"comment":"The sentence 'we extensive experiments' should read 'we conducted extensive experiments.'","section":"Abstract"},{"comment":"The caption uses 'CodeBLUE' instead of 'CodeBLEU.'","section":"Figure 6"},{"comment":"The prompt template has an unmatched closing tag '<test_prod>' that should likely be '</old_test>', and the placeholder 'The answer is yes/no' is awkwardly worded.","section":"Figure 3"},{"comment":"The parameter name 'prod_samlpe' is a typo for 'prod_sample.'","section":"Section 3.4.2"},{"comment":"The phrase 'remaining chord similarity' should be 'cosine similarity.'","section":"Equation 4 and surrounding text"},{"comment":"The iteration analysis says 74 samples were successfully updated, while Table 5 is based on 103 samples; clarify the denominator used in Figure 7e.","section":"Section 4.3.4"},{"comment":"The 'Data Set' header is malformed; the columns should be labeled clearly as Train/Test and Positive/Negative.","section":"Table 1"},{"comment":"The text says 'The first two columns record each project's name and the number of collected commits,' but Table 2 has three columns (Project, Commits, Samples); please align the description.","section":"Section 4.2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a software-engineering venue and the authors have made a replication package available. The main issue is not novelty or presentation but the adequacy of the success criterion: the update metric is also the loop's optimization target, so the reported accuracy does not yet establish that the co-evolved tests meaningfully validate changed behavior. I view this as fixable within the manuscript's scope by adding mutation/assertion-preservation/manual-inspection analyses, clarifying held-out prompt tuning, and being transparent about the 103-sample update evaluation. The reader's reject verdict is understandable given the strength of the circularity concern, but I would not reject outright because the claimed method remains plausible and the required evidence is obtainable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know about arXiv:2411.11033. First, the system is real and reasonably engineered: REACCEPT combines an LLM with RAG and a compile/test/coverage feedback loop to both identify and update obsolete Java tests, and the authors release a 537-project, 23k-sample dataset. Second, the headline update numbers (71.84% UCR, 60.16% end-to-end) are not yet trustworthy, because the success metric is the same condition the dynamic validation loop optimizes.\n\nWhat is genuinely new here is the full pipeline. Prior work like CEPROT did identification and attempted updates, but REACCEPT's integration of retrieval-augmented generation and dynamic validation feedback for both phases is a step forward. The identification experiment is the stronger part: 2,231 test-set samples, F1 around 96-98%, competitive with CEPROT. The ablation studies are detailed and honestly reported, including the odd result that gpt-4o does slightly worse than gpt-4-0125-preview. The paper is clearly written, and the threats-to-validity section is candid about data noise and retrieval limitations.\n\nThe soft spot is the update metric. Section 3.4.4 defines a valid update as code that compiles, passes, and covers the changed statements; the loop then feeds compiler, JUnit, and JaCoCo feedback to the LLM until exactly that condition holds. So the UCR and the end-to-end accuracy measure how often the system satisfies its own stopping condition. A test can satisfy it by deleting or loosening assertions, aligning expected values to observed output, or invoking the changed method with arguments that happen to reach the modified statements while asserting nothing. The paper reports no assertion-preservation check, no mutation analysis, and no manual semantic review. The comparison to CEPROT is fair in the sense that both run on the same 103-sample benchmark, but contrasting a loop with dynamic oracle feedback against a one-shot baseline inflates the apparent gap.\n\nTwo additional concerns are smaller but worth noting. The dynamic evaluation rests on 103 runnable samples from six projects, which is thin for the strong claims. And the prompts and 'experience' were tuned with the evaluation results in the loop, so the numbers may be partly overfit; a held-out validation set would help. No significance testing is reported.\n\nBottom line: this is a useful engineering contribution with a valuable dataset, and it deserves a serious referee. But the update accuracy claims need tighter evaluation before they can be believed. I would send it to review with a request for major revision, not desk-reject it.","headline":"Useful LLM-driven test co-evolution pipeline and a large public dataset, but the headline update accuracy is circular and needs an independent semantic check before it can be believed.","tokens_in":22255,"tokens_out":3603,"would_cite":true,"duration_ms":37684,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"REACCEPT claims an LLM feedback loop updates 60.16% of obsolete Java tests, doubling the prior best.","keywords":["product-test co-evolution","obsolete test identification","test code repair","large language models","dynamic validation","retrieval-augmented generation","ReAct mechanism","Java unit testing"],"falsifier":"Take a random sample of REACCEPT's successfully updated tests, mutate the changed production code so that the covered statements remain present but the behavior changes, for example, flip a comparison or return a wrong value, and rerun the updated tests: if most still pass, the tests are not validating the new behavior and the accuracy claim is an artifact of the validity predicate. A second check is to compare each updated test's assertions with the original test's assertions and count how many were deleted, weakened, or adjusted to match the new output.","tokens_in":21206,"feed_emoji":"🧪","tokens_out":6879,"duration_ms":68289,"temperature":0.7,"pith_summary":"The paper claims that the full production-test co-evolution loop, deciding which tests are obsolete after a code change and rewriting those tests, can be automated well enough for routine use. It proposes REACCEPT, an LLM-based pipeline that identifies obsolete Java tests, generates replacements, then runs each candidate through compile, execution, and coverage checks, feeding failures back into the model for up to eight rounds. On a benchmark built from 537 Java projects, REACCEPT updated 60.16% of correctly identified obsolete tests, roughly double the 31.62% of the previous best method. The intended payoff is a tool that keeps test suites synchronized with evolving code with limited human review.","feed_headline":"Auto-repair fixes 60% of outdated Java tests","feed_subtitle":"ReAccept feeds compile, test, and coverage errors back into the model, beating the prior best approach's 31.62% by 90%","key_machinery":"The central mechanism is the dynamic-validation feedback loop. REACCEPT decomposes test quality into four levels, compilation failure, test failure, coverage failure, and success, and uses the failure category to select a specialized prompt template; retrieval-augmented generation supplies a similar historical production/test diff pair as a few-shot example, and the ReAct-style agent interleaves reasoning with tool access. The identity carrying the argument is the valid-update predicate: a rewritten test counts as correct only if javac compiles it, JUnit runs it without failure, and JaCoCo shows it covers the changed statements in the production code.","core_discovery":"On its own terms, REACCEPT's central discovery is that dynamic feedback closes the gap that defeated earlier learning-based updaters. Earlier methods generated test code in one shot and scored it with textual similarity; REACCEPT instead treats update generation as an iterative repair loop. The LLM proposes an updated test, the Java compiler, the JUnit runner, and the JaCoCo coverage tool check it, the error, failure, or uncovered-statement messages are turned into new prompts, and the cycle repeats until the test compiles, passes, and covers the changed production statements or a cutoff is reached. With that loop, 71.84% of update tasks succeeded end-to-end, and the paper attributes the improvement to the dynamic-validation feedback, which reduced LLM hallucination and raised update effectiveness in ablation.","pith_inferences":["The reported accuracy may overstate semantic correctness, because a test that compiles, passes, and covers changed statements can still pass after an LLM weakens or deletes an assertion; a stronger metric would compare assertion strength or run mutation testing on the changed code.","The 60.16% figure is conditional on correct identification and was measured on 103 dynamically evaluable samples from six projects, so end-to-end usefulness in the wild depends on pairing the updater with an identifier at least as accurate as CEPROT's 97.5% identification accuracy.","Because 70% of updates succeed on the first generation, a cheaper two-stage design, one-shot LLM update followed by targeted repair only for failures, could capture most of the benefit at lower token cost.","The paper itself notes that retrieval failures on dddlib trace to similar edit structures with different identifiers, suggesting that replacing textual-similarity retrieval with structure-aware retrieval over AST changes is a testable extension."],"forward_implications":["REACCEPT's dynamic validation raises the share of updates that compile, pass, and cover changes to 71.84%, versus 35.92% for the best earlier method.","The identification phase is separate and can be swapped: the paper says CEPROT identifies obsolete tests slightly better, while REACCEPT wins overall through its update phase, so combining a stronger identifier with REACCEPT's updater should improve end-to-end accuracy.","Ablation shows both RAG and dynamic validation contribute; removing dynamic validation drops the coverage-based success rate to 52.43%, and removing RAG drops it to 50.49%.","Most successful updates, 70.27%, pass on the first iteration, and 22 of the remaining 29 are fixed through feedback, so the loop's value concentrates on hard cases.","The evaluation is on Java, but the paper states that the underlying pipeline, which relies on a compiler, a test runner, and a coverage tool, can transfer to other programming languages."],"supporting_citations":[{"why":"provides CEPROT, the strongest baseline and the dataset of obsolete test cases that ReAccept extends and must outperform.","marker":"[21]"},{"why":"provides the CHOSEN identification method and part of the co-evolution dataset, and grounds the claim that most change pairs are EDIT-EDIT.","marker":"[51]"},{"why":"provides SITAR and the historical co-evolution samples mined from repositories that form the knowledge base.","marker":"[53]"},{"why":"supplies the ReAct reasoning-and-acting pattern that lets the LLM interleave deduction with calls to dynamic validation.","marker":"[56]"},{"why":"supplies retrieval-augmented generation, used to fetch similar historical production/test diffs into the prompt.","marker":"[29]"},{"why":"defines CodeBLEU, the textual similarity metric used alongside dynamic metrics to compare generated test code with ground truth.","marker":"[45]"},{"why":"JaCoCo coverage messages drive the coverage-failure feedback branch of the update loop.","marker":"[3]"},{"why":"javac compilation errors drive the first feedback branch for broken test code.","marker":"[4]"},{"why":"JUnit execution failures drive the second feedback branch and define the pass requirement.","marker":"[5]"}],"fun_headline_variants":["LLM loop fixes 60% of outdated Java tests","Dynamic feedback boosts test repair to 60%","Test update accuracy jumps 90% with LLM feedback","REACCEPT: 60% fix rate for obsolete tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on treating compiles, passes, and covers the changed statements as a correct test update; if those three checks do not guarantee that the test meaningfully checks the new behavior, the reported 60 to 72 percent accuracy overstates how well REACCEPT preserves test intent.","fun_headline_variants_meta":{"raw":{"variants":["LLM loop fixes 60% of outdated Java tests","Dynamic feedback boosts test repair to 60%","Test update accuracy jumps 90% with LLM feedback","REACCEPT: 60% fix rate for obsolete tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000377,"raw_usage":{"total_tokens":2005,"prompt_tokens":941,"completion_tokens":1064,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":998}},"tokens_in":557,"tokens_out":1064,"duration_ms":11499,"temperature":1.0,"reasoning_tokens":998,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:00:33.632324+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of REACCEPT's successfully updated tests, mutate the changed production code so that the covered statements remain present but the behavior changes, for example, flip a comparison or return a wrong value, and rerun the updated tests: if most still pass, the tests are not validating the new behavior and the accuracy claim is an artifact of the validity predicate. A second check is to compare each updated test's assertions with the original test's assertions and count how many were deleted, weakened, or adjusted to match the new output.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides CEPROT, the strongest baseline and the dataset of obsolete test cases that ReAccept extends and must outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the CHOSEN identification method and part of the co-evolution dataset, and grounds the claim that most change pairs are EDIT-EDIT."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides SITAR and the historical co-evolution samples mined from repositories that form the knowledge base."},{"cited_title":"JaCoCo Java Code Coverage Library","cited_arxiv_id":null,"evidence_quote":"JaCoCo coverage messages drive the coverage-failure feedback branch of the update loop."},{"cited_title":"javac - Java programming language compiler","cited_arxiv_id":null,"evidence_quote":"javac compilation errors drive the first feedback branch for broken test code."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"JUnit execution failures drive the second feedback branch and define the pass requirement."}],"review_version":1}