{"id":"bf4bcf77-c918-4c10-94de-47f656b5f0d2","arxiv_id":"2608.00924","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An agentic repair pipeline with static fixes and test-guided LLM iterations raises the cumulative pass rate of LLM-generated Java refactorings to about 94%, though it does not verify that the intended refactoring was preserved.","lead":"This paper measures how often large language models break Java code when asked to refactor it, then builds an agent that repairs the broken code using test logs, code diffs, and error explanations. The agent raises the cumulative unit-test pass rate of generated refactorings from about 68% to roughly 94% in the best configuration.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Repaired outputs are never re-checked for refactoring preservation, so the 93.4%/94.2% pass rates may include reverts or unrelated rewrites that are not refactorings.","rationale":"The reader's weakest assumption is also the single most load-bearing concern for this paper. The empirical pipeline is detailed and the static/agentic distinction is well motivated, but the central claim is about reliable refactoring, not merely about passing tests. The paper itself uses RMiner to validate that the initial LLM output is a refactoring, which shows the authors know that structural change needs verification; the absence of the same check after repair is therefore a concrete, fixable gap rather than an unfalsifiable objection. A repair loop that is allowed to revert changes and is evaluated only by compile-and-test can trivially maximize pass rates by undoing the refactoring. The fact that the repair prompt encourages minimal edits and signature preservation makes reverting even more likely, since the cheapest way to satisfy both constraints is often to restore the original code. I also note the 93.4% versus 94.2% difference appears traceable to the RAG versus no-RAG configurations in Table 1, but the paper should state this reconciliation explicitly because Sections 4.3.3 and 4.4.3 present both numbers as headline results. I would not move the verdict to REJECT: the concern can be settled by a single additional validation step, and much of the paper's empirical evidence remains useful regardless. UNCHANGED reflects that the reader already assigned CONDITIONAL with exactly this condition.","tokens_in":23697,"tokens_out":5258,"duration_ms":47262,"concrete_test":"Run RefactoringMiner on the final repaired file versus the original for every successful case of the best configuration (Diff + ErrLog + Expl, GPT-4o). For each case, record whether RMiner detects at least one refactoring operation, whether the detected operation matches the one found on the initial LLM output, and how many final outputs are byte-identical to the original file. If a non-trivial fraction (e.g., >5%) of successful repairs are byte-identical reverts or are assigned no refactoring by RMiner, then the 93.4%/94.2% pass-rate claims do not measure refactoring preservation and the central claim should be scaled back.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RQ3 and RQ4 define success by Eq. (1): a repaired output is correct if it compiles and passes the paired unit test. RefactoringMiner is applied only to the initial LLM output (Section 4.1.2), not to the final repaired output. The repair prompt (Section 2.3.2) instructs the repair LLM to keep edits minimal, avoid top-level changes, and preserve public signatures; it does not prohibit reverting to the original code, and the static repair stage itself restores removed imports and missing delimiters. Because no specific refactoring goal is attached to each method (the Figure 3 prompt only asks for 'a refactored version'), there is no ground truth for what restructuring must survive. A repair that reverts a rename, deletes an extracted method, or otherwise undoes the initial refactoring is scored as a success as long as the unit tests pass. Therefore the headline repair rate (70.8% of remaining failures) and cumulative pass rate (94.2%) can substantially overstate refactoring reliability by counting non-refactorings as successful repairs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates how often LLM-generated Java refactorings pass unit tests, analyzes the causes of failures, and proposes RefactorAssist, a two-stage repair pipeline that first applies static fixes and then uses an iterative agentic loop with error logs, code diffs, retrieved context, and LLM-generated explanations. Using 10,000 method-test pairs from 10 open-source Java projects, the authors report that the best configuration raises the cumulative unit-test pass rate from 68.4% to 94.2%, repairing 70.8% of the failures that remain after static intervention. The paper also presents a failure taxonomy, a leakage-aware dataset construction, and an ablation study of the repair components. The central claim is that test-guided, context-aware agentic repair can make LLM-generated refactorings reliable enough for practical integration.","tokens_in":23927,"tokens_out":6302,"duration_ms":53333,"significance":"If the results hold, the paper provides a useful empirical contribution: a large-scale measurement of LLM refactoring failure rates, a manually validated failure taxonomy, and a reproducible repair pipeline with clear ablations. The use of StarCoder2 with Stack-v2 leakage filtering and a public replication package are strengths, as is the decomposition of repair effectiveness over iterations. However, the headline numbers are currently interpretable only as unit-test pass-rate improvements of LLM-modified code, not as evidence of refactoring preservation, because repaired outputs are never re-checked for whether a refactoring still exists. The significance of the paper for the refactoring community therefore depends on the authors adding a preservation check on the final repaired outputs and recalibrating the claims accordingly.","major_comments":[{"comment":"The paper never re-checks repaired outputs for refactoring preservation, so the reported success rates may include reverts or unrelated rewrites. This is load-bearing for the central claim that RefactorAssist improves the reliability of LLM-generated refactorings.","section":"Section 4.3.2, Eq. (1), Section 4.1.2, Section 2.3.2"},{"comment":"The task definition does not specify the intended refactoring for each target method. The prompt in Figure 3 only asks for 'a refactored version' of the code, and Section 3.1 states that the goal is to repair failures 'while preserving the intended refactoring behavior,' but no per-target refactoring goal is attached to each method. As a result, both the initial RMiner filter (which accepts any detected structural operation) and the repair evaluation (which accepts any compile-and-test-passing output) are agnostic to what specific transformation should survive. This means the evaluation cannot distinguish a successful repair that preserves the intended extraction, rename, or other operation from a behavior-preserving rewrite. Please either fix a concrete refactoring target per method before generation, or explicitly reframe the contribution as improving the unit-test pass rate of LLM-modified code rather than the reliability of refactoring preservation.","section":"Section 3.1 and Figure 3"}],"minor_comments":[{"comment":"The sentence 'Adding retrieval (\"+RAG\") yields modest gains in the first one to two iterations (e.g., 21.7% vs. 24.3% at iteration one without RAG)' is contradicted by Table 1, where the RAG configuration is lower than the non-RAG configuration at both iteration 1 (21.7 vs. 24.3) and iteration 2 (34.9 vs. 38.0). Please correct the sentence or the table.","section":"Section 4.4.3 and Table 1"},{"comment":"The text says a statistically representative sample of 350 failures is selected, but then states the final evaluation set consists of 300 failures because some repositories have fewer than 35 failures. This discrepancy should be clarified, since the sample-size justification appears to be based on 350 while the actual analyzed sample is 300.","section":"Section 4.2.2"},{"comment":"The ACM Reference Format line on the first page lists 2018 and a J. ACM article number that do not match the submission date or venue; this front matter should be updated.","section":"Front matter"},{"comment":"The phrase 'even models with larger parameter sizes and coding capabilities' is not directly supported by the reported data, since parameter counts for the evaluated models are not provided; consider rephrasing to refer to model capability or reporting the parameter sizes.","section":"Section 4.1.3"},{"comment":"The construct-validity threat discussion acknowledges that unit-test pass rates may not capture code quality, but it does not address the more direct threat that a passing repaired output may no longer be a refactoring; this should be added to the threats section.","section":"Section 5.3"}],"recommendation":"major_revision","confidential_remarks":"The empirical study is substantial and the replication package is a strength, but the missing post-repair refactoring-preservation check is the key validity issue. If the authors can rerun a refactoring detector on the final repaired outputs and show that the large majority of 'successful' repairs still contain a refactoring, the paper could be acceptable; if not, the claims should be reframed. The self-citations to the authors' prior work on LLM refactoring are used appropriately as background and do not introduce circularity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nHere's the short version: this is a genuinely useful empirical study of LLM refactoring failures, but the headline repair numbers are overstated because the evaluation never checks that a repaired output is still a refactoring.\n\nThe paper's real contributions are (1) a large-scale measurement of how often four LLMs break behavior when asked to refactor Java code (10k method-test pairs, 10 projects), with leakage control for StarCoder2; (2) a failure taxonomy derived by clustering plus manual validation (kappa 0.78); and (3) a two-stage repair pipeline — static fixes followed by an iterative agentic loop using diffs, error logs, and an explanation LLM. The ablation study is well designed and shows clearly that explanation guidance matters more than retrieval or any single signal. That is worth having in the literature.\n\nThe soft spot is load-bearing. Success is defined by Eq. (1) as compile plus unit-test pass. RefactoringMiner is applied only to the initial LLM output to filter non-refactorings; it is never re-run on repaired code. So a repair that reverts the refactoring, or introduces an unrelated rewrite that passes tests, is scored as a success. The prompt tells the repair model to keep edits minimal, but that is not verification. The paper even says the agent should preserve the intended refactoring behavior, yet never measures that. This makes the 70.8% repair rate and the 94.2% cumulative pass rate upper bounds on true refactoring-preserving repairs.\n\nThere's also a numeric inconsistency: RQ3 reports 93.4% after 10 iterations with the full pipeline, while RQ4's best configuration (diff+error log+explanation, no RAG) repairs 70.8% of the remaining failures, which computes to 94.2%. The full RQ3 pipeline included RAG, and RQ4's RAG variant repairs 67.3%, which computes to about 93.6%. The paper never reconciles these, and readers are left to guess which number belongs to which setup.\n\nBoth problems are fixable. Re-running RMiner on all repaired outputs — ideally checking that the originally detected refactoring operation is still present — would directly address the main threat. Re-reporting the RQ3 numbers with the exact configuration would clear up the confusion.\n\nMinors: the iteration limit of 10 is arbitrary, the static import-restoration step can itself undo LLM changes, and the Methods2Test mapping noise is inherited. None of these undermine the value.\n\nWho is this for? Anyone working on LLM-based refactoring tools, automated repair, or test-based correctness evaluation. I'd engage with it.\n\nRecommendation: send it to peer review. The taxonomy and ablation are valuable, and the preservation gap is a clearly addressable flaw rather than an unsound foundation. Accept with major revision.","headline":"Useful large-scale study of LLM refactoring failures and a two-stage repair agent, but the headline repair rates rest on a definition of success that never re-checks whether the repaired code is still a refactoring.","tokens_in":24458,"tokens_out":5080,"would_cite":true,"duration_ms":41432,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A repair agent lifts LLM refactorings' unit-test pass rate from 68.4% to 93.4%.","keywords":["code refactoring","large language models","agentic repair","unit test pass rate","static repair","failure taxonomy","Java","automated program repair"],"falsifier":"Re-run the paper's repair pipeline on the same failing refactorings, then run a refactoring-detection tool on each repaired diff and count how many repaired cases still contain the originally detected refactoring operation, such as the renamed method or extracted fragment. If a large fraction of the 70.8% repaired cases no longer contain that operation, the repair rate measures test-passing rather than refactoring reliability.","tokens_in":23492,"feed_emoji":"🔧","tokens_out":5626,"duration_ms":42671,"temperature":0.7,"pith_summary":"This paper tries to establish that the functional failures introduced by LLM-generated code refactorings are mostly recoverable, and that a cheap two-stage repair pipeline can recover them before human review. It reports that across 10 open-source Java projects, model-written refactorings pass their original unit tests only 68.4% of the time overall, and that adding a static repair stage raises that to 80.3%, with an iterative agentic loop then lifting it to 93.4%. The strongest configuration repairs 70.8% of the failures that remain after static intervention, for a cumulative pass rate of 94.2%. If correct, the work matters because it suggests that LLM refactoring tools should be built as generate-validate-repair workflows rather than single-pass generators.","feed_headline":"Repair agent lifts LLM refactorings' test pass rate to 93%","feed_subtitle":"Free static fixes first, then an agentic loop using error logs and diffs repairs most remaining failures.","key_machinery":"The load-bearing mechanism is the two-stage RefactorAssist pipeline: a zero-LLM static repair stage that restores missing imports, balances brackets, and patches simple structural mismatches, followed by an iterative agentic repair loop capped at ten iterations. Within the loop, four tools feed evidence to the repair model: an Error Log Processor that turns compiler and test output into structured signatures and locations, a Code Diff Generator that isolates hunks overlapping the failure, a Context Retriever that pulls relevant project symbols and definitions, and a Test Runner that recompiles and reruns the tests. The diagnosis step uses a separate LLM to convert this evidence into a {root_cause, hint} tuple, and the repair prompt constrains the code-generation model to minimal, localized edits. The iterate-diagnose-repair-recompile-rerun structure is what converts raw failure evidence into successive repair attempts, with most gains concentrated in the first three iterations.","core_discovery":"The paper's central claim is that most unit-test failures in LLM refactorings come from a small set of recurring, diagnosable problems, namely context misunderstanding or hallucination (24.3%), inconsistent renaming (15.3%), added functionality (13.7%), incomplete code (11.3%), syntax and structural errors (9.7%), unhandled edge cases (9%), improper type handling (8.7%), and out-of-scope variables (8%), and that these failures can be systematically repaired. RefactorAssist first applies conservative static fixes for missing imports, unbalanced brackets, and obvious type mismatches, then, for remaining failures, enters an iterative loop that feeds compiler and test logs, a code diff, retrieved project context, and an LLM-produced root-cause explanation and hint into a repair prompt. The paper reports that this pipeline raises the overall unit-test pass rate from 68.4% to 80.3% after static repair and to 93.4% after ten iterations, with the best configuration repairing 70.8% of the remaining failures and reaching a 94.2% cumulative pass rate. A key secondary claim is that the explanation component, not the retrieved context, drives the repair gains, since removing retrieval-augmented context slightly improves final recovery while removing the explanation sharply reduces it.","pith_inferences":["The reported 70.8% repair rate counts a refactoring as fixed if it compiles and passes the original tests; a natural next step is to re-run a refactoring detector on the repaired diff to check that the intended transformation actually survived the repair rather than being reverted.","The same generate-diagnose-repair loop could be evaluated with mutation testing or property-based tests instead of the original unit tests, which would reveal whether repairs merely satisfy the existing assertions or genuinely preserve behavior across a wider input space.","Because the failure taxonomy is dominated by context and naming errors at 39.6% combined, a testable extension is to feed the repair loop the full call graph or a symbol table rather than retrieved snippets, predicting larger gains on exactly those failure classes.","The result suggests a division of labor for multi-agent coding systems: a dedicated refactoring agent proposes structural changes while a separate test-guided agent verifies and repairs them, with the static pass acting as a cheap pre-filter."],"forward_implications":["LLM-based refactoring tools should expect roughly one in three single-shot refactorings to fail tests and should therefore ship with a validation-and-repair loop rather than trust one generation pass.","A no-LLM static pass that restores imports and fixes brackets is a cheap first line of defense, adding about 12 percentage points of pass rate before any agent runs.","The explanation LLM's quality directly controls repair effectiveness: swapping the diagnostic model from GPT-4o to Llama-3 8B drops the repair rate from 70.8% to 58.1%, so investing in diagnosis pays off more than adding retrieval context.","Retrieval-augmented context primarily accelerates early fixes and can slightly reduce final repair coverage, so it should be treated as an optional accelerator rather than a default component.","With 6.6% of failures still unresolved after ten iterations, a bounded repair budget is practical because most recoverable failures are fixed in the first three iterations."],"supporting_citations":[{"why":"Supplies StarCoder2, the open-source model used both to generate the initial refactorings and to generate repairs, and the Stack v2 training corpus used to filter data leakage.","marker":"[32]"},{"why":"Provides the Methods2Test dataset of Java methods mapped to unit tests, from which the 10 evaluation repositories and method-test pairs are drawn.","marker":"[57]"},{"why":"RefactoringMiner 2.0 is used to verify that a generated output is a genuine refactoring before its test outcome is counted.","marker":"[56]"},{"why":"GPT-4o serves as one of the initial refactoring models and as the strongest error-diagnostic explanation LLM in the ablation study.","marker":"[40]"},{"why":"Claude 3.5 Sonnet is evaluated as an alternative explanation and initial refactoring model, establishing the effect of diagnostic-model quality.","marker":"[2]"},{"why":"Qwen2.5-Coder-32B-Instruct is one of the four models compared in the first research question, providing the open-weight coding-model baseline.","marker":"[24]"},{"why":"JavaParser is the library used to extract production methods and test methods and to map tests to target methods.","marker":"[25]"},{"why":"The Mann-Whitney U test is applied to compare pass rates across method-level and class-level refactorings in the first research question.","marker":"[34]"}],"fun_headline_variants":["Agentic repair fixes 70.8% of remaining LLM refactor failures","RefactorAssist: static fixes then agentic loop lift pass rate to 94%","Why LLM refactorings fail: root causes and a repair agent","LLM refactor repair: context misunderstanding top failure at 24%","Agentic refinement boosts refactoring pass rate from 68% to 94%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire evaluation judges a repaired refactoring as successful if it compiles and passes the original unit tests; the paper never re-runs a refactoring detector on the repaired code, so a repair that quietly reverts or weakens the intended refactoring would still be counted as a success.","fun_headline_variants_meta":{"raw":{"variants":["Agentic repair fixes 70.8% of remaining LLM refactor failures","RefactorAssist: static fixes then agentic loop lift pass rate to 94%","Why LLM refactorings fail: root causes and a repair agent","LLM refactor repair: context misunderstanding top failure at 24%","Agentic refinement boosts refactoring pass rate from 68% to 94%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000626,"raw_usage":{"total_tokens":3008,"prompt_tokens":1166,"completion_tokens":1842,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":782,"completion_tokens_details":{"reasoning_tokens":1737}},"tokens_in":782,"tokens_out":1842,"duration_ms":12043,"temperature":1.0,"reasoning_tokens":1737,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:13:45.955844+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the paper's repair pipeline on the same failing refactorings, then run a refactoring-detection tool on each repaired diff and count how many repaired cases still contain the originally detected refactoring operation, such as the renamed method or extracted fragment. If a large fraction of the 70.8% repaired cases no longer contain that operation, the repair rate measures test-passing rather than refactoring reliability.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RefactoringMiner 2.0 is used to verify that a generated output is a genuine refactoring before its test outcome is counted."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Claude 3.5 Sonnet is evaluated as an alternative explanation and initial refactoring model, establishing the effect of diagnostic-model quality."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"JavaParser is the library used to extract production methods and test methods and to map tests to target methods."}],"review_version":1}