Pith. sign in

REVIEW 2 major objections 5 minor 63 references

RefactorAssist: Agentic Refinement for Reliable Code Refactoring

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A repair agent lifts LLM refactorings' unit-test pass rate from 68.4% to 93.4%.

desk verdict Useful large-scale study of LLM refactoring failures and a two-stage repair agent, but the headline repair rates rest on a definition of success that never re-checks whether the repaired code is still a refactoring. read the letter →

arxiv 2608.00924 v1 pith:AWIIIXLG submitted 2026-08-02 cs.SE cs.AI

Jonathan Cordeiro , Shayan Noei , Ying Zou This is my paper · ORCID
classification cs.SEcs.AI
keywords coderefactoringlargelanguagemodelsagenticrepairunittestpassratestaticfailuretaxonomyJavaautomatedprogram
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the functional failures introduced by LLM-generated code refactorings are mostly recoverable, and that a cheap two-stage repair pipeline can recover them before human review. It reports that across 10 open-source Java projects, model-written refactorings pass their original unit tests only 68.4% of the time overall, and that adding a static repair stage raises that to 80.3%, with an iterative agentic loop then lifting it to 93.4%. The strongest configuration repairs 70.8% of the failures that remain after static intervention, for a cumulative pass rate of 94.2%. If correct, the work matters because it suggests that LLM refactoring tools should be built as generate-validate-repair workflows rather than single-pass generators.

What carries the argument

The load-bearing mechanism is the two-stage RefactorAssist pipeline: a zero-LLM static repair stage that restores missing imports, balances brackets, and patches simple structural mismatches, followed by an iterative agentic repair loop capped at ten iterations. Within the loop, four tools feed evidence to the repair model: an Error Log Processor that turns compiler and test output into structured signatures and locations, a Code Diff Generator that isolates hunks overlapping the failure, a Context Retriever that pulls relevant project symbols and definitions, and a Test Runner that recompiles and reruns the tests. The diagnosis step uses a separate LLM to convert this evidence into a {root_cause, hint} tuple, and the repair prompt constrains the code-generation model to minimal, localized edits. The iterate-diagnose-repair-recompile-rerun structure is what converts raw failure evidence into successive repair attempts, with most gains concentrated in the first three iterations.

What would settle it

Re-run the paper's repair pipeline on the same failing refactorings, then run a refactoring-detection tool on each repaired diff and count how many repaired cases still contain the originally detected refactoring operation, such as the renamed method or extracted fragment. If a large fraction of the 70.8% repaired cases no longer contain that operation, the repair rate measures test-passing rather than refactoring reliability.

Watch

Extended reading notes

Core claim

The paper's central claim is that most unit-test failures in LLM refactorings come from a small set of recurring, diagnosable problems, namely context misunderstanding or hallucination (24.3%), inconsistent renaming (15.3%), added functionality (13.7%), incomplete code (11.3%), syntax and structural errors (9.7%), unhandled edge cases (9%), improper type handling (8.7%), and out-of-scope variables (8%), and that these failures can be systematically repaired. RefactorAssist first applies conservative static fixes for missing imports, unbalanced brackets, and obvious type mismatches, then, for remaining failures, enters an iterative loop that feeds compiler and test logs, a code diff, retrieved project context, and an LLM-produced root-cause explanation and hint into a repair prompt. The paper reports that this pipeline raises the overall unit-test pass rate from 68.4% to 80.3% after static repair and to 93.4% after ten iterations, with the best configuration repairing 70.8% of the remaining failures and reaching a 94.2% cumulative pass rate. A key secondary claim is that the explanation component, not the retrieved context, drives the repair gains, since removing retrieval-augmented context slightly improves final recovery while removing the explanation sharply reduces it.

Load-bearing premise

The entire evaluation judges a repaired refactoring as successful if it compiles and passes the original unit tests; the paper never re-runs a refactoring detector on the repaired code, so a repair that quietly reverts or weakens the intended refactoring would still be counted as a success.

Editorial extensions

If this is right

  • LLM-based refactoring tools should expect roughly one in three single-shot refactorings to fail tests and should therefore ship with a validation-and-repair loop rather than trust one generation pass.
  • A no-LLM static pass that restores imports and fixes brackets is a cheap first line of defense, adding about 12 percentage points of pass rate before any agent runs.
  • The explanation LLM's quality directly controls repair effectiveness: swapping the diagnostic model from GPT-4o to Llama-3 8B drops the repair rate from 70.8% to 58.1%, so investing in diagnosis pays off more than adding retrieval context.
  • Retrieval-augmented context primarily accelerates early fixes and can slightly reduce final repair coverage, so it should be treated as an optional accelerator rather than a default component.
  • With 6.6% of failures still unresolved after ten iterations, a bounded repair budget is practical because most recoverable failures are fixed in the first three iterations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported 70.8% repair rate counts a refactoring as fixed if it compiles and passes the original tests; a natural next step is to re-run a refactoring detector on the repaired diff to check that the intended transformation actually survived the repair rather than being reverted.
  • The same generate-diagnose-repair loop could be evaluated with mutation testing or property-based tests instead of the original unit tests, which would reveal whether repairs merely satisfy the existing assertions or genuinely preserve behavior across a wider input space.
  • Because the failure taxonomy is dominated by context and naming errors at 39.6% combined, a testable extension is to feed the repair loop the full call graph or a symbol table rather than retrieved snippets, predicting larger gains on exactly those failure classes.
  • The result suggests a division of labor for multi-agent coding systems: a dedicated refactoring agent proposes structural changes while a separate test-guided agent verifies and repairs them, with the static pass acting as a cheap pre-filter.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper evaluates how often LLM-generated Java refactorings pass unit tests, analyzes the causes of failures, and proposes RefactorAssist, a two-stage repair pipeline that first applies static fixes and then uses an iterative agentic loop with error logs, code diffs, retrieved context, and LLM-generated explanations. Using 10,000 method-test pairs from 10 open-source Java projects, the authors report that the best configuration raises the cumulative unit-test pass rate from 68.4% to 94.2%, repairing 70.8% of the failures that remain after static intervention. The paper also presents a failure taxonomy, a leakage-aware dataset construction, and an ablation study of the repair components. The central claim is that test-guided, context-aware agentic repair can make LLM-generated refactorings reliable enough for practical integration.

Significance. If the results hold, the paper provides a useful empirical contribution: a large-scale measurement of LLM refactoring failure rates, a manually validated failure taxonomy, and a reproducible repair pipeline with clear ablations. The use of StarCoder2 with Stack-v2 leakage filtering and a public replication package are strengths, as is the decomposition of repair effectiveness over iterations. However, the headline numbers are currently interpretable only as unit-test pass-rate improvements of LLM-modified code, not as evidence of refactoring preservation, because repaired outputs are never re-checked for whether a refactoring still exists. The significance of the paper for the refactoring community therefore depends on the authors adding a preservation check on the final repaired outputs and recalibrating the claims accordingly.

major comments (2)
  1. [Section 4.3.2, Eq. (1), Section 4.1.2, Section 2.3.2] The paper never re-checks repaired outputs for refactoring preservation, so the reported success rates may include reverts or unrelated rewrites. This is load-bearing for the central claim that RefactorAssist improves the reliability of LLM-generated refactorings.
  2. [Section 3.1 and Figure 3] The task definition does not specify the intended refactoring for each target method. The prompt in Figure 3 only asks for 'a refactored version' of the code, and Section 3.1 states that the goal is to repair failures 'while preserving the intended refactoring behavior,' but no per-target refactoring goal is attached to each method. As a result, both the initial RMiner filter (which accepts any detected structural operation) and the repair evaluation (which accepts any compile-and-test-passing output) are agnostic to what specific transformation should survive. This means the evaluation cannot distinguish a successful repair that preserves the intended extraction, rename, or other operation from a behavior-preserving rewrite. Please either fix a concrete refactoring target per method before generation, or explicitly reframe the contribution as improving the unit-test pass rate of LLM-modified code rather than the reliability of refactoring preservation.
minor comments (5)
  1. [Section 4.4.3 and Table 1] The sentence 'Adding retrieval ("+RAG") yields modest gains in the first one to two iterations (e.g., 21.7% vs. 24.3% at iteration one without RAG)' is contradicted by Table 1, where the RAG configuration is lower than the non-RAG configuration at both iteration 1 (21.7 vs. 24.3) and iteration 2 (34.9 vs. 38.0). Please correct the sentence or the table.
  2. [Section 4.2.2] The text says a statistically representative sample of 350 failures is selected, but then states the final evaluation set consists of 300 failures because some repositories have fewer than 35 failures. This discrepancy should be clarified, since the sample-size justification appears to be based on 350 while the actual analyzed sample is 300.
  3. [Front matter] The ACM Reference Format line on the first page lists 2018 and a J. ACM article number that do not match the submission date or venue; this front matter should be updated.
  4. [Section 4.1.3] The phrase 'even models with larger parameter sizes and coding capabilities' is not directly supported by the reported data, since parameter counts for the evaluated models are not provided; consider rephrasing to refer to model capability or reporting the parameter sizes.
  5. [Section 5.3] The construct-validity threat discussion acknowledges that unit-test pass rates may not capture code quality, but it does not address the more direct threat that a passing repaired output may no longer be a refactoring; this should be added to the threats section.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the central claims are direct pass-rate measurements; the only flagged issue is a construct-validity gap about unverified refactoring preservation, not circularity.

full rationale

RefactorAssist's central quantitative claims (68.4% to 80.3% to 93.4% pass rate; 70.8% remaining-failure repair rate; 94.2% cumulative pass rate) are direct measurements defined by Eq. (1): a refactoring is considered correct if it compiles and its associated unit tests pass. No fitted parameter is renamed as a prediction, no self-citation supplies the repair result, and no uniqueness theorem or ansatz is imported from the authors' prior work. The authors' self-citations ([10], [11]) appear only as background motivation, while RQ1 independently measures LLM refactoring pass rates and RQ3/RQ4 evaluate the pipeline's behavior on the same held-out test suites. The one caveat is a construct-validity gap, not circularity: RefactoringMiner validation is applied only to the initial LLM output (Section 4.1.2), while repaired outputs are scored solely by compilation and unit-test passage (Eq. (1), Section 4.3.2), and the repair prompt's preservation constraints (Section 2.3.2) are not verified on the final code. Thus the 70.8% and 94.2% figures could count reverts or unrelated rewrites as successful repairs, which would overstate refactoring reliability; however, this does not make the pass-rate measurement a derivation from its own inputs. The paper is therefore self-contained against an external benchmark (Methods2Test), and its reported pass rates are honestly operationalized.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central result depends on the choice of 10 Maven projects with rich tests, the mapping of unit tests to target methods, the use of RMiner to decide which LLM outputs count as refactorings, and the iteration cap of 10. None of these are fitted constants in a derivation; they are evaluation design choices that bound the claims.

free parameters (4)
  • Agentic repair iteration limit = 10
    Chosen by the authors to balance repair opportunity and computational cost; the final repair rates are measured under this cap, so the headline numbers depend on it.
  • Minimum runnable tests per repository = 1,000
    Project inclusion threshold that shaped the 10-project dataset; affects generalizability to less-tested codebases.
  • Manual failure sample size = 350 targeted, 300 final
    Stratified sample used for the RQ2 taxonomy; the percentages in Figure 7 are based on 300 cases.
  • Cluster count for failure taxonomy = 8
    Selected by silhouette score (0.34) among candidate counts 2 to 12; affects the reported taxonomy but is partially validated by the manual kappa of 0.78.
assumptions (4)
  • domain assumption Unit-test pass rate is a sufficient operationalization of functional correctness for the evaluation.
    Eq. (1) defines pass rate as compile plus all associated tests passing; Section 5.3 acknowledges that tests do not capture all quality aspects.
  • domain assumption Repository-name overlap with Stack v2 is sufficient to prevent data leakage for StarCoder2.
    Section 3.2.1 excludes repositories whose names appear in the training corpus; this assumes name matching catches all training exposure.
  • domain assumption RefactoringMiner (RMiner) detection correctly identifies whether an LLM output is a refactoring.
    Section 4.1.2 uses RMiner to filter outputs that contain at least one detected refactoring operation; this is a tool-based proxy for refactoring validity.
  • domain assumption Methods2Test method-test mappings are correct after executable validation.
    Section 3.2.2 and Section 5.3 rely on compiling and running mapped tests to validate the method-test pairs before evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RefactorAssist: Agentic Refinement for Reliable Code Refactoring." pith.science (2026). https://pith.science/paper/AWIIIXLG

@misc{pith2026260800924,
  author       = {Pith},
  title        = {Pith review of: RefactorAssist: Agentic Refinement for Reliable Code Refactoring},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AWIIIXLG}},
  note         = {Machine review of arXiv:2608.00924}
}
read the original abstract

Code refactoring aims to enhance the internal structure of source code without affecting its functional behavior. The recent advancements of Large Language Models (LLMs) have demonstrated potential for automating software engineering tasks, such as code refactoring. However, the refactorings produced by LLMs often introduce subtle errors, leading to functional behavior changes and failed unit tests, which limit their practical adoption. To address the limitations of LLM-generated refactorings, we analyze the root causes of their failures and develop the RefactorAssist agent to improve the functional correctness of LLM-generated refactorings. To this end, we use 10 open-source Java projects with their native test suites and manually evaluate why LLM-generated refactorings fail unit tests. We then design an agentic approach that leverages unit-test logs, error explanations, project context retrieval, and code diffs to guide the iterative refactoring. Our findings show that the main reasons for failure are context misunderstanding/hallucination (24.3%), incorrect or inconsistent renaming (15.3%), adding new functionality or variables (13.7%), code incompleteness (11.3%), syntax and structural errors (9.7%), edge cases not handled (9%), improper type handling (8.7%), and variables outside scope (8%). To make our approach cost-effective, RefactorAssist first applies a static repair step for missing imports, unbalanced brackets, and compilation errors without LLMs. For remaining failures, RefactorAssist incorporates error logs and code diffs, achieving up to a 70.8% repair rate on the remaining failures and a 94.2% cumulative pass rate under the best-performing configuration. These results indicate that static checks and test-guided, context-aware agentic repair can increase the reliability of LLM-generated refactorings, bringing them closer to practical integration within developer workflows.

Figures

Figures reproduced from arXiv: 2608.00924 by the authors.

Figure 1
Figure 1. Overview of RefactorAssist. Failing LLM-generated refactorings first undergo static repair and, if still failing, enter an iterative [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Overview of the research approach. project to measure its functional correctness. When a refactored method fails the unit tests, we analyze the resulting test failures to understand common failure patterns and identify opportunities for improvement. Leveraging identified failure patterns, we implement a repair pipeline to rectify and refine the initial refactoring attempts. Therefore, we employ an agentic approach t… view at source ↗
Figure 3
Figure 3. Zero-shot instruction used to prompt each evaluated LLM for refactoring. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Distribution of unit-test pass rates across evaluated LLMs. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Distribution of unit-test pass rates across prompting settings. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Distribution of unit-test pass rates for method-level and class-level refactorings. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Breakdown of the reasons for unit test failures from manual analysis. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Project-level unit test pass rates for original and static-fixed refactorings. Boxes show the distribution across projects. [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Overall unit test pass rate across the full refactoring repair pipeline. The first point shows the base pass rate of the original [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Cumulative percentage of failures remaining after the static-fix stage repaired by iteration [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 22 canonical work pages

  1. [1]

    Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2025. Can LLMs Replace Manual Annotation of Software Engineering Artifacts?. In2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). 526–538. doi:10.1109/MSR66628.2025.00086

    arXiv 2025

  2. [2]

    Anthropic. 2024. Claude 3.5 Sonnet. Model release. https://www.anthropic.com/news/claude-3-5-sonnet Released June 20, 2024; available via Claude.ai, iOS app, Anthropic API, Amazon Bedrock, Google Vertex AI

    work page 2024

  3. [3]

    Vaibhav Anu, Wenhua Hu, Jeffrey C Carver, Gursimran S Walia, and Gary Bradshaw. 2018. Development of a human error taxonomy for software requirements: A systematic literature review.Information and Software Technology103 (2018), 112–124. doi:10.1016/j.infsof.2018.06.011

    work page doi:10.1016/j.infsof.2018.06.011 2018

  4. [4]

    Bittman, R

    M. Bittman, R. Roos, and G.M. Kapfhammer. 2001. Creating a Free, Dependable Software Engineering Environment for Building Java Applications. In1st Workshop on Open Source Software Engineering at ICSE 2001

    work page 2001

  5. [5]

    Jason Brownlee. 2020. Data leakage in machine learning. https://machinelearningmastery.com/data-leakage-machine-learning/

    work page 2020

  6. [6]

    Yupeng Chang et al. 2024. A Survey on Evaluation of Large Language Models.ACM Trans. Intell. Syst. Technol.15, 3, Article 39 (March 2024), 45 pages. doi:10.1145/3641289

    doi:10.1145/3641289 2024

  7. [7]

    Mark Chen et al. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG] https://arxiv.org/abs/2107.03374

    arXiv 2021

  8. [8]

    Jacob Cohen. 1960. A coefficient of agreement for nominal scales.Educational and psychological measurement20, 1 (1960), 37–46

    1960

Show all 63 references
  1. [9]

    1988.Statistical Power Analysis for the Behavioral Sciences(2 ed.)

    Jacob Cohen. 1988.Statistical Power Analysis for the Behavioral Sciences(2 ed.). Routledge. doi:10.4324/9780203771587

    1988 doi

  2. [10]

    Jonathan Cordeiro, Shayan Noei, and Ying Zou. 2025. LLM-Driven Code Refactoring: Opportunities and Limitations. In2025 IEEE/ACM Second IDE Workshop (IDE). 32–36. doi:10.1109/IDE66625.2025.00011

    2025

  3. [11]

    Jonathan Cordeiro, Shayan Noei, and Ying Zou. 2026. An Empirical Study on the Code Refactoring Capability of Large Language Models.ACM Trans. Softw. Eng. Methodol.(March 2026). doi:10.1145/3801158 Just Accepted

    2026 doi

  4. [12]

    Alexis Dinno. 2015. Nonparametric pairwise multiple comparisons in independent groups using Dunn’s test.The Stata Journal15, 1 (2015), 292–300

    2015

  5. [13]

    Jelihovschi, Jose Claudio Faria, and Ivan Bezerra Allaman

    Enio G. Jelihovschi, Jose Claudio Faria, and Ivan Bezerra Allaman. 2014. ScottKnott: A Package for Performing the Scott-Knott Clustering Algorithm in R.Trends in Applied and Computational Mathematics15, 1 (2014), 3–17. https://tema.sbmac.org.br/tema/article/view/646/643

    2014

  6. [14]

    Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023. Large Language Models for Software Engineering: Survey and Open Problems. In2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineeri...

    2023

  7. [15]

    Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. 2023. Automated Repair of Programs from Large Language Models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 1469–1481. doi:10.1109/ICSE48619.2023.00128

    2023

  8. [16]

    2018.Refactoring: improving the design of existing code

    Martin Fowler. 2018.Refactoring: improving the design of existing code. Addison-Wesley Professional

    2018

  9. [17]

    Fowler, K

    M. Fowler, K. Beck, J. Brant, W. Opdyke, and D. Roberts. 1999.Refactoring: Improving the Design of Existing Code. Addison-Wesley Professional, Berkeley, CA, USA

    1999

  10. [18]

    Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. InProceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering(Szeged, Hungary)(ESEC/FSE ’11). Association fo...

    2011

  11. [19]

    Aaron Grattafiori et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783

    2024 arXiv

  12. [20]

    Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017. Deepfix: Fixing common c language errors by deep learning. InProceedings of the aaai conference on artificial intelligence, Vol. 31. Manuscript submitted to ACM 26 Cordeiro et al

    2017

  13. [21]

    Refactoring Guru. 2024. https://refactoring.guru/refactoring/techniques

    2024

  14. [22]

    John A Hartigan and Manchek A Wong. 1979. Algorithm AS 136: A k-means clustering algorithm.Journal of the royal statistical society. series c (applied statistics)28, 1 (1979), 100–108

    1979

  15. [23]

    Lei Huang et al. 2025. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions.ACM Trans. Inf. Syst.43, 2, Article 42 (Jan. 2025), 55 pages. doi:10.1145/3703155

    2025 doi

  16. [24]

    Binyuan Hui et al. 2024. Qwen2.5-Coder Technical Report. arXiv:2409.12186 [cs.CL] https://arxiv.org/abs/2409.12186

    2024 arXiv

  17. [25]

    JavaParser Community. 2024. JavaParser. https://javaparser.org/ Open-source library for parsing and analyzing Java source code

    2024

  18. [26]

    Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why don’t software developers use static analysis tools to find bugs?. In2013 35th International Conference on Software Engineering (ICSE). IEEE, 672–681

    2013

  19. [27]

    JUnit Team. 2024. JUnit 5. https://junit.org/junit5/ Java testing framework for unit and integration testing

    2024

  20. [28]

    Sayash Kapoor and Arvind Narayanan. 2023. Leakage and the reproducibility crisis in machine-learning-based science.Patterns4, 9 (2023)

    2023

  21. [29]

    Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. 2011. Genprog: A generic method for automatic software repair.Ieee transactions on software engineering38, 1 (2011), 54–72

    2011

  22. [30]

    Valentina Lenarduzzi, Fabiano Pecorelli, Nyyti Saarimaki, Savanna Lujan, and Fabio Palomba. 2023. A critical comparison on six static analysis tools: Detection, agreement, and precision.Journal of Systems and Software198 (2023), 111575. doi:10.1016/j.jss.2022.111575

    2023

  23. [31]

    Bo Liu, Yanjie Jiang, Yuxia Zhang, Nan Niu, Guangjie Li, and Hui Liu. 2024. An Empirical Study on the Potential of LLMs in Automated Software Refactoring. arXiv:2411.04444 [cs.SE] https://arxiv.org/abs/2411.04444

    2024 arXiv

  24. [32]

    Anton Lozhkov et al. 2024. StarCoder 2 and The Stack v2: The Next Generation. arXiv:2402.19173 [cs.SE] https://arxiv.org/abs/2402.19173

    2024 arXiv

  25. [33]

    McKight and Julius Najab

    Patrick E. McKight and Julius Najab. 2010.Kruskal-Wallis Test. John Wiley & Sons, Ltd, 1–1. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/9780470479216.corpsy0491 doi:10.1002/9780470479216.corpsy0491

    2010 doi

  26. [34]

    McKnight and Julius Najab

    Patrick E. McKnight and Julius Najab. 2010.Mann-Whitney U Test. John Wiley & Sons, Ltd, 1–1. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/9780470479216.corpsy0524 doi:10.1002/9780470479216.corpsy0524

    2010 doi

  27. [35]

    Sergey Mechtaev, Jooyong Yi, and Abhik Roychoudhury. 2016. Angelix: scalable multiline program patch synthesis via symbolic analysis. In Proceedings of the 38th International Conference on Software Engineering(Austin, Texas)(ICSE ’16). Association for Computing Machinery, New ...

    2016

  28. [36]

    Miller, Agnes F

    Frederic P. Miller, Agnes F. Vandome, and John McBrewster. 2010. Apache Maven. https://api.semanticscholar.org/CorpusID:86808749

    2010

  29. [37]

    Emerson Murphy-Hill and Andrew P Black. 2008. Refactoring tools: Fitness for purpose.IEEE software25, 5 (2008), 38–44

    2008

  30. [38]

    Shayan Noei, Heng Li, Stefanos Georgiou, and Ying Zou. 2023. An Empirical Study of Refactoring Rhythms and Tactics in the Software Development Process.IEEE Transactions on Software Engineering49, 12 (2023), 5103–5119. doi:10.1109/TSE.2023.3326775

    2023

  31. [39]

    Shayan Noei, Heng Li, and Ying Zou. 2025. An Empirical Study on Release-Wise Refactoring Patterns.Proc. ACM Softw. Eng.2, FSE, Article FSE019 (June 2025), 22 pages. doi:10.1145/3715734

    2025 doi

  32. [40]

    OpenAI, :, Aaron Hurst, et al. 2024. GPT-4o System Card. arXiv:2410.21276 [cs.CL] https://arxiv.org/abs/2410.21276

    2024 arXiv

  33. [41]

    Mohamed Ouf, Shayan Noei, Zeph Van Iterson, Mariam Guizani, and Ying Zou. 2026. An Empirical Analysis of Community and Coding Patterns in OSS4SG vs. Conventional OSS.arXiv preprint arXiv:2601.03430(2026)

    2026

  34. [42]

    Sheena Panthaplackel, Pengyu Nie, Milos Gligoric, Junyi Jessy Li, and Raymond Mooney. 2020. Learning to Update Natural Language Comments Based on Code Changes. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Dan Jurafsky, Joyce Chai, ...

    2020 doi

  35. [43]

    Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Chapter Six - Mutation Testing Advances: An Analysis and Survey. Advances in Computers, Vol. 112. Elsevier, 275–378. doi:10.1016/bs.adcom.2018.03.015

    2019 doi

  36. [44]

    Sanyogita Piya and Allison Sullivan. 2024. LLM4TDD: Best Practices for Test Driven Development Using Large Language Models. InProceedings of the 1st International Workshop on Large Language Models for Code(Lisbon, Portugal)(LLM4Code ’24). Association for Computing Machinery, N...

    2024

  37. [45]

    PMD Open Source Contributors. 2024. PMD. https://pmd.github.io Software for Static Code Analysis

    2024

  38. [46]

    Dorin Pomian, Abhiram Bellur, Malinda Dilhara, Zarina Kurbatova, Egor Bogomolov, Andrey Sokolov, Timofey Bryksin, and Danny Dig. 2024. EM-Assist: Safe Automated ExtractMethod Refactoring with LLMs. InCompanion Proceedings of the 32nd ACM International Conference on the Foundat...

    2024

  39. [47]

    Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis.Journal of computational and applied mathematics20 (1987), 53–65

    1987

  40. [48]

    Yunfan Shao, Linyang Li, Zhaoye Fei, Hang Yan, Dahua Lin, and Xipeng Qiu. 2024. Balanced Data Sampling for Language Model Training with Clustering. InFindings of the Association for Computational Linguistics: ACL 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Asso...

    2024 doi

  41. [49]

    Sharma, P

    T. Sharma, P. Mishra, and R. Tiwari. 2016. Designite: A software design quality assessment tool. InProceedings of the 1st International Workshop on Bringing Architectural Design Thinking into Developers’ Daily Activities. 1–4

    2016

  42. [50]

    Atsushi Shirafuji, Yusuke Oda, Jun Suzuki, Makoto Morishita, and Yutaka Watanobe. 2023. Refactoring Programs Using Large Language Models with Few-Shot Examples. In2023 30th Asia-Pacific Software Engineering Conference (APSEC). 151–160. doi:10.1109/APSEC60848.2023.00025 Manuscr...

    2023

  43. [51]

    SonarSource S.A. 2024. SonarQube. https://www.sonarsource.com/products/sonarqube Software for Static Code Analysis

    2024

  44. [52]

    SpotBugs Team. 2024. SpotBugs. https://spotbugs.github.io Software for Static Code Analysis

    2024

  45. [53]

    2009.EMF: Eclipse Modeling Framework 2.0(2nd ed.)

    David Steinberg, Frank Budinsky, Marcelo Paternostro, and Ed Merks. 2009.EMF: Eclipse Modeling Framework 2.0(2nd ed.). Addison-Wesley Professional

    2009

  46. [54]

    Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. IntelliCode compose: code generation using transformer. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering ...

    2020

  47. [55]

    Wei Tao, Yanlin Wang, Ensheng Shi, Lun Du, Shi Han, Hongyu Zhang, Dongmei Zhang, and Wenqiang Zhang. 2021. On the Evaluation of Commit Message Generation Models: An Experimental Study. In2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). 126–136....

    2021

  48. [56]

    Nikolaos Tsantalis, Ameya Ketkar, and Danny Dig. 2020. RefactoringMiner 2.0.IEEE Transactions on Software Engineering48, 3 (2020), 930–950

    2020

  49. [57]

    Michele Tufano, Shao Kun Deng, Neel Sundaresan, and Alexey Svyatkovskiy. 2022. Methods2Test: a dataset of focal methods mapped to test cases. InProceedings of the 19th International Conference on Mining Software Repositories(Pittsburgh, Pennsylvania)(MSR ’22). Association for ...

    2022

  50. [58]

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compres- sion of Pre-Trained Transformers. InAdvances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal...

    2020

  51. [59]

    Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang. 2023. Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations ...

    2023

  52. [60]

    Chunqiu Steven Xia and Lingming Zhang. 2024. Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA ’24). ACM, 819–831. doi:10.114...

    2024

  53. [61]

    Cong Xu, Gayathri Saranathan, Mahammad Parwez Alam, Arpit Shah, James Lim, Soon Yee Wong, Foltin Martin, and Suparna Bhattacharya

  54. [62]

    Chunli Yu, Shayan Noei, Haoxiang Zhang, and Ying Zou. 2026. An Empirical Study on the Characteristics of Reusable Code Clones.ACM Trans. Softw. Eng. Methodol.(Jan. 2026). doi:10.1145/3793251 Just Accepted. Manuscript submitted to ACM

    2026 doi

  55. [2024]

    arXiv:2406.15527 [cs.LG] https: //arxiv.org/abs/2406.15527

    Data Efficient Evaluation of Large Language Models and Text-to-Image Models via Adaptive Sampling. arXiv:2406.15527 [cs.LG] https: //arxiv.org/abs/2406.15527

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.