REVIEW 3 major objections 5 minor 42 references
ActRef: Enhancing the Understanding of Python Code Refactoring with Action-Based Analysis
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read ActRef detects Python refactorings by tracking fine-grained code-change actions, with reported precision 0.80 and recall 0.92 on 1,914 validated instances.
desk verdict ActRef's action-based design is a genuine advance for Python refactoring mining, but the evaluation's self-constructed oracle makes the reported precision and recall unreliable until independently re-annotated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the action set: atomic AST-level operations insert, delete, move, and update, plus module-level slice operations computed by program slicing—isolating the statements involved in a computation—and code similarity. The AST-level calculator extends an established AST differencing engine, and ActRef's contribution is a context-aware post-processing layer that checks signatures, usage context, and whether original nodes were fully or partially deleted before accepting a move or update. The module-level calculator pairs files and slices that AST-level matching cannot handle, which is what lets the framework span variables, methods, classes, and modules. The rule table encodes each refactoring type as a structural pattern over these actions, making the detector modular and extensible.
What would settle it
Re-annotate a random sample of the 500 commits with independent judges who have not seen ActRef's rule table, then recompute precision and recall; if the resulting overall $F_1$ falls well below the reported 0.85, the performance claim is not robust.
Extended reading notes
Core claim
The paper's central claim is that a Python refactoring detector will be more accurate and broader in coverage if it treats code changes as tree-edit actions rather than as statement-level matches. ActRef first pairs files by name, path, and similarity, detecting coarse-grained module moves, renames, extracts, and inlines through program slicing and code similarity. Within paired files, it then computes AST-level insert, delete, move, and update actions using an extended AST differencing engine, applies context- and signature-aware post-processing to correct misalignments, and passes unmatched actions to a cross-file stage. The resulting action patterns are matched against a rule table encoding 15 refactoring types. In the authors' evaluation, this yields an overall $F_1$ of 0.85, and the paper reports that it outperforms all four baselines on every supported refactoring type while also detecting module- and variable-level refactorings the rule-based baseline does not support.
Load-bearing premise
The load-bearing premise is that the manually extended oracle reflects true refactorings, even though the paper counts ActRef–baseline consensus as true without separate review and validates with the same rule table that defines ActRef's detections.
Editorial extensions
If this is right
- Adding a new refactoring type becomes a matter of adding an action-pattern rule, so the detector can grow without replacing the matching engine.
- Variable-level and module-level coverage allows refactoring activity to be tracked at a finer granularity than method- and class-level tools provide.
- The runtime result indicates that broader coverage does not necessarily cost extra analysis time, supporting use on large repositories.
- High recall (0.92) on the evaluated commit set means action analysis can recover refactorings that are entangled with other edits, a case statement matching tends to miss.
- Across all supported types, the reported F1 exceeds the rule-based and LLM baselines, making structured action analysis the stronger current option for Python refactoring mining.
Reading between the lines
- If the action abstraction is language-neutral, the same two-stage pipeline could be applied to other dynamically typed languages, an extension the paper leaves for future work.
- Action sequences could serve as supervision for a compact, fast classifier that labels refactoring types without invoking a large language model.
- A cautious reading of the evaluation: because the oracle was partly built from consensus between ActRef and the rule-based baseline and validated with ActRef's rule table, an independent re-annotation study would be the cleanest check on the reported numbers.
- Refactoring-aware downstream tools—merge tools, regression-test selectors, and code review assistants—could consume action streams rather than raw diffs if this representation becomes standard.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. ActRef proposes an action-based framework for detecting refactorings in Python. Instead of matching whole statements, it computes AST-level edit actions (insert, delete, move, update) using an extended GumTree and module-level actions using program slicing and similarity, then matches these action patterns against rules for 15 refactoring types at variable, method, class, and module granularity. The authors evaluate on a dataset of 1,914 instances from 500 commits in 136 open-source Python projects, comparing with PyRef, PyRef combined with MLRefScanner, DeepSeek-R1, and ChatGPT-4. They report precision 0.80, recall 0.92, and F1 0.85, concluding that ActRef consistently outperforms the baselines while matching PyRef in runtime. A replication package is provided.
Significance. The action-based representation and the staged module-level/AST-level pipeline are potentially valuable: they target a genuine limitation of statement-matching tools in dynamic Python code, and the inclusion of LLM baselines makes the comparison timely. A replication package is provided, and the discussion of GumTree matcher choices in Section 6.2 is candid. If the reported numbers survive independent validation, ActRef would be a useful, scalable addition to Python refactoring-mining tooling, and the extended dataset could be a community resource. However, the evaluation oracle is not independent of ActRef, so the quantitative claims are not yet established.
major comments (3)
- [Section 4.3 and Table 1; Eqs. (1)-(3)] The ground-truth oracle is partially self-constructed. The text states that 'refactoring instances that were consistently detected by both ActRef and PyRef were also considered true positives without further review' and that manual validation 'followed a predefined set of rigorous rules (shown in Table 1),' where Table 1 is the same rule table used by Algorithms 3-4 to generate ActRef's detections. This creates a circularity: shared ActRef-PyRef detections enter the TP set without independent confirmation, and the validator checks ActRef's candidates against ActRef's own patterns. If ActRef and PyRef share systematic false positives, those errors are silently counted as correct, inflating precision; if the oracle is extended with ActRef's detections, ActRef's misses are underrepresented, inflating recall. The headline P/R/F1 values in Table 2 therefore do not measure detection against independent reality. The authors should re-derive the oracle with independent manual annotation, blind to tool outputs, using an a priori taxonomy that is not ActRef's rule table, and report inter-annotator agreement. Section 6.3 discusses dataset bias toward simpler refactorings but does not address this self-constructed oracle.
- [Sections 5.1.1 and 5.1.2; Table 2] The baseline comparisons are asymmetric. For PyRef, the authors report that it detected 26 instances that should be classified as Extract Class, Move Class, or other module-level refactoring and then 'decided to omit these instances from PyRef's result.' Omitting these detections changes PyRef's TP and FP counts relative to ActRef, and Table 2 does not disclose how these 26 instances are counted in the oracle or in ActRef's totals. For the LLM baselines, 21 commits were excluded for DeepSeek-R1 and 5 for ChatGPT-4 because of token limits, so the aggregate P/R/F1 values for ActRef (computed on all commits) are not directly comparable to those baselines. All methods should be scored on the identical commit set, or per-commit-set results should be reported.
- [Section 6.2 and Section 4.3] Matcher and threshold selection appears to have been performed on the evaluation data, and the manual validation procedure is under-specified. Section 6.2 states that the chosen GumTree matcher and threshold were selected after testing alternatives and that similarity thresholds materially affect detected actions. If these choices were made on the same 500 commits used in Table 2, the reported performance includes selection bias. In addition, Section 4.3 says only that 'in rare instances of ambiguity, a second validator was consulted,' with no inter-rater reliability measure and no statement of how many instances were double-checked. The authors should evaluate with a separate validation set or demonstrate through sensitivity analysis that the headline numbers are stable across reasonable threshold choices, and they should report validation statistics.
minor comments (5)
- [Table 2] The row sums do not match the reported totals: the five RQ1.1 types sum to 1,120, not 1,111, and the full 15-type total of 1,921 exceeds the stated dataset size of 1,914. Please reconcile the counts.
- [Algorithms 2-4] Algorithm 2 line 19 returns an extra D ('return Rmodule,D,Dunpaired,Iunpaired') that is not part of the declared output; Algorithm 3 line 10 assigns Actions_rem_i <- Actions_i, which appears to keep the full action set rather than the remaining actions after matched actions are removed; Algorithm 4 line 11 removes matched actions after adding the refactoring, but the order in which delete/add action pairs are considered is unspecified and may affect the results.
- [Section 4.1.2] The PR+MS baseline takes the intersection of commits detected by MLRefScanner and PyRef and then labels them with PyRef; this makes PR+MS a commit-level filter rather than a directly comparable refactoring-type detector, so the claim that this setup 'enables a fair comparison' should be softened or justified.
- [Figure 5] The violin plot is hard to read: the numeric annotations such as '1.311.72' appear to run together, and the y-axis should state explicitly that it is a log-scaled runtime in seconds.
- [Abstract] The phrase '1,914 manually validated refactoring instances' overstates the process described in Section 4.3, where some instances were accepted as true positives without manual review; the abstract should be aligned with the actual validation procedure.
Circularity Check
Oracle for ActRef's precision/recall is built from ActRef's own detections and validated with ActRef's Table 1 rules, so the headline numbers are not against an independent ground truth.
-
fitted input called prediction
[Section 4.3, Dataset (oracle extension paragraph)]
"To extend the oracle, we executed ActRef, PyRef, DeepSeek-R1, and ChatGPT-4 on all 500 commits. We then manually validated the detected refactorings. During the manual detection phase, we directly classified results that matched the original dataset as true positives. Additionally, refactoring instances that were consistently detected by both ActRef and PyRef were also considered true positives without further review."
The ground truth used to compute ActRef's precision (0.80) and recall (0.92) is partially constructed from ActRef's own output: any instance detected by both ActRef and PyRef is accepted as a true positive without independent verification. Shared false positives are therefore silently counted as correct, inflating measured precision, and ActRef's misses are less visible because ActRef helped define which refactorings exist in the oracle. The headline evaluation thus reduces, in part, to ActRef's own detections rather than to an external benchmark.
-
self definitional
[Section 4.3, Dataset (manual validation sentence)]
"The validation process followed a predefined set of rigorous rules (shown in Table 1) to ensure consistency."
Table 1 is the same rule table ActRef uses for detection in Algorithms 3 and 4 (e.g., 'if α can match rule in Table 1 then Rintra ← corresponding refactoring'). Manual validation against Table 1 means the 'manually validated' oracle encodes ActRef's own detection definitions, so checking ActRef against this oracle is partly checking ActRef against its own rules rather than against independent human judgment or an externally derived dataset. This makes the reported precision and recall partially definitional.
full rationale
The central evaluation is partially circular. ActRef's headline precision and recall are computed against an oracle that was extended by running ActRef itself: instances detected by both ActRef and PyRef were accepted as true positives without further review, and manual validation followed Table 1, which is the same rule table ActRef's Algorithms 3-4 use to produce detections. This means shared false positives can be counted as correct, and the oracle partially encodes ActRef's own detection definitions, inflating the reported numbers and preventing the claim of consistently superior detection from being established against an independent ground truth. The external-threat paragraph in Section 6.3 acknowledges bias toward simpler refactorings, but it does not address this self-constructed oracle. This is partial circularity rather than full definitional equivalence: the original dataset from prior work [7,12] and some manual review of divergent cases provide external grounding, and self-citation is not the load-bearing issue. The restriction of the dataset to refactoring types supported by Table 1 is a selection concern, but it is not itself a circular reduction. Because the main precision/recall results are partially forced by the evaluation design while the method still has independent algorithmic content, a score of 6 is appropriate.
Assumptions & free parameters
free parameters (3)
- GumTree matcher and self-adaptive threshold =
not reported
- is_similar similarity threshold =
not reported
- Module-level slicing and similarity parameters =
not reported
assumptions (4)
- domain assumption AST structural similarity is sufficient to identify behavior-preserving refactorings in Python
- ad hoc to paper The manually extended dataset is an unbiased ground truth
- domain assumption GumTree's AST differencing reliably represents Python code changes without static type information
- domain assumption The prior PyRef and Python-Adaptive-RefactoringMiner datasets contain valid refactoring instances
Cite this review
Pith. "Pith review of ActRef: Enhancing the Understanding of Python Code Refactoring with Action-Based Analysis." pith.science (2026). https://pith.science/paper/B7JSM5AT
@misc{pith2026250506553,
author = {Pith},
title = {Pith review of: ActRef: Enhancing the Understanding of Python Code Refactoring with Action-Based Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/B7JSM5AT}},
note = {Machine review of arXiv:2505.06553}
}
read the original abstract
Refactoring, the process of improving the code structure of a software system without altering its behavior, is crucial for managing code evolution in software development. Identifying refactoring actions in source code is essential for understanding software evolution and guiding developers in maintaining and improving the code quality. This study presents an action-based Refactoring Analysis Framework named ActRef, a novel algorithm designed to advance the detection and understanding of Python refactorings through a unique code change action-based analysis of code changes. ActRef mining multiple refactoring types (e.g., move, rename, extract, and inline operations) based on diff actions, covering multiple granularity levels including variable, method, class, and module levels. By focusing on the code change actions, ActRef provides a Python-adaptive solution to detect intricate refactoring patterns. Our evaluation, conducted on 1,914 manually validated refactoring instances from 136 open-source Python projects. The evaluation results show that ActRef achieves high precision(0.80) and recall(0.92), effectively identifying multiple refactoring types. Compared with leading baselines, including PyRef, PyRef with MLRefScanner, DeepSeek-R1 and ChatGPT-4, ActRef consistently demonstrates superior performance in detecting Python refactorings across various types. While matching PyRef in runtime efficiency, ActRef supports a broader spectrum of refactoring types and more refactoring mining levels. ActRef shows an effective and scalable approach for mining refactorings in dynamic Python codebases and introduces a new perspective on understanding code.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
[n. d.]. Replication package. https://figshare.com/s/984c7a39266137e29c37
-
[2]
2024. Imagaug. https://github.com/aleju/imgaug
work page 2024
-
[3]
Eman Abdullah Alomar. 2019. Towards Better Understanding Developer Perception of Refactoring. IEEE (2019)
work page 2019
-
[4]
Eman Abdullah AlOmar, Mohamed Wiem Mkaouer, Ali Ouni, and Marouane Kessentini. 2019. On the impact of refactoring on the relationship between quality attributes and design metrics. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) . IEEE, 1–11
work page 2019
-
[5]
Eman Abdullah AlOmar, Tianjia Wang, Vaibhavi Raut, Mohamed Wiem Mkaouer, Christian Newman, and Ali Ouni
-
[6]
Maurício Aniche, Christoph Treude, Andy Zaidman, Arie Van Deursen, and Marco Aurélio Gerosa. 2016. SATT: Tailoring code metric thresholds for different software architectures. In 2016 IEEE 16th international working conference on source code analysis and manipulation (SCAM) . IEEE, 41–50
work page 2016
-
[7]
Hassan Atwi, Bin Lin, Nikolaos Tsantalis, Yutaro Kashiwa, Yasutaka Kamei, Naoyasu Ubayashi, Gabriele Bavota, and Michele Lanza. 2021. PyRef: refactoring detection in Python projects. In 2021 IEEE 21st international working conference on source code analysis and manipulation (SCAM) . IEEE, 136–141
work page 2021
-
[8]
Abdulrahman Ahmed Bobakr Baqais and Mohammad Alshayeb. 2020. Automatic software refactoring: a systematic literature review. Software Quality Journal 28, 2 (2020), 459–502
work page 2020
Show all 42 references
-
[9]
Zhiyuan Chen, Hai-Feng Guo, and Myoungkyu Song. 2018. Improving regression test efficiency with an awareness of refactoring changes. Information and Software Technology 103 (2018), 174–187
2018
-
[10]
DeepSeek. 2025. DeepSeek-R1. https://www.deepseek.com/
2025
-
[11]
Kayla DePalma, Izabel Miminoshvili, Chiara Henselder, Kate Moss, and Eman Abdullah AlOmar. 2024. Exploring ChatGPT’s code refactoring capabilities: An empirical study. Expert Systems with Applications 249 (2024), 123602. https://doi.org/10.1016/j.eswa.2024.123602
2024
-
[12]
Malinda Dilhara, Ameya Ketkar, Nikhith Sannidhi, and Danny Dig. 2022. Discovering repetitive code changes in python ml systems. In Proceedings of the 44th International Conference on Software Engineering . 736–748
2022
-
[13]
Jean-Rémy Falleri and Matias Martinez. 2024. Fine-grained, accurate and scalable source differencing. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–12
2024
-
[14]
Jean-Rémy Falleri, Floréal Morandat, Xavier Blanc, Matias Martinez, and Martin Monperrus. 2014. Fine-grained and accurate source code differencing. In Proceedings of the 29th ACM/IEEE international conference on Automated software engineering. 313–324
2014
-
[15]
Martin Fowler. 2018. Refactoring. Addison-Wesley Professional
2018
-
[16]
DuBose, and Emerson Murphy-Hill
Xi Ge, Quinton L. DuBose, and Emerson Murphy-Hill. 2012. Reconciling manual and automatic refactoring. In 2012 34th International Conference on Software Engineering (ICSE) . 211–221. https://doi.org/10.1109/ICSE.2012.6227192
2012
-
[17]
Martina Iammarino, Fiorella Zampetti, Lerina Aversano, and Massimiliano Di Penta. 2019. Self-Admitted Technical Debt Removal and Refactoring Actions: Co-Occurrence or More?. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME) . 186–190. https://...
2019
-
[18]
Martina Iammarino, Fiorella Zampetti, Lerina Aversano, and Massimiliano Di Penta. 2019. Self-admitted technical debt removal and refactoring actions: Co-occurrence or more?. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 186–190
2019
-
[19]
Toshihiro Kamiya, Shinji Kusumoto, and Katsuro Inoue. 2002. CCFinder: A multilinguistic token-based code clone detection system for large scale source code. IEEE transactions on software engineering 28, 7 (2002), 654–670
2002
-
[20]
Miryung Kim, Matthew Gee, Alex Loh, and Napol Rachatasumrit. 2010. Ref-finder: a refactoring reconstruction tool based on logic query templates. In Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering. 371–372
2010
-
[21]
Yun Lin, Xin Peng, Yuanfang Cai, Danny Dig, Diwen Zheng, and Wenyun Zhao. 2016. Interactive and guided architectural refactoring with search-based recommendation. InProceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering . 535–546
2016
-
[22]
Bo Liu, Hui Liu, Nan Niu, Yuxia Zhang, Guangjie Li, and Yanjie Jiang. 2023. Automated Software Entity Matching Between Successive Versions. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering , Vol. 1, No. 1, Article . Publication date: September 2...
2023
-
[23]
Zheng Ma, Yuexiu Gao, Lei Lyu, and Chen Lyu. 2022. MMF3: neural code summarization based on multi-modal fine- grained feature fusion. In Proceedings of the 16th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 171–182
2022
-
[24]
Shayan Noei, Heng Li, and Ying Zou. 2025. Detecting Refactoring Commits in Machine Learning Python Projects: A Machine Learning-Based Approach. ACM Transactions on Software Engineering and Methodology 34, 3 (2025), 1–25
2025
-
[25]
OpenAI. 2025. ChatGPT-4. https://openai.com/index/chatgpt
2025
-
[26]
Juri Opitz and Sebastian Burst. 2019. Macro f1 and macro f1. arXiv preprint arXiv:1911.03347 (2019)
2019 arXiv
-
[27]
Napol Rachatasumrit and Miryung Kim. 2012. An empirical investigation into the impact of refactoring on regression testing. In 2012 28th ieee international conference on software maintenance (icsm) . IEEE, 357–366
2012
-
[28]
Bo Shen, Wei Zhang, Haiyan Zhao, Guangtai Liang, Zhi Jin, and Qianxiang Wang. 2019. Intellimerge: A refactoring- aware software merging technique. Proceedings of the ACM on Programming Languages 3, OOPSLA (2019), 1–28
2019
-
[29]
Christopher M Siefert, Timothy A Smith, and Elliott M Ridgway. 2021. Evaluation of Programming Language-A ware Diffs for Improving Developer Productivity. Technical Report. Sandia National Lab.(SNL-NM), Albuquerque, NM (United States)
2021
-
[30]
Danilo Silva, Joao Paulo da Silva, Gustavo Santos, Ricardo Terra, and Marco Tulio Valente. 2020. Refdiff 2.0: A multi-language refactoring detection tool. IEEE Transactions on Software Engineering 47, 12 (2020), 2786–2802
2020
-
[31]
Danilo Silva, Nikolaos Tsantalis, and Marco Tulio Valente. 2016. Why we refactor? confessions of github contributors. In Proceedings of the 2016 24th acm sigsoft international symposium on foundations of software engineering . 858–870
2016
-
[32]
Danilo Silva and Marco Tulio Valente. 2017. Refdiff: detecting refactorings in version histories. In 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR) . IEEE, 269–279
2017
-
[33]
Tiobe. 2025. Tiobe2025. https://www.tiobe.com/tiobe-index/
2025
-
[34]
Nikolaos Tsantalis, Ameya Ketkar, and Danny Dig. 2020. RefactoringMiner 2.0. IEEE Transactions on Software Engineering 48, 3 (2020), 930–950
2020
-
[35]
Nikolaos Tsantalis, Matin Mansouri, Laleh M Eshkevari, Davood Mazinanian, and Danny Dig. 2018. Accurate and efficient refactoring detection in commit history. In Proceedings of the 40th international conference on software engineering. 483–494
2018
-
[36]
Kaiyuan Wang, Chenguang Zhu, Ahmet Celik, Jongwook Kim, Don Batory, and Milos Gligoric. 2018. Towards refactoring-aware regression test selection. In Proceedings of the 40th International Conference on Software Engineering . 233–244
2018
-
[37]
Peter Weißgerber and Stephan Diehl. 2006. Identifying refactorings from source-code changes. In 21st IEEE/ACM international conference on automated software engineering (ASE’06) . IEEE, 231–240
2006
-
[38]
Zhenchang Xing and Eleni Stroulia. 2005. UMLDiff: an algorithm for object-oriented design differencing. InProceedings of the 20th IEEE/ACM international Conference on Automated software engineering . 54–65
2005
-
[39]
Zhenchang Xing and Eleni Stroulia. 2006. Refactoring Detection based on UMLDiff Change-Facts Queries. In 2006 13th Working Conference on Reverse Engineering. 263–274. https://doi.org/10.1109/wcre.2006.48
2006 doi
-
[40]
Zhenchang Xing and Eleni Stroulia. 2008. The JDEvAn tool suite in support of object-oriented evolutionary development. In Companion of the 30th international conference on Software engineering . 951–952
2008
-
[41]
Yilin Yang, Tianxing He, Yang Feng, Shaoying Liu, and Baowen Xu. 2022. Mining Python fix patterns via analyzing fine-grained source code changes. Empirical Software Engineering 27, 2 (2022), 48. , Vol. 1, No. 1, Article . Publication date: September 2025
2022
-
[2022]
Innovations in Systems and Software Engineering (2022), 1–31
Refactoring for reuse: an empirical study. Innovations in Systems and Software Engineering (2022), 1–31
2022
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.