REVIEW 5 major objections 4 minor 1 cited by
LastMerge: A language-agnostic structured tool for code integration
T0 review · 5 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Generic structured merge tools can achieve accuracy and speed comparable to language-specific tools, according to a replay of 5,229 real merge scenarios.
desk verdict Solid tool paper whose central accuracy comparison is undermined by a circular ground-truth oracle: LastMerge judges its own outputs. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is LastMerge's core merge engine, which operates on concrete syntax trees (CSTs) produced by Tree-sitter, a parser framework with grammars for more than 350 languages. A thin configuration interface supplies language-specific aspects: which nonterminal nodes are unordered, how to extract node identifiers using Tree-sitter queries, and optional parsing handlers that restructure trees, such as grouping imports. Matching adapts jDime's algorithms to these generic trees, using Yang's algorithm for ordered children and a linear-programming maximum matching for unordered children; the merge phase reports conflicts only when differing changes affect corresponding tree nodes. Mergiraf, a second generic tool, adapts Spork's algorithm with GumTree matching and auto-tuning, providing evidence that the result generalizes beyond one particular design.
What would settle it
Recompute the aFP/aFN counts with an independent ground-truth mechanism—for example, human-annotated conflict labels on a random sample of the 400 and 601 disagreement scenarios—and check whether LastMerge's 15% fewer false positives and Mergiraf's 42% fewer false negatives persist; expanding the manual analysis from 40 scenarios to the full disagreement set would also confirm or overturn the claim that most differences are configuration details.
Extended reading notes
Core claim
The paper claims that generic structured merge tools can effectively replace language-specific ones, achieving similar levels of accuracy and efficiency. Concretely, it argues that a merge engine operating over language-independent concrete syntax trees, configured by a thin per-language description, reproduces the behavior of Java-specific tools: LastMerge reports 15% fewer added false positives than jDime, Mergiraf misses 42% fewer added false negatives than Spork, and both generic tools run at least as fast. The 7.53% and 12.22% disagreement rates with their specific counterparts stem mostly from configuration choices and implementation details that could be adjusted, not from the language-independent design.
Load-bearing premise
The accuracy counts depend on using LastMerge itself to parse and tree-match tool outputs against the repository merge commit as ground truth; if LastMerge's parsing or matching is systematically biased, the relative false-positive and false-negative numbers for every tool are skewed.
Editorial extensions
If this is right
- Structured merge support for a new language reduces to providing a Tree-sitter grammar plus configuration queries and handlers, instead of implementing a full language-specific merge engine.
- Developers working in multi-language projects could get structured merge accuracy everywhere, without maintaining a separate tool per language.
- The observed differences between generic and specific tools would shrink if configurations were aligned, since most disagreements are attributed to adjustable configuration details.
- Generic structured merge does not impose a runtime penalty over the studied specific tools, so adopting it does not cost performance.
- The same configuration interface can emulate semistructured merge behavior by stopping the tree at an intermediate level, a flexibility language-specific tools lack.
Reading between the lines
- If the finding generalizes beyond Java, the economics of structured merge change: the fixed cost per language is small, so covering many languages becomes a configuration effort rather than a research and engineering effort.
- The accuracy verdict leans on LastMerge itself to judge whether another tool's output matches the merge commit; an independent ground-truth check would strengthen or revise the reported aFP/aFN ratios.
- Auto-tuning, which Mergiraf already uses, could be added to LastMerge-style tools to get conflict-free speed when no conflict exists, a combination the paper notes is easy to implement.
- Since jDime and Spork only support Java, the same paired comparison for other languages is the natural next test; until then, the Java-only evidence leaves open how much of the result is language-specific.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces LastMerge, a language-agnostic structured merge tool built on Tree Sitter's concrete syntax trees and a thin language configuration interface. It reports an experiment comparing LastMerge and another generic tool, Mergiraf, against Java-specific tools jDime and Spork on 5,229 merge scenarios from an existing dataset. The evaluation uses relative added false positive (aFP) and added false negative (aFN) metrics, supplemented by test execution and a manual analysis of 40 scenarios. The authors report no evidence that generic structured merge significantly impacts accuracy, note that LastMerge has 15% fewer aFPs than jDime while Mergiraf has 42% fewer aFNs than Spork, and observe comparable or better runtime performance. They conclude that generic structured merge tools can effectively replace language-specific ones.
Significance. If the central claim holds, the paper would be an important contribution: it would show that the high cost of building language-specific structured merge tools can be avoided, enabling structured merge for a wide range of languages. The study has real strengths: it uses a large, publicly available dataset of merge scenarios; compares two independent generic tools against two language-specific baselines; measures runtime with repeated executions; and provides an online appendix with scripts and data. The paper is also honest about several threats to validity, including the Java-only evaluation and the reliance on test passing as an approximation of correctness. However, the accuracy comparison has a load-bearing circularity concern, the manual attribution relies on a small sample, and the broad conclusion overstates what the relative metrics and single-language evaluation can support.
major comments (5)
- [Section IV-C and Table II] The syntactic equivalence check that decides aFPs and aFNs uses LastMerge's own parser and matching algorithm: 'we use LAST MERGE to parse the files into trees and later rely on its matching algorithm to verify whether their program root nodes match entirely.' For the LastMerge vs. jDime comparison, LastMerge is therefore both the tool under test and the oracle. If this matcher is systematically permissive toward LastMerge's output representation, the aFP/aFN counts in Table II would be skewed in LastMerge's favor. The paper provides no independent validation of the matcher as a sound and complete equivalence relation. Please add an independent equivalence oracle (e.g., a separate parser or a normalized text diff) and report agreement on a sample, or manually evaluate a sample of equivalence decisions.
- [Section IV-A and Tables II-III] The aFP and aFN metrics count only scenarios where the two tools disagree on conflict existence; scenarios in which both tools make the same mistake are invisible. This relative design is appropriate for comparing tools, but it cannot support the abstract's claim that generic tools 'achieve similar levels of accuracy' in an absolute sense. Please either add an absolute accuracy assessment on a sample with known conflict status, or explicitly restrict the accuracy claim to relative differences between tools.
- [Section IV-C and Section V-A] The conclusion that most observed differences stem from 'implementation details and configuration choices' is based on manually analyzing only 5 aFPs and 5 aFNs per tool (40 scenarios total), drawn from 400 and 601 disagreement scenarios for the two tool pairs. This sample is too small to support the general claim, and no confidence intervals or inter-rater reliability are reported. Please enlarge the manual sample, report the uncertainty of the manual classification, or soften the generalization to the analyzed scenarios.
- [Sections IV-B and V-C] The experiment instantiates both generic tools only for Java, as Section V-C acknowledges. The abstract and conclusion, however, claim that generic structured merge tools can effectively replace language-specific ones and 'pav[e] the way for broader adoption of structured merge in industry.' This is an unsupported generalization beyond the Java evidence. Please either scope the claims to Java or provide an evaluation on at least one additional language.
- [Section V-A and Abstract] The statement 'our results show no evidence that generic structured merge significantly impacts merge accuracy' is not backed by any statistical test or confidence interval, and the observed relative differences are substantial: LastMerge has 85 aFNs versus jDime's 29, and Mergiraf has 290 aFPs versus Spork's 150. Please add significance tests or confidence intervals, or rephrase the conclusion to say that meaningful differences cannot be ruled out.
minor comments (4)
- [Sections V-A and V-B] Please fix the typos: 'auto tunnig' should be 'auto-tuning' (Section V-A), 'wether' should be 'whether' (Sections V-A and V-C), and 'to to an intermediate representation' should be 'to an intermediate representation' (Section V-B).
- [Throughout] The tool name is written inconsistently as 'LAST MERGE', 'LASTMERGE', 'LastMerge', and 'LASTMerge' (e.g., Figure 8 caption). Please choose a single notation and apply it consistently.
- [Table I] The footnote to Table I says 'The total sum in each column can vary because not all scenarios were successfully integrated by each tool.' This is unclear, since the sums exceed the 5,229-scenario sample size. Please clarify which scenarios are excluded from each column and report the effective number of scenarios.
- [Section IV-A] The phrase 'explained in detail latter' should be 'explained in detail later'.
Circularity Check
Accuracy comparison is partly self-referential: LastMerge's own parser and matcher act as the syntactic-equivalence oracle in the aFP/aFN classification (Section IV-C), so part of the headline accuracy counts is defined by the system under test; external merge-commit, test-suite, and runtime anchors keep the central claim from reducing entirely to its inputs.
-
self definitional
[Section IV-C (Checking Merge Accuracy and Performance), Figure 7; reported in Tables II–III.]
"we use LAST MERGE to parse the files into trees and later rely on its matching algorithm to verify whether their program root nodes match entirely. If the files are found to be syntactically equivalent, we assume B correctly resolved the conflict reported by A, and classify the scenario as an aFP for A."
The headline evidence that generic tools 'can effectively replace language-specific ones' is the aFP/aFN counts in Tables II–III, computed by the Figure 7 procedure. Its first oracle is LastMerge's own matcher: a disagreement is classified as an added false positive for the conflict-reporting tool whenever the other tool's output is judged by LastMerge's matcher to be equivalent to the developer's merge commit, preempting the external test step. LastMerge's merge engine (Sections III-A2–A3) uses that same matching relation (same-kind node pairing, identifier extraction, unordered-children maximum matching) to decide that node pairs are non-conflicting.
full rationale
The paper's central claim — that generic structured merge tools can effectively replace language-specific ones with similar accuracy and efficiency — rests on RQ1 (aFP/aFN counts in Tables II–III) and RQ2 (runtime, Figure 11). RQ2 is fully external and independent of the circularity concern: runtimes are measured directly, and the paper credits implementation language (Rust vs Java) and auto-tuning for the differences. RQ1, however, inherits one genuine self-referential component: the syntactic-equivalence predicate in Section IV-C is implemented by LastMerge's own parser (Tree Sitter) and matching algorithm. The step is detailed above. Several features keep this from being a full by-construction circularity: (1) the ground-truth merge commits come from the external Schesch et al. dataset; (2) when the matcher reports non-equivalence, classification falls back to executing the project's own external test suites, so the matcher is not the only judge; (3) the matcher is applied symmetrically to both tools' outputs and to the merge commit, so any systematic permissiveness or strictness affects both directions; (4) the observed outcome — LastMerge having nearly three times more aFNs (85 vs 29) — is not the direction a simply pro-LastMerge oracle would produce; and (5) the metric is comparative by design, ignoring cases where the tools agree. The paper also discloses related limitations in Section V-C: the aFP/aFN analysis 'relies on a heuristic that combines static and semantic analysis to approximate the existence of conflicts'; the manual attribution of differences to 'implementation details' rests on 40 scenarios inspected by the authors themselves, who note 'the authors possess in-depth knowledge of the design and behavior of each tool'; and the evaluation is Java-only. These are honesty markers that lower suspicion of hidden circularity, but the self-oracle nonetheless reduces the independence of the accuracy evidence. Self-citations [11], [16], [17] supply background and the aFP/aFN methodology; they are peer-reviewed and parameter-free, not unverified load-bearing premises. There is no fitted-parameter-called-prediction step, no imported uniqueness theorem, and no renaming of a known result. Net assessment: one partial self-referential evaluation step, score 3.
Assumptions & free parameters
free parameters (2)
- Java configuration of LastMerge =
unordered node kinds (class body, field declarations, import groups), identifier queries, parser handler for imports
- Dataset filtering thresholds =
require at least one mutually modified file; remove scenarios with tool crashes; remove scenarios from unavailable…
assumptions (4)
- domain assumption The merge commit in the repository is a good approximation of the expected merge result.
- domain assumption Passing the project test suite indicates that a merge output is non-conflicting.
- domain assumption The Schesch et al. dataset is representative of real-world Java merge scenarios.
- domain assumption Tree-sitter grammars for Java are correct and complete enough for the merged files.
Cite this review
Pith. "Pith review of LastMerge: A language-agnostic structured tool for code integration." pith.science (2026). https://pith.science/paper/RUYPEQS2
@misc{pith2026250719687,
author = {Pith},
title = {Pith review of: LastMerge: A language-agnostic structured tool for code integration},
year = {2026},
howpublished = {\url{https://pith.science/paper/RUYPEQS2}},
note = {Machine review of arXiv:2507.19687}
}
read the original abstract
Unstructured line-based merge tools are widely used in practice. Structured AST-based merge tools show significantly improved merge accuracy, but are rarely used in practice because they are language specific and costly, consequently not being available for many programming languages. To improve merge accuracy for a wide range of languages, we propose LastMerge, a generic structured merge tool that can be configured through a thin interface that significantly reduces the effort of supporting structured merge. To understand the impact that generic structured merge might have on merge accuracy and performance, we run an experiment with four structured merge tools: two Java specific tools, jDime and Spork, and their generic counterparts, respectively LastMerge and Mergiraf. Using each tool, we replay merge scenarios from a significant dataset, and collect data on runtime, behavioral divergences, and merge accuracy. Our results show no evidence that generic structured merge significantly impacts merge accuracy. Although we observe a difference rate of approximately 10% between the Java specific tools and their generic counterparts, most of the differences stem from implementation details and could be avoided. We find that LastMerge reports 15% fewer false positives than jDime while Mergiraf misses 42% fewer false negatives than Spork. Both generic tools exhibit comparable runtime performance to the state of the art language specific implementations. These results suggest that generic structured merge tools can effectively replace language-specific ones, paving the way for broader adoption of structured merge in industry.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 1 Pith paper
-
MergirafSemi: A Language-Agnostic Semistructured Merge Tool
MergirafSemi, a language-agnostic semistructured merge tool built on truncated syntax trees, reduces spurious conflicts relative to diff3 and matches a Java-specific semistructured tool's accuracy at far lower runtime...
Reference graph
Works this paper leans on
-
[1]
A state-of-the-art survey on software merging,
T. Mens, “A state-of-the-art survey on software merging,” IEEE Trans- actions on Software Engineering , vol. 28, no. 5, pp. 449–462, 2002
work page 2002
-
[2]
A formal investigation of diff3,
S. Khanna, K. Kunal, and B. C. Pierce, “A formal investigation of diff3,” in FSTTCS 2007: Foundations of Software Technology and Theoretical Computer Science (V . Arvind and S. Prasad, eds.), (Berlin, Heidelberg), pp. 485–496, Springer Berlin Heidelberg, 2007
work page 2007
-
[3]
Extensible language-aware merging,
J. Hunt and W. Tichy, “Extensible language-aware merging,” in In- ternational Conference on Software Maintenance, 2002. Proceedings. , pp. 511–520, 2002
work page 2002
-
[4]
Semistruc- tured merge: rethinking merge in revision control systems,
S. Apel, J. Liebig, B. Brandl, C. Lengauer, and C. K ¨astner, “Semistruc- tured merge: rethinking merge in revision control systems,” in Proceed- ings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering , ESEC/FSE ’11, (New York, NY , USA), p. 190–200, Association for Computing Ma- chinery, 2011
work page 2011
-
[5]
Enhancing precision of structured merge by proper tree matching,
F. Zhu, F. He, and Q. Yu, “Enhancing precision of structured merge by proper tree matching,” in 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) , pp. 286–287, 2019
work page 2019
-
[6]
Structure-oriented merging of revisions of software documents,
B. Westfechtel, “Structure-oriented merging of revisions of software documents,” in Proceedings of the 3rd International Workshop on Software Configuration Management, SCM ’91, (New York, NY , USA), p. 68–79, Association for Computing Machinery, 1991
work page 1991
-
[7]
Textual merge based on language-specific syntactic separators,
J. Clementino, P. Borba, and G. Cavalcanti, “Textual merge based on language-specific syntactic separators,” in Proceedings of the XXXV Brazilian Symposium on Software Engineering , SBES ’21, (New York, NY , USA), p. 243–252, Association for Computing Machinery, 2021
work page 2021
-
[8]
J. Buffenbarger, “Syntactic software merging,” in Software Configuration Management (J. Estublier, ed.), (Berlin, Heidelberg), pp. 153–172, Springer Berlin Heidelberg, 1995
work page 1995
Show all 23 references
-
[9]
Spork: Struc- tured Merge for Java With Formatting Preservation ,
S. Larsen, J.-R. Falleri, B. Baudry, and M. Monperrus, “ Spork: Struc- tured Merge for Java With Formatting Preservation ,” IEEE Transactions on Software Engineering , vol. 49, pp. 64–83, Jan. 2023
2023
-
[10]
Structured merge with auto- tuning: balancing precision and performance,
S. Apel, O. Leßenich, and C. Lengauer, “Structured merge with auto- tuning: balancing precision and performance,” in 2012 Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, pp. 120–129, 2012
2012
-
[11]
Leveraging structure in software merge: An empirical study,
G. Seibt, F. Heck, G. Cavalcanti, P. Borba, and S. Apel, “Leveraging structure in software merge: An empirical study,” IEEE Transactions on Software Engineering, vol. 48, no. 11, pp. 4590–4610, 2022
2022
-
[12]
Evaluation of version control merge tools,
B. Schesch, R. Featherman, K. J. Yang, B. Roberts, and M. D. Ernst, “Evaluation of version control merge tools,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ASE ’24, (New York, NY , USA), p. 831–83, Association for Computing ...
2024
-
[13]
tree-sitter/tree-sitter: v0.25.3,
M. Brunsfeld, “tree-sitter/tree-sitter: v0.25.3,” Mar. 2025
2025
-
[14]
Identifying syntactic differences between two programs,
W. Yang, “Identifying syntactic differences between two programs,” Softw. Pract. Exper., vol. 21, p. 739–755, June 1991
1991
-
[15]
Fine-grained and accurate source code differencing,
J.-R. Falleri, F. Morandat, X. Blanc, M. Martinez, and M. Monperrus, “Fine-grained and accurate source code differencing,” in Proceedings of the International Conference on Automated Software Engineering , pp. 313–324, 2014
2014
-
[16]
Evaluating and improving semistructured merge,
G. Cavalcanti, P. Borba, and P. Accioly, “Evaluating and improving semistructured merge,” Proc. ACM Program. Lang. , vol. 1, Oct. 2017
2017
-
[17]
Semistructured merge with language-specific syntactic separators,
G. Cavalcanti, P. Borba, L. d. Anjos, and J. Clementino, “Semistructured merge with language-specific syntactic separators,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software En- gineering, ASE ’24, (New York, NY , USA), p. 1032–1043, Associati...
2024
-
[18]
Greatest hits
GitHub, “Greatest hits.” https://archiveprogram.github.com/ greatest-hits/, nov 2020
2020
-
[19]
Curating github for engineered software projects,
N. Munaiah, S. Kroh, C. Cabrey, and M. Nagappan, “Curating github for engineered software projects,” Empirical Softw. Engg. , vol. 22, p. 3219–3253, Dec. 2017
2017
-
[20]
Renaming and shifted code in structured merging: Looking ahead for precision and performance,
O. Leßenich, S. Apel, C. K ¨astner, G. Seibt, and J. Siegmund, “Renaming and shifted code in structured merging: Looking ahead for precision and performance,” in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) , pp. 543–553, 2017
2017
-
[21]
Build conflicts in the wild,
L. Da Silva, P. Borba, and A. Pires, “Build conflicts in the wild,” J. Softw. Evol. Process, vol. 34, Apr. 2022
2022
-
[22]
Energy efficiency across programming languages: how do energy, time, and memory relate?,
R. Pereira, M. Couto, F. Ribeiro, R. Rua, J. Cunha, J. a. P. Fernandes, and J. a. Saraiva, “Energy efficiency across programming languages: how do energy, time, and memory relate?,” in Proceedings of the 10th ACM SIGPLAN International Conference on Software Language Engineerin...
2017
-
[23]
Conflict resolution for structured merge via version space algebra,
F. Zhu and F. He, “Conflict resolution for structured merge via version space algebra,” Proc. ACM Program. Lang. , vol. 2, Oct. 2018
2018
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.