Automated Generation of Commit Messages in Software Repositories

Abbas Heydarnoori; Varun Kumar Palakodeti

REVIEW 4 major objections 6 minor 21 references

Automated Generation of Commit Messages in Software Repositories

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A lightweight TF-IDF cosine-similarity model with nearest-neighbor lookup reaches a BLEU score of 16.82 on the standard commit-message benchmark, beating several earlier neural and retrieval baselines while running on a CPU.

desk verdict A thin, reproducible benchmark showing a 0.4 BLEU edge over NNGen that probably doesn't survive matching preprocessing; deserves a workshop-level referee but not a top venue. read the letter →

arxiv 2504.12998 v1 pith:Y3SO3UMK submitted 2025-04-17 cs.SE

Varun Kumar Palakodeti , Abbas Heydarnoori This is my paper · ORCID

classification cs.SE

keywords commitmessagegenerationcodedocumentationTF-IDFcosinesimilaritynearestneighborsBLEUscorelightweightmachinelearninglargelanguagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that lightweight, locally runnable machine-learning models can generate commit messages from code diffs without the GPU demands of neural translation or retrieval-augmented systems. On the standard Java benchmark, a cosine-similarity model using TF-IDF vectors and a nearest-neighbor lookup achieves a BLEU score of 16.82, the highest among the models the authors built and higher on this metric than NMT, NNGen, CoDiSum, Lucene, and CommitGen. Manual evaluations add a second claim: ChatGPT produces messages that human evaluators prefer over all the ML/NLP models, especially for large diffs, while the lightweight models match human expectations best on small diffs. If correct, the results imply that simple retrieval baselines remain competitive for small changes and that the main advantage of large language models appears where diffs are large.

What carries the argument

The load-bearing machinery is the pairing of TF-IDF vectorization with cosine similarity and a nearest-neighbor search over training diffs. TF-IDF turns each diff into a weighted bag-of-words vector; the model then finds the most similar training diff and returns the corresponding human-written commit message. No training phase is needed, which is why the method runs in seconds on a CPU. This same retrieval pattern is what lets a simple model exceed several generative approaches on BLEU, since retrieved real messages are well-formed and grounded in actual developer writing.

What would settle it

Run the released code on the benchmark's unchanged test split with the original preprocessing and the same BLEU script, then recompute NNGen's score under the same conditions; if the cosine model no longer beats NNGen or the 16.82 score does not reproduce, the central quantitative claim fails.

Watch

Extended reading notes

Core claim

The central empirical claim is that on the benchmark dataset of Java code changes, the Cosine Similarity with TF-IDF and Nearest Neighbors model reaches BLEU 16.82, which the authors report as surpassing NNGen, NMT, CoDiSum, Lucene, and CommitGen on the same benchmark while remaining below RACE's 25.66. A second claim is that manual evaluation reverses the ranking for quality: ChatGPT was rated best across diff sizes and clearly outclassed all proposed models on large diffs, whereas models like Logistic Regression with TF-IDF often produced messages closer to the actual ones on small diffs. The paper also claims that these results support using lightweight models in resource-constrained and privacy-sensitive settings, since they run in seconds or minutes on a CPU or a laptop chip.

Load-bearing premise

The comparison stands on the assumption that the authors' data split, preprocessing, and BLEU calculation match the benchmark used to produce the earlier published scores; if the extra stop-word removal or lemmatization changes the test distribution, the headline ranking could be an artifact.

Editorial extensions

If this is right

A retrieval-based baseline can outperform several neural generators on BLEU without training, so future commit-message work would benefit from reporting such a baseline for calibrating gains.
The best BLEU score need not correspond to the best human-rated quality; evaluation should combine automatic metrics with manual review of small, medium, and large diffs.
Lightweight models make automatic commit-message generation feasible on laptops and in settings where sending code to external language models is undesirable.
Large diffs remain the territory where LLMs show clear advantage; for small diffs, simpler models are competitive or even preferred.
Model rankings depend on diff size, so a single aggregate BLEU number hides an important per-size pattern.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Controlling for preprocessing would test whether the 16.82 score comes from the model itself or from extra stop-word removal and lemmatization; the paper does not isolate this.
A hybrid system that routes small diffs to the lightweight model and large diffs to an LLM would be a natural and testable extension of the paper's own evidence.
The same pipeline could in principle transfer to other languages, but the Java-only dataset means such transfer is unverified and would need new data and preprocessing.
Because the nearest-neighbor method inherits the vocabulary and phrasing of the training set, its ceiling may be tied to how repetitive commit messages are in the repository; the paper does not measure that ceiling.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper's headline result — BLEU 16.82 for a TF-IDF cosine-similarity nearest-neighbor model — is a hair above NNGen's 16.42, but I would not trust that margin. The authors use a custom preprocessing pipeline (stop-word removal, lemmatization) and never show that their BLEU computation sees the same tokens as the published baselines. Without that verification, the 0.4-point edge is plausibly an artifact. So the main claim that a lightweight model surpasses NNGen is not established.

What the paper does well: it's an honest, small empirical study on a standard dataset (Liu et al.'s, originally Jiang et al.'s), with code on Zenodo and a standard BLEU script. That is reproducible, and the paper frames its contribution as efficiency on small hardware, not state-of-art quality. The manual comparison with ChatGPT on large diffs is useful qualitative context, even if it's not a rigorous study.

Where it's soft: (1) The comparability issue above is real. They say their preprocessing is 'more thorough' than prior work, which is exactly the kind of change that can shift BLEU by more than the observed difference. There are no significance tests or variance estimates. (2) The LSTM score of 0.68 BLEU is implausibly low. That's not a performance limitation; it suggests a broken pipeline (data leakage, short training, or bad tokenization). The authors just call it a limitation, which is not credible. (3) Table 3 is clearly a copy-paste of Table 2 — the header says Word2Vec but the rows say TF-IDF. (4) The manual evaluation is unblinded and likely author-run. They mention it as a threat to validity, but they still use it to draw conclusions about ChatGPT vs. the models.

Who this is for: someone building a baseline for commit message generation, or someone studying how much preprocessing affects retrieval-based methods. It is not a state-of-the-art paper. With heavy revision — fixing the table, computing confidence intervals, clarifying the BLEU preprocessing, and reporting dataset splits — it could be a solid workshop or short-paper contribution. I would accept it for peer review if framed as a replication study, but I would expect the reviewers to demand those fixes.

Regards.

Referee Report

4 major / 6 minor

Summary. The paper proposes lightweight machine learning and NLP models for automated commit message generation, trained on the Java repository dataset used by Liu et al. and Jiang et al. The evaluated approaches are cosine similarity with TF-IDF and nearest neighbors, logistic regression with TF-IDF, word2vec-based similarity and logistic regression, and a PyTorch LSTM. The main quantitative result is a BLEU score of 16.82 for the TF-IDF cosine similarity model, which the authors compare with published scores for RACE, NNGen, NMT, CoDiSum, and other baselines. A small manual evaluation compares the model outputs with ChatGPT-generated messages, with the authors reporting that ChatGPT is best on large diffs. The paper argues that simple, locally runnable models are competitive with more resource-intensive neural approaches.

Significance. If the comparative BLEU result is valid, the paper provides useful evidence that a lightweight TF-IDF retrieval method can be competitive with early neural commit message generators on a public benchmark, at much lower computational cost. The use of a public benchmark, the public code repository, and the use of the Liu et al. BLEU script are concrete strengths that facilitate replication. The significance is tempered, however, by the fact that the winning model is essentially TF-IDF-weighted nearest-neighbor retrieval, which is only a small variant of NNGen, and by the absence of statistical tests, unverified comparability of preprocessing with the published baselines, and a non-blinded manual evaluation. The paper's value is therefore more in the empirical comparison and resource-efficiency argument than in methodological novelty.

major comments (4)

[3.3-3.5 and Table 5] The headline comparison with published baselines is not yet established. The paper applies lowercasing, special-character removal, stop-word elimination, lemmatization/stemming, and tokenization before vectorization, but it does not state whether the same normalization is applied to both candidate and reference messages when computing BLEU, nor whether the train/test split is identical to the split used by Liu et al. for NNGen and by Jiang et al. for NMT. Since the margin over NNGen is only 0.40 BLEU points (16.82 vs. 16.42), these uncontrolled choices can easily flip the ranking. The authors should report the exact split, commit the split and preprocessing scripts to the repository, provide bootstrap confidence intervals or other significance measures, and include a sensitivity analysis with and without stop-word removal and lemmatization.
[3.5 and Table 4] The LSTM result of 0.68 BLEU is so far below the other models that it strongly suggests an implementation flaw rather than a genuine limitation of the architecture. No hyperparameters (embedding size, hidden size, layers, dropout, optimizer, learning rate, epochs, batch size) or decoding procedure are reported, so the result is not reproducible and cannot support the conclusion that LSTM is ineffective. The authors should either repair the pipeline, report full hyperparameters and training details, or remove the LSTM claim from the comparative analysis.
[3.6 and Section 5 (RQ2/RQ3)] The manual evaluation is not blinded, no number of raters is given, no inter-rater agreement is reported, and the sampling procedure is described only as 'a script selected random diffs.' The claims that ChatGPT 'outperformed all ML and NLP models' and that logistic regression matched human understanding rest entirely on this unsubstantiated protocol. The authors should use multiple independent raters, blind the outputs, report agreement statistics, and disclose whether the authors themselves were the evaluators.
[4.2 and Table 5] The messages produced by the cosine similarity model are copied from training-set commit messages, so the 16.82 BLEU result is a retrieval result rather than a generation result. The paper should state this explicitly, discuss the potential for duplicated or near-duplicated diffs between the training and test sets, and clarify the exact technical difference from NNGen (TF-IDF weighting versus bag-of-words) so that readers can judge whether the comparison is meaningful or merely a retuning of the same method.

minor comments (6)

[1 and 4/5] The paper's organization is confusing: the Introduction says Section 4 reviews related work and Section 5 discusses results, but in the actual text Section 5 (Discussion) appears before Section 4 (Related Work). The ordering should be fixed.
[Table 3] Table 3 is a copy-paste of Table 2: the columns still read 'Logistic Regression with TF-IDF' and 'Cosine Similarity with TF-IDF,' even though the table is supposed to compare Word2Vec-based models. The table should be corrected.
[Abstract and Section 3.4] The abstract states that the data is split into training, validation, and testing sets, but no validation set is used in the reported results; the authors should clarify whether validation was used for early stopping or hyperparameter selection and how.
[Section 5] The statement that ChatGPT's BLEU score 'could not be included as it was not released' is unclear, since BLEU can be computed locally from generated outputs. The authors should clarify whether the outputs were not saved or the model was not accessible locally.
[Throughout] The manuscript contains numerous typos and grammatical errors, including 'approch', 'chose', and incomplete sentences in Section 3.3. A careful proofreading pass is needed.
[Section 6.3] The reliability section claims that 'the evaluators are proven to show a significant level of experience and expertise with the Java programming language,' but no evidence or details about the evaluators are provided. This claim should be either substantiated or removed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: benchmark evaluation is self-contained; self-citations are motivational only.

full rationale

The paper's central empirical claim is that a lightweight TF-IDF cosine similarity with nearest-neighbors model achieves BLEU 16.82 on the Liu et al. benchmark, surpassing several published baselines. Nothing in the derivation is defined in terms of the outcome: the model is constructed from training data via TF-IDF vectorization and nearest-neighbor retrieval, then scored with the Liu et al./Jiang et al. BLEU script against held-out reference messages. The comparison to published baselines depends on shared dataset and evaluation script; any mismatch in preprocessing or split would be a reproducibility/comparability concern, but not a circularity. The only self-citations ([8], [14]) are motivational background citations by a co-author and are not load-bearing: they do not define the models, the benchmark, the BLEU computation, or the result. No fitted parameter is renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work. Therefore no circular step is identified.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests mainly on a public benchmark and a standard BLEU script, with a few hyperparameter choices (K=1 for retrieval, 100-token LSTM padding) and generic assumptions about vector representation of diffs. No new entities are introduced.

free parameters (3)

Number of neighbors k in KNN retrieval = 1 (not explicitly stated)
The model 'selecting the nearest vector from the training data' implies k=1, a hand-chosen hyperparameter affecting the retrieved commit message.
LSTM input sequence length = 100 tokens
The paper standardizes inputs to 100 tokens by padding/trimming in Section 2.3, a manual choice influencing LSTM performance.
Pre-trained Word2Vec model = Google News 300d
Word2Vec variants rely on Google's pre-trained vectors as an external resource; the vector space is not learned from the commit data.

assumptions (3)

domain assumption The Liu et al./Jiang et al. dataset is a valid benchmark for commit message generation and the BLEU script from those works is the correct metric.
The paper uses this dataset and script for all comparisons; if the benchmark is flawed, the central numbers lose meaning.
domain assumption Code diffs can be represented as bags of words (via TF-IDF or Word2Vec) without losing the information needed for commit messages.
The retrieval-based models assume that semantic similarity of diffs in vector space corresponds to similar commit messages; this is implicit in Section 2.4.
standard math Cosine similarity is an appropriate measure of vector similarity for nearest-neighbor retrieval.
The method uses cosine similarity; this is a standard mathematical choice but is assumed to be effective.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automated Generation of Commit Messages in Software Repositories." pith.science (2026). https://pith.science/paper/Y3SO3UMK

@misc{pith2026250412998,
  author       = {Pith},
  title        = {Pith review of: Automated Generation of Commit Messages in Software Repositories},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y3SO3UMK}},
  note         = {Machine review of arXiv:2504.12998}
}

read the original abstract

Commit messages are crucial for documenting software changes, aiding in program comprehension and maintenance. However, creating effective commit messages is often overlooked by developers due to time constraints and varying levels of documentation skills. Our research presents an automated approach to generate commit messages using Machine Learning (ML) and Natural Language Processing (NLP) by developing models that use techniques such as Logistic Regression with TF-IDF and Word2Vec, as well as more sophisticated methods like LSTM. We used the dataset of code changes and corresponding commit messages that was used by Liu et al., which we used to train and evaluate ML/NLP models and was chosen because it is extensively used in previous research, also for comparability in our study. The objective was to explore which ML/NLP techniques generate the most effective, clear, and concise commit messages that accurately reflect the code changes. We split the dataset into training, validation, and testing sets and used these sets to evaluate the performance of each model using qualitative and quantitative evaluation methods. Our results reveal a spectrum of effectiveness among these models, with the highest BLEU score achieved being 16.82, showcasing the models' capability in automating a clear and concise commit message generation. Our paper offers insights into the comparative effectiveness of different machine learning models for automating commit message generation in software development, aiming to enhance the overall practice of code documentation. The source code is available at https://doi.org/10.5281/zenodo.10888106.

Figures

Figures reproduced from arXiv: 2504.12998 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 18 canonical work pages

[1]

https://lucene.apache.org/

Apache lucene. https://lucene.apache.org/. Ac- cessed: 07-01-2024

work page 2024
[2]

https://doi.org/10.5281/ zenodo.10888106

Automated Generation of Commit Messages in Soft- ware Repositories. https://doi.org/10.5281/ zenodo.10888106. Accessed: 07-01-2024

work page 2024
[3]

https://code.google.com/archive/ p/word2vec/

Word2Vec. https://code.google.com/archive/ p/word2vec/. Accessed: 07-01-2024

work page 2024
[4]

R. P. Buse and W. R. Weimer. Automatically documenting program changes. In Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineer- ing, pages 33–42, 2010

work page 2010
[5]

Harnessing Knowledge and Reasoning for Human-Like Natural Language Generation: A Brief Review

J. Chen and Y . Xiao. Harnessing knowledge and reasoning for human-like natural language generation: A brief review. arXiv preprint arXiv:2212.03747, 2022

work page Pith review arXiv 2022
[6]

J. Dong, Y . Lou, D. Hao, and L. Tan. Revisiting learning- based commit message generation. In Proceedings of the 45th IEEE/ACM International Conference on Software En- gineering, pages 794–805, 2023

work page 2023
[7]

R. Dyer, H. A. Nguyen, H. Rajan, and T. N. Nguyen. BOA: A language and infrastructure for analyzing ultra-large-scale software repositories. In Proceedings of the 35th IEEE/ACM International Conference on Software Engineering , pages 422–431, 2013

work page 2013
[8]

Jafari, F

M. Jafari, F. Majidi, and A. Heydarnoori. Prioritizing app reviews for developer responses on Google Play. In Pro- ceedings of the 30th International DMS Conference on Vi- sualization and Visual Languages, October 2024

work page 2024

Show all 21 references

[9]

Jiang, A

S. Jiang, A. Armaly, and C. McMillan. Automatically gen- erating commit messages from diffs using neural machine translation. In Proceedings of the 32nd IEEE/ACM Inter- national Conference on Automated Software Engineering , pages 135–146, 2017

2017
[10]

Lavie and M

A. Lavie and M. Denkowski. The METEOR metric for au- tomatic evaluation of machine translation. Machine Trans- lation, 23:105–115, September 2009

2009
[11]

C.-Y . Lin. ROUGE: A package for automatic evaluation of summaries. In Proceedings of the ACL Workshop: Text Sum- marization Braches Out, January 2004

2004
[12]

Z. Liu, X. Xia, A. E. Hassan, D. Lo, Z. Xing, and X. Wang. Neural-machine-translation-based commit message gener- ation: How far are we? In Proceedings of the 33rd ACM/IEEE International Conference on Automated Soft- ware Engineering, pages 373–384, 2018

2018
[13]

Loyola, E

P. Loyola, E. Marrese-Taylor, and Y . Matsuo. A neural ar- chitecture for generating natural language descriptions from source code changes. arXiv preprint arXiv:1704.04856 , 2017

2017 arXiv
[14]

P. R. Mazrae, M. Izadi, and A. Heydarnoori. Automated recovery of issue-commit links leveraging both textual and non-textual data. In Proceedings of the 37th IEEE Interna- tional Conference on Software Maintenance and Evolution, pages 263–273, 2021

2021
[15]

Papineni, S

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, 2002

2002
[16]

E. Shi, Y . Wang, W. Tao, L. Du, H. Zhang, S. Han, D. Zhang, and H. Sun. RACE: Retrieval-augmented commit message generation. arXiv preprint arXiv:2203.02700, 2022

2022 arXiv
[17]

Y . Tian, Y . Zhang, K.-J. Stol, L. Jiang, and H. Liu. What makes a good commit message? In Proceedings of the 44th IEEE/ACM International Conference on Software Engineer- ing, pages 2389–2401, 2022

2022
[18]

Touvron, T

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971, 2023

2023 arXiv
[19]

S. Xu, Y . Yao, F. Xu, T. Gu, H. Tong, and J. Lu. Commit message generation for source code changes. In Proceed- ings of the 28th International Joint Conference on Artificial Intelligence, pages 3975–3981, 2019

2019
[20]

Zhang, J

L. Zhang, J. Zhao, C. Wang, and P. Liang. Using large lan- guage models for commit message generation: A prelimi- nary study. arXiv preprint arXiv:2401.05926, 2024

2024 arXiv
[21]

Zhang, Z

Y . Zhang, Z. Qiu, K. Stol, W. Zhu, J. Zhu, Y . Tian, and H. Liu. Automatic commit message generation: A critical review and directions for future work. IEEE Transactions on Software Engineering, 50(4):816–835, April 2024

2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

https://lucene.apache.org/

Apache lucene. https://lucene.apache.org/. Ac- cessed: 07-01-2024

work page 2024

[2] [2]

https://doi.org/10.5281/ zenodo.10888106

Automated Generation of Commit Messages in Soft- ware Repositories. https://doi.org/10.5281/ zenodo.10888106. Accessed: 07-01-2024

work page 2024

[3] [3]

https://code.google.com/archive/ p/word2vec/

Word2Vec. https://code.google.com/archive/ p/word2vec/. Accessed: 07-01-2024

work page 2024

[4] [4]

R. P. Buse and W. R. Weimer. Automatically documenting program changes. In Proceedings of the 25th IEEE/ACM International Conference on Automated Software Engineer- ing, pages 33–42, 2010

work page 2010

[5] [5]

Harnessing Knowledge and Reasoning for Human-Like Natural Language Generation: A Brief Review

J. Chen and Y . Xiao. Harnessing knowledge and reasoning for human-like natural language generation: A brief review. arXiv preprint arXiv:2212.03747, 2022

work page Pith review arXiv 2022

[6] [6]

J. Dong, Y . Lou, D. Hao, and L. Tan. Revisiting learning- based commit message generation. In Proceedings of the 45th IEEE/ACM International Conference on Software En- gineering, pages 794–805, 2023

work page 2023

[7] [7]

R. Dyer, H. A. Nguyen, H. Rajan, and T. N. Nguyen. BOA: A language and infrastructure for analyzing ultra-large-scale software repositories. In Proceedings of the 35th IEEE/ACM International Conference on Software Engineering , pages 422–431, 2013

work page 2013

[8] [8]

Jafari, F

M. Jafari, F. Majidi, and A. Heydarnoori. Prioritizing app reviews for developer responses on Google Play. In Pro- ceedings of the 30th International DMS Conference on Vi- sualization and Visual Languages, October 2024

work page 2024