{"id":"5a4f189e-c187-4d55-b4cd-e12f2a7e2891","arxiv_id":"2501.09861","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A commit-message optimization method that starts from human-written messages and uses GPT-4 plus automated evaluators outperforms message generation and completion methods on three of four quality metrics.","lead":"This paper presents CMO, a system that takes a developer's written commit message and iteratively improves it using GPT-4 and automated quality scores, rather than generating a message from scratch. In human evaluations it was ranked above state-of-the-art generation and completion methods on rationality, comprehensiveness, and expressiveness, but not on conciseness.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 88.2%–95.4% comes only from the two in-house researchers' rankings; the 22 independent Apache developers ranked CMO first far less often (Expressiveness 38.6%, Conciseness 6.8%), and no significance tests support 'significantly'.","rationale":"The reader's weakest assumption—that the automated evaluators approximate human judgment—is real and affects the automated results (Table 5) and the search objective. However, the paper's headline '88.2%–95.4%' is explicitly the researchers' first-rank rate from Table 3, not the automated score. The human evaluation is therefore the primary support for the central claim, and the reader also notes it is 'small and lacks statistical characterization.' I identified a sharper, more load-bearing problem: the independent developer survey—the only evaluation by actual open-source developers—shows markedly lower first-rank rates for CMO, especially on Expressiveness (38.6% vs the researchers' 95.4%) and Conciseness (6.8%, where CMO is last). If developer preferences are the appropriate standard, the abstract overstates the result by cherry-picking the most favorable evaluator. The lack of significance testing means we cannot even determine whether the developer preferences are statistically robust. This concern, if it lands, would not sink the paper—the developer survey still ranks CMO first on Rationality and Comprehensiveness—but it would force qualification of the headline and require statistical support. I thus agree with the reader's conditional verdict and recommend no change, while flagging a different weakest link than the reader's primary assumption.","tokens_in":22171,"tokens_out":10721,"duration_ms":100189,"concrete_test":"Ask the authors to release the per-commit rankings (or recompute from their data) underlying Tables 3 and 4, then run pairwise Wilcoxon signed-rank tests (and exact binomial tests on first-rank counts) comparing CMO to OMG, CMC, and the human-written message per metric per evaluator group, plus Cohen's kappa between the two researchers and between researchers and developers on the 10 overlapping commits. If CMO is not significantly preferred by the 22 Apache developers on Expressiveness, or if researcher–developer agreement is poor, the '88.2%–95.4%' headline should be replaced with metric- and population-specific claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central quantitative claim ('outperforming state-of-the-art CMG methods and human messages 88.2%–95.4% of the time') is computed from Table 3, where two researchers (likely the authors) ranked four messages per commit across 76 commits. The independent developer survey (Table 4; 22 Apache developers, 10 commits) yields far lower first-rank rates for CMO: Rationality 65.0% (vs 88.8% for researchers), Comprehensiveness 63.2% (vs 88.2%), Expressiveness 38.6% (vs 95.4%), and Conciseness 6.8% (vs 41.4% for researchers). If developer judgments are the more ecologically valid criterion, the claim of consistent superiority is overstated and metric-dependent. Moreover, no statistical significance tests (e.g., Wilcoxon signed-rank, exact binomial) are reported for either human-ranking experiment, and no inter-rater reliability (e.g., Cohen's kappa) is given for the two researchers, so 'significantly more Rational, Comprehensive, and Expressive' is not established. The human evaluation is the load-bearing evidence; its small size, lack of statistics, and researcher–developer disagreement undermine the abstract's headline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Commit Message Optimization (CMO), a search-based approach that starts from a human-written commit message and iteratively refines it with GPT-4, using automatically retrieved software contexts and two automated evaluators (a fine-tuned GPT-3.5 quality scorer and a retrieval-based similarity scorer) as the objective function. The authors conduct a qualitative study (RQ1) that identifies eight context themes human developers consider but that the OMG generation technique misses, then evaluate CMO against OMG, CMC, and human-written messages using both human rankings (two researchers on 76 commits; 22 Apache developers on 10 commits) and automated scores (Table 5). The abstract claims that CMO is 'significantly more Rational, Comprehensive, and Expressive' and outperforms baselines and human messages 88.2%-95.4% of the time.","tokens_in":22392,"tokens_out":9887,"duration_ms":87051,"significance":"The core idea—refining an existing human-written message via search-based LLM optimization rather than generating from scratch—is a timely and useful contribution to commit message research, and the RQ1 qualitative analysis of missed context themes is a concrete asset. The inclusion of an independent developer survey and an ablation study is also commendable. However, the strength of the evidence as presented does not support the headline claims: the automatic evaluation is circular (the outcome measure is the same combined score that is optimized), hyperparameters are selected without a clear holdout, the human evaluations lack significance tests and inter-rater reliability, and the independent developer results are substantially weaker than the researcher-derived numbers. If these issues are addressed, the paper could make a solid contribution; in its current form the claims outrun the evidence.","major_comments":[{"comment":"The Combined Evaluation Score (Eq. 2) is used both as the objective function in Algorithm 1 (the EVALUATE function) and as the outcome measure in Table 5, and its weights (SimCoeff and LLMCoeff) are computed from Pearson correlations on the 76-commit testing split that is later used for the automated comparison. Consequently, the Welch's t-test results in Table 5 compare methods on the very function that CMO is designed to maximize, and the evaluation metric itself is fitted to the test set. This circularity means the automatic gains are not independent evidence of higher message quality. Please provide a held-out evaluation with pre-fixed weights, or make the human evaluation the primary outcome with proper statistical analysis.","section":"Section 5.3, Eq. (2), Table 5"},{"comment":"Hyperparameters (p=5, temperature=1, step_limit=50, Equation 2) are selected via grid search on 10 randomly sampled commits with manual review by four authors, and the text does not state whether these 10 commits are drawn from the 76-commit testing split used for Table 5 and the human evaluations. If they are, the test set is used for model selection and the reported results are in-sample. Even if the 10 commits are from the training split, the Pearson-based weighting in Eq. (2) is estimated on the testing split (Section 5.3), so the evaluation measure itself is test-derived. Please clarify the provenance of the 10 commits and, if necessary, re-evaluate on a genuinely held-out set with a pre-specified metric.","section":"Section 5.5, Section 6"},{"comment":"The abstract's claim of outperforming baselines and human messages 88.2%-95.4% of the time is based solely on the two researchers' rankings (Table 3), not on the 22 independent Apache developers (Table 4), where CMO's first-rank rates are 65.0% (Rationality), 63.2% (Comprehensiveness), 38.6% (Expressiveness), and 6.8% (Conciseness). No significance tests (e.g., Wilcoxon signed-rank, exact binomial, Friedman) or inter-rater reliability (e.g., Cohen's kappa) are reported for either human evaluation. The independent developer results actually contradict the implication of superiority on Conciseness, where human messages were ranked first 45.5% of the time versus 6.8% for CMO. The claims must be restricted to the specific metric/researcher subset and supported by appropriate statistical tests.","section":"Section 6, Tables 3-4, Abstract"},{"comment":"Section 4.3 states that researchers ranked eight commit messages (CMO, baselines, and CMO variants), but Tables 3 and 4 report only four message types with column sums of 152 and 220, respectively. It is unclear whether the reported ranks are conditional on the four main messages or were recomputed from the full eight-message ranking after discarding variants. This ambiguity affects the interpretation of every headline percentage; please clarify the ranking procedure and report the actual distribution if full eight-message rankings were used.","section":"Section 4.3, Tables 3-4 vs Table 6"},{"comment":"The Conciseness component of the automated evaluation is degenerate: because the retrieval-based Sim Score has no significant correlation with human Conciseness judgments, Eq. (2) uses only the LLM Score for Conciseness, and Table 5 reports 4.0 for every method. The automated evaluation therefore provides no discrimination on the very dimension where the human evaluators found CMO to be worst, and the 'Total' scores in Table 5 are inflated by a constant. The paper should either develop a valid automated Conciseness measure or exclude Conciseness from the automated comparison.","section":"Section 5.3, Table 5"},{"comment":"The construction of the retrieval corpus is underspecified: the paper does not state whether the target commit's own diff or human-written message is excluded from the top-10 retrieved diffs, nor does it report the corpus size or its overlap with the 381-commit evaluation dataset. If a target commit appears in its own retrieval set, the Sim Score would reward staying close to the original human message, which would bias the optimization; if it is excluded, that should be stated explicitly. This is important because the retrieval-based evaluator is part of the objective function.","section":"Section 5.4"}],"minor_comments":[{"comment":"There are several typos and grammatical errors, including 'best best-performing hyper-parameters' in Section 4 and 'could not exhaustively tune the of these hyper-parameters' in Section 8; these should be corrected.","section":"Section 4, Section 8"},{"comment":"The sentence 'This reinforces the finding that CMO outperforms OMG/CMC and enhances human-written messages' overstates the developer results: for Expressiveness the first-rank rate is 38.6%, and for Conciseness CMO is last. Suggest tempering the language to 'directionally consistent for Rationality and Comprehensiveness'.","section":"Section 6, Table 4"},{"comment":"The constant Conciseness value of 4.0 for all methods is not commented on in the text; please add a sentence explaining that no statistical test was run for Conciseness because the automated score has zero variance.","section":"Table 5"},{"comment":"No link to code or data is provided, which would be helpful for reproducing the search-based optimization and the evaluator training; consider adding an artifact or repository link.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The two researchers who performed the primary human evaluation are not explicitly identified as the authors; if they are authors, the absence of disclosure and of inter-rater reliability statistics is a concern. The paper's title and abstract overstate the results relative to the independent developer survey, and this should be addressed in revision. The scope fit for a software engineering venue is appropriate, but the evaluation methodology needs substantial strengthening before the claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe genuinely new thing here is the formulation: instead of generating a commit message from a diff or completing a human prefix, CMO starts from the full human-written message and runs a search-based optimization over it, using GPT-4 plus automated evaluators as the objective. That framing is sensible and, as far as I can tell, new. The paper also does real work on RQ1: the eight context themes missed by OMG are concrete and plausible, and the authors are honest about which contexts they could and couldn't automate, with reasoning grounded in tool performance (e.g., issue-commit link recovery is too noisy).\n\nWhat it does well: two human evaluations, including 22 actual Apache developers on 10 commits, which is more than most CMG papers bother with. The ablation shows the search component matters: CMO-Search is significantly worse by their metric. The failure to beat CMC on BLEU/ROUGE is reported openly, and they correctly argue those metrics are unsuitable for this task.\n\nThe soft spots are real and load-bearing for the central quantitative claim. The combined evaluator in Equation 2 is the objective function in Algorithm 1 and also the outcome in Table 5, so the automatic comparison is circular: they optimized against that score and then reported winning on that score. The retrieval component's top-k was selected on the testing split of the evaluator, and hyperparameters (p, temperature, step_limit, equation) were chosen on 10 sampled commits without a holdout. That is test-set leakage, though its magnitude is hard to judge.\n\nThe 88.2%–95.4% headline comes from the two researchers' ranking over 76 commits. The 22 developers ranked CMO first 65% of the time on Rationality, 63.2% on Comprehensiveness, but only 38.6% on Expressiveness and 6.8% on Conciseness. No significance tests and no inter-rater reliability are reported for either human study. So 'significantly more Rational, Comprehensive, and Expressive' is not established from the human data, and the automated t-tests are undercut by the circularity.\n\nStill, the developer data does show CMO leading on two of four dimensions, which is evidence the approach has merit. The paper is a solid technical contribution to CMG; it just overstates the evidence. A serious editor should send it out, but reviewers should push for an independent holdout, proper significance testing of the human rankings, and release of the prompts and evaluators.","headline":"Novel optimization-over-generation idea with real developer evidence on two of four metrics, but the headline percentages are the researchers' rankings and the automated metric is both objective and outcome—a solid but overstated paper.","tokens_in":22983,"tokens_out":3167,"would_cite":true,"duration_ms":30838,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Improving human commit messages beats generating new ones","keywords":["commit message optimization","commit message generation","large language models","search-based optimization","software context","human-written commit message","automated evaluation","open-source Java projects"],"falsifier":"Blind human raters who have no access to the evaluator scores should rank CMO-optimized messages above the original human messages on Rationality, Comprehensiveness, and Expressiveness on a fresh sample of commits; if they do not, the claimed improvement is an artifact of optimizing the proxy.","tokens_in":21886,"feed_emoji":"💬","tokens_out":8249,"duration_ms":77298,"temperature":0.7,"pith_summary":"This paper tries to establish that the most effective way to produce a high-quality commit message is to start from the developer's own draft and optimize it, rather than to generate a message from the code change or complete a human-typed prefix. The proposed method, CMO, iteratively revises the human-written message one software context at a time, using an LLM to incorporate contexts that automated tools can retrieve and automated evaluators to score each candidate. Blind human evaluations by two researchers and 22 external developers both ranked CMO-optimized messages first on Rationality, Comprehensiveness, and Expressiveness more often than the original human messages, the OMG generation baseline, and the CMC completion baseline; the researcher rankings put CMO first on 88.2%-95.4% of commits for those three metrics. Conciseness is the exception: original human messages remained more concise. The paper concludes that selective, search-guided optimization of human input is a better paradigm for commit-message tooling than context-heavy generation from scratch.","feed_headline":"Improving human commit messages beats generating new ones","feed_subtitle":"A search-based LLM optimizer ranks first on three of four quality metrics against humans, generation, and completion baselines.","key_machinery":"The mechanism that carries the argument is the one-context-per-step priority-queue search over candidate commit messages. Starting from the human-written seed, CMO's UPDATE step asks the LLM to improve the current best candidate using exactly one new software context from the seven retrievable ones (file information, commit type, pull-request or issue titles, class and method summaries, enclosing statement blocks, invoked methods, and variable data types), and EVALUATE scores each candidate with the combined objective: the mean of a fine-tuned LLM metric classifier's Likert-style scores and a retrieval-based semantic-similarity score against high-quality human messages for nearby diffs. The priority queue keeps the highest-scoring candidate at the front; stopping is governed by a decaying improvement threshold and a step limit. This design lets the search include only contexts relevant to the specific commit and avoids flooding the LLM with irrelevant context.","core_discovery":"CMO is a search-based optimizer rather than a generator. It takes the human-written commit message as the initial candidate, evaluates it with an objective function that averages a fine-tuned LLM scoring model and a retrieval-based similarity score against high-quality human messages for semantically similar git diffs, and then runs a priority-queue search. At each step it dequeues the best-scoring candidate, has the LLM revise it by incorporating exactly one additional software context, scores the new candidate, and enqueues it; a decaying improvement threshold and a step limit terminate the loop. The paper claims this design outperforms both state-of-the-art CMG/CMC techniques and the original human messages on three of four evaluation dimensions in both researcher and external-developer blind evaluations, and that removing the search component (feeding all contexts at once) significantly degrades quality. It also claims the automated objective and the selective context injection are what make the difference, while traditional similarity metrics (BLEU, METEOR, ROUGE-L) do not capture the improvement.","pith_inferences":["The same optimize-the-human-draft-with-one-context-per-step loop should transfer to other human-written software artifacts, such as code review comments, issue reports, and changelog entries, where the author's implicit context is hard to retrieve automatically; the paper does not test this.","Because the evaluator's Rationality and Comprehensiveness accuracy is about 0.719, a testable next step is to replace the objective with a stronger evaluator and see whether CMO's measured gains increase.","Adding a conciseness penalty or an explicit completeness-brevity trade-off to the objective function could plausibly close the Conciseness gap the paper documents; this is a direct extension of its own results.","The paper's decision not to build tools for the harder contexts (adhoc maintenance goals, project requirements, personal mistakes) suggests CMO's ceiling is bounded by what the initial human draft already contains; if a developer's draft is poor, the optimizer may have little to build on."],"forward_implications":["If the evaluation results hold, commit-message tools should treat the developer's draft as an asset to refine, since CMO improved human messages on Rationality, Comprehensiveness, and Expressiveness in both researcher and external-developer rankings.","Conciseness is a remaining gap: the paper's results show original human messages ranked first on Conciseness more often than optimized ones, so optimization as currently rewarded trades brevity for completeness.","The ablation result implies that giving an LLM all available software contexts at once is worse than feeding one context per optimization step, which extends to prompt construction for code-intelligence tasks.","Standard text-similarity metrics (BLEU, METEOR, ROUGE-L) systematically favor prefix-completion over optimization and should not be used to judge commit-message quality.","The approach depends on the automated quality evaluators, so improvements in the evaluators would directly translate into better optimization targets."],"supporting_citations":[{"why":"Defines the OMG generation baseline and provides the 381-commit dataset with human scores and evaluation metrics.","marker":"[48]"},{"why":"Defines the CMC prefix-completion baseline that CMO extends and is compared against.","marker":"[32]"},{"why":"Establishes the 'What'/'Why' quality view of commit messages used for the quality criteria and for filtering the retrieval corpus.","marker":"[72]"},{"why":"Supplies the classifier that decides whether a commit message contains 'What'/'Why' information, used to select high-quality messages.","marker":"[47]"},{"why":"Provides the method-level code summarization technique used to summarize invoked methods.","marker":"[35]"},{"why":"Supplies the code-change embedding model used to retrieve semantically similar git diffs.","marker":"[50]"},{"why":"Supplies the sentence embedding model used to compute semantic similarity between generated and high-quality human messages.","marker":"[20]"},{"why":"Motivates the LLM-as-optimizer search pattern that CMO adapts.","marker":"[53]"}],"fun_headline_variants":["Optimizing human commit messages beats generating them","Search-based commit message optimizer beats generation","Refine human commit messages, don't generate new ones","CMO outperforms commit message generation via search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated objective function—whose Rationality and Comprehensiveness classifier agrees with humans only about 72% of the time, and whose retrieval-based similarity score shows no significant correlation with human Conciseness—still rewards the right qualities closely enough that optimizing it genuinely improves human message quality.","fun_headline_variants_meta":{"raw":{"variants":["Optimizing human commit messages beats generating them","Search-based commit message optimizer beats generation","Refine human commit messages, don't generate new ones","CMO outperforms commit message generation via search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1470,"prompt_tokens":886,"completion_tokens":584,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":526}},"tokens_in":502,"tokens_out":584,"duration_ms":6669,"temperature":1.0,"reasoning_tokens":526,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:35:48.571984+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Blind human raters who have no access to the evaluator scores should rank CMO-optimized messages above the original human messages on Rationality, Comprehensiveness, and Expressiveness on a fresh sample of commits; if they do not, the claimed improvement is an artifact of optimizing the proxy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the OMG generation baseline and provides the 381-commit dataset with human scores and evaluation metrics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CMC prefix-completion baseline that CMO extends and is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the classifier that decides whether a commit message contains 'What'/'Why' information, used to select high-quality messages."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the code-change embedding model used to retrieve semantically similar git diffs."},{"cited_title":"sentence-transformers/all-mpnet-base-v2","cited_arxiv_id":null,"evidence_quote":"Supplies the sentence embedding model used to compute semantic similarity between generated and high-quality human messages."}],"review_version":1}