{"id":"733da306-27d0-439c-a042-1fa2517760bc","arxiv_id":"2411.19099","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A random forest learning-to-rank model using past co-change history and code features ranks co-changed methods with NDCG@5 of 0.84, but its edge over a historical frequency baseline is only 4.7%.","lead":"This paper trains ranking models to suggest which functions in a Java codebase will likely be changed together, using past pull-request history and code features. A random forest model beat simpler baselines, though the gain over simply ranking by past co-change frequency was small.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 1's exclusion of zero-label queries inflates the reported NDCG@5; evaluation should be redone on the full query set.","rationale":"The paper proposes a reasonable learning-to-rank pipeline for co-changed method identification, with a large 150-project dataset, a replication package, and internal comparisons across seven LtR models. The strongest evidence for the approach is the consistent internal ranking: Random Forest outperforms the other LtR models, and the Wilcoxon tests support the significance of the differences. However, the evaluation protocol in Algorithm 1 explicitly removes queries with no future co-changes. This is a form of outcome-based sample selection: the test set is conditioned on the exact future event the model is supposed to predict. For a maintenance tool, a developer may query any method, and methods with no co-changes are not edge cases; they are a large and important part of the real workload. Excluding them means the reported NDCG@5 reflects only the easier subset of queries, so the central numeric claim of a mean NDCG@5 of 0.84 does not generalize to the stated use case. The reader's weakest assumption identifies the same issue, and I agree with that assessment. The relative comparison against baselines is less affected because all methods are filtered identically, which is why the verdict should remain CONDITIONAL rather than moving to REJECT: the core method and internal comparisons are likely sound, but the headline performance and practical utility need to be re-estimated on the full query distribution.","tokens_in":18086,"tokens_out":7531,"duration_ms":74144,"concrete_test":"Recompute the RQ1 and RQ2 results without the Algorithm 1 line 21 exclusion: include every query method from Mvalid, define NDCG@5 as 0 for queries with zero positive labels (or equivalently report Precision@5 with empty relevant sets scored as 0), and report the fraction of queries removed by the exclusion. Compare the resulting mean NDCG@5 for the Random Forest model and for the support-ranking baseline. If the fraction is substantial and the RF mean drops well below 0.84, or if the RF advantage over support ranking shrinks below significance, the headline performance claim is overstated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is in Algorithm 1, line 21: ranking lists where all labels are zero are excluded from the dataset because 'results will be 0/0 and thus not meaningful.' This turns the evaluation from 'rank co-changed methods for any method a developer queries' into 'rank co-changed methods for queries known to have at least one future co-change.' In real use, many query methods will have no co-changed methods in the next six months; the model would still output a ranking, and that ranking would be entirely wrong. By dropping those queries, the reported mean NDCG@5 of 0.84 and the 4.7%–573.5% margins are computed on an easier, outcome-conditioned subset. The paper mentions in Section 6 that assigning relevance from history 'could bias the results towards frequently modified methods,' but it does not discuss the exclusion itself as a threat. The same filter applies to all baselines, so the relative ranking may survive, but the central practical claim—that the tool achieves a mean NDCG@5 of 0.84 for developers—is not supported for arbitrary queries.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a learning-to-rank (LtR) approach for identifying co-changed methods at the pull-request level. It extracts ten features from source-code structure and version-control history, labels method pairs by the number of future co-changes in a six-month window, and trains seven LtR models on 150 open-source Java projects. The Random Forest model is reported to achieve the best NDCG@5 (mean 0.84) and to outperform five baselines (support ranking, file proximity, code clone detection, FCP2Vec, and StarCoder 2). The paper also analyzes feature importance and studies how the training and testing label periods affect performance, concluding that models should be retrained roughly every two months.","tokens_in":18480,"tokens_out":5430,"duration_ms":52035,"significance":"If the reported evaluation is valid, the paper would be a useful empirical contribution: it is large-scale (150 projects, 634,216 pull requests), uses a non-overlapping temporal split, provides a replication package, and compares multiple ranking models and baselines. The permutation-importance analysis and the study of labeling periods are also constructive. However, two evaluation choices compromise the central quantitative claims: the exclusion of queries with no future co-changes in Algorithm 1, and the mismatched project sets for the StarCoder 2 baseline. These issues are fixable by re-analysis, so the underlying idea remains defensible, but the current numbers overstate the practical value of the tool.","major_comments":[{"comment":"The dataset creation excludes all ranking lists whose labels are all zero, with the comment that NDCG would be 0/0 and thus not meaningful. This is a selection on the outcome: only queries that are known to have at least one future co-change are scored. In real use, a developer can query any method, and many methods will have no co-change in the next six months; NDCG@5 for those queries should be defined (e.g., as 0) and included. Without that, the reported mean NDCG@5 of 0.84 (Section 4.1.3) and the margins in Table 5 are conditional on the query having at least one relevant item. Please re-run the evaluation on the full query set with an explicit convention for the undefined denominator, and report the fraction of queries excluded by the current filter.","section":"Section 3.5, Algorithm 1 (line 21)"},{"comment":"StarCoder 2 was applied to only 45 randomly selected projects while the RF model was evaluated on all 150 projects. Table 5 compares aggregate means over different project sets, so the reported margin over StarCoder 2 confounds model quality with project selection. Please report a paired comparison on the same 45 projects, or run StarCoder 2 on all 150 projects, and state the number of projects underlying each row of Table 5.","section":"Section 4.2.2, Table 5"},{"comment":"The most important feature, 'Number of co-changes,' is exactly the quantity used by the support-ranking baseline and is closely related to the label (future co-change count). The feature and label periods are sequential, so this is not direct label leakage, but the RQ2 comparison to support ranking (4.7% at NDCG@5) measures only the marginal value of the remaining features. Because the paper claims a learning-to-rank advantage, please add an ablation that removes the number-of-co-changes feature or uses only static features, and discuss how the 0.38 permutation importance bears on the interpretation of the RQ2 results. The threat stated in Section 6 about historical relevance biasing toward frequently modified methods applies directly here.","section":"Section 3.2, Section 4.3, Table 6"}],"minor_comments":[{"comment":"The text says CodeBERT produces a vector with 764 scalar values, but the standard CodeBERT base model produces 768-dimensional embeddings; please verify and correct this number.","section":"Section 3.2, Semantic similarity"},{"comment":"The margin over code-clone detection is reported as 537.5% in the abstract but as 573.5% in Section 4.2.3 and the conclusion; Table 5's NDCG@5 values imply 573.5%, so the abstract appears inconsistent.","section":"Abstract, Section 4.2.3, Conclusion"},{"comment":"The sentence 'The RF model achieves the highest performance of 0.91 NDCG@5' should specify whether 0.91 is the median, a per-project maximum, or the mean; Table 4 lists the mean NDCG@5 as 0.8394 and the median as 0.9106.","section":"Section 4.1.3"},{"comment":"The figure caption should state the axis labels explicitly, since the current text refers to 'days of testing data' but the figure itself is not self-contained.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical study, but the central evaluation needs to be redone: the exclusion of zero-label queries conditions the reported NDCG on an easier subset, and the StarCoder 2 comparison uses different project sets. The direction is publishable if the authors can re-run the experiments on the full query set and provide paired baseline comparisons; I would not reject on the current evidence, but the headline numbers cannot be accepted as they stand."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: the paper's headline number is not trustworthy. Algorithm 1 line 21 removes every query where all method-pair labels are zero in the test window. That means the model is never evaluated on a method with no co-change in the next six months. In practice a developer could query any method; many will have no co-change, and the model would recommend something, so the failure would count against it. Dropping those queries inflates NDCG@5 to 0.84 and makes the baselines comparison lopsided. The same filter applies to all systems, so the relative ordering likely survives, but the absolute claim is unsupported.\n\nWhat is new: using learning-to-rank at the pull-request level, combining historical co-change counts with static features (path, authors, CodeBERT semantic similarity, clone score, etc.) in a single model. That combination has not been tried for method-level co-change prediction, and the scale is respectable: 150 Java projects, 634k PRs, with a replication package. The paper is surprisingly candid about its own feature analysis: number-of-co-changes dominates (permutation importance 0.38), path similarity is the only other feature with any real weight, and clone and semantic similarity are almost useless. The decay result (performance drops after 60 days of test data) is a practical insight worth keeping.\n\nThe soft spots, in proportion: (1) the selection bias above is the major one. (2) StarCoder2 was evaluated on 45 projects while the proposed model got 150; the paper discloses this, but then it is not a fair comparison. (3) The advantage over support ranking is only 4.7% NDCG@5, and support ranking is essentially the same as the model's most important feature (historical co-change count). That is not circular—labels come from the future—but it does mean the static features add little, and the paper doesn't confront this directly.\n\nThe paper is written by people who know the area and did the work carefully; the internal comparisons between LtR models and the feature analysis are probably sound. The evaluation design, though, needs to be redone on the full query set, or the authors need to argue why zero-label queries are irrelevant. I'd send this to peer review, not desk reject, because the core idea and dataset have value. But I'd push for major revision and re-analysis before acceptance.","headline":"A solid LtR application with a large dataset, undermined by an evaluation filter that drops all zero-label queries, so the headline NDCG is optimistic.","tokens_in":18797,"tokens_out":2522,"would_cite":true,"duration_ms":22901,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A learning-to-rank model trained on pull-request histories ranks co-changed methods well enough to beat five baselines, reaching a mean NDCG@5 of 0.84 across 150 Java projects.","keywords":["co-change","evolutionary coupling","learning to rank","random forest","pull requests","software maintenance","repository mining","NDCG"],"falsifier":"Re-run evaluation without excluding zero-label queries: take every query method from a held-out period, rank all candidates, and compute NDCG@5; if the ranking on those queries is near chance, the reported performance does not extend to the arbitrary queries a developer would actually issue.","tokens_in":17928,"feed_emoji":"🔧","tokens_out":6654,"duration_ms":56330,"temperature":0.7,"pith_summary":"This paper tries to establish that the methods a developer is likely to need to edit together can be predicted and ranked before the edit is made, using only source-code properties and past change history. It proposes a learning-to-rank approach at the pull-request level, where a pull request rather than a single commit counts as one logical change, and reports that a Random Forest model reaches a mean NDCG@5 of 0.84 across 150 Java projects. If the claim holds, a developer changing one method can be shown a short ranked list of other methods likely to need the same edit, reducing missed dependencies and false positives. The paper also claims that 90 days of history is enough to train the model and that it needs retraining about every two months.","feed_headline":"Co-change ranker beats five baselines on Java projects","feed_subtitle":"History-based ranking reaches 0.84 NDCG@5 across 150 projects and flags when retraining is needed.","key_machinery":"The central machinery is a learning-to-rank pipeline that turns each query-method and candidate-method pair into a feature vector and trains a ranker over lists of candidates. The Random Forest model is the mechanism that carries the result; with features such as historical co-change count, author similarity, path similarity, code dependency, hierarchy similarity, clone similarity, and argument type and name similarity, and with labels equal to co-change counts in the next six months, it orders candidates by predicted relevance. The pull-request level is the unit of analysis, and NDCG@k is the evaluation device used to measure how well the top of the ranked list matches the true co-change labels.","core_discovery":"The paper claims that co-change relationships between methods can be predicted and ranked by a learning-to-rank model. Each query method is paired with every other non-test method in the same repository; ten features describe the pair, from historical co-change count and shared authors to file-path similarity, code dependency, inheritance, clone similarity, and argument and signature similarity. The relevance label is the number of future pull-request co-changes in a six-month window. Trained on 150 open-source Java projects, a Random Forest model ranks candidates with a mean NDCG@5 of 0.84 and a median of 0.91, outperforming five baselines, with the historical co-change count by far the strongest feature.","pith_inferences":["Editorial inference: since the co-change count feature dominates with permutation importance 0.38, much of the top-5 signal is the same signal a developer would get from counting past co-edits, and the added value of the machine-learning model lies mainly in path and author similarity.","Editorial inference: the low importance of semantic and clone similarity suggests the ranking is not finding new conceptual couplings, so a testable next step is to fine-tune a code model on pull-request co-change data instead of using off-the-shelf cosine similarity.","Editorial inference: a bi-monthly retraining rule follows directly from the reported 60-day performance decline, and it can be tested prospectively by comparing a model retrained every 60 days against one retrained every 180 days on live pull-request streams.","Editorial inference: because labels are counted at pull-request level, the method may favour changes that developers deliberately bundle and could miss co-changes split across separate pull requests; measuring against commit-level labels would reveal how much of the gain comes from the pull-request unit itself."],"forward_implications":["A developer editing one method can inspect a short ranked list of the methods most likely to need the same edit, reducing missed dependencies during maintenance.","Pull-request-level analysis captures changes spread over multiple commits that commit-level co-change detection would miss.","Ninety days of history is enough to train a usable model, and longer histories do not improve its performance significantly.","Prediction quality degrades after about 60 days of unretrained use, so the model should be retrained roughly every two months.","The best results appear on medium-size and younger Java projects; long-lived projects see significantly lower ranking quality."],"supporting_citations":[{"why":"Supplies the Random Forest model, which the paper selects as the best-performing ranker.","marker":"[5]"},{"why":"Supplies the NiCad clone detector used both for the clone feature and for the clone baseline.","marker":"[9]"},{"why":"Supplies CodeBERT, the code-embedding model used to compute the semantic similarity feature.","marker":"[10]"},{"why":"Supplies FinerGit, the method-level history miner that provides the historical features and pull-request-level labels.","marker":"[14]"},{"why":"Supplies StarCoder 2, the large language model used as one of the five baselines.","marker":"[18]"},{"why":"Supplies the HistoRank baselines, support ranking and file proximity ranking, that the approach must beat.","marker":"[21]"},{"why":"Supplies FCP2Vec, the file-path vectorization baseline that the approach outperforms.","marker":"[1]"},{"why":"Supplies the Wilcoxon signed-rank test used to establish statistical significance.","marker":"[37]"}],"fun_headline_variants":["Co-change ranker beats five baselines on 150 Java projects","Random Forest ranks co-changed methods with NDCG@5 of 0.84","Pull-request co-change model needs retraining every two months","Historical co-change count is the top feature for method ranking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim relies on treating future co-change frequency in a fixed six-month window as the ground truth for relevance, and it removes any query whose methods never co-change in that window before measuring NDCG.","fun_headline_variants_meta":{"raw":{"variants":["Co-change ranker beats five baselines on 150 Java projects","Random Forest ranks co-changed methods with NDCG@5 of 0.84","Pull-request co-change model needs retraining every two months","Historical co-change count is the top feature for method ranking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1485,"prompt_tokens":922,"completion_tokens":563,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":486}},"tokens_in":538,"tokens_out":563,"duration_ms":8206,"temperature":1.0,"reasoning_tokens":486,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:32:18.626989+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run evaluation without excluding zero-label queries: take every query method from a held-out period, rank all candidates, and compute NDCG@5; if the ranking on those queries is near chance, the reported performance does not extend to the arbitrary queries a developer would actually issue.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Random Forest model, which the paper selects as the best-performing ranker."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NiCad clone detector used both for the clone feature and for the clone baseline."},{"cited_title":"Roy, and Kevin A","cited_arxiv_id":null,"evidence_quote":"Supplies the HistoRank baselines, support ranking and file proximity ranking, that the approach must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies FCP2Vec, the file-path vectorization baseline that the approach outperforms."}],"review_version":1}