{"id":"63b4841d-ab11-402c-b399-f17612262047","arxiv_id":"2607.23610","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Framing one-to-many issue–commit recovery as issue-centric learning-to-rank with iterative pick–remove–renormalize selection substantially beats pairwise and LLM baselines on a new six-project dataset.","lead":"LinkRank recovers multi-commit fixes for software issues by ranking candidate commits per issue and iteratively selecting them, instead of scoring pairs alone. It matters because incomplete issue–commit links hurt maintenance, debugging, and impact analysis in real repositories.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"RDS candidate pools are anchored on ground-truth commit timestamps, and the \">98% coverage\" validity check is tautological — coverage is guaranteed by construction. Part of the 25-point F1 gap may reflect oracle-centered retrieval rather than ranking formulation.","rationale":"The reader correctly identified the oracle-informed RDS pool as the weakest assumption; I agree and sharpen it. The additional observation is that the paper's own mitigation for this threat (§6.2's 98% coverage verification) is vacuous by construction, so the realism claim currently has no supporting evidence at all — the reader noted the anchoring but not the tautology. The concern is load-bearing because the central claim is comparative and methodological (\"formulation beats capacity\"), and the strongest single feature group (metadata/temporal) interacts directly with how the pool is temporally centered. It is not fatal: all baselines face identical pools, significance is consistent across folds, and the ablation shows textual features contribute independently. So this warrants continued CONDITIONAL rather than REJECT, with the condition being a blind-pool replication. Secondary issues (abstract quoting 74.54%/68.84% vs body's 73.17%/65.94%; LinkAnchor altered to an 80-commit cap on a different model) are real but less central, and the reader already discounted for them.","tokens_in":21994,"tokens_out":1704,"duration_ms":43320,"concrete_test":"Rebuild candidate pools for all six projects using only information available at recovery time: window = [issue creation date − δ, issue close date + δ] (or creation + T where T is estimated from the historical issue-resolution-time distribution), never touching C_i timestamps. Rerun the identical pipeline (same folds, features, Optuna budget, Known-K) and compare per-project F1 against Table 4. If average Known-K F1 drops more than ~8 points, or the margin over LinkAnchor's 48.39% shrinks below ~10 points, the headline claim holds only under oracle-informed retrieval and should be restated as such. A supplementary run with no temporal window (whole-repo pool) on one project would bound the worst case.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1.4 defines each issue's candidate pool as all commits in [t_min(C_i) − δ, t_max(C_i) + δ], where t_min and t_max are the earliest/latest timestamps of the ground-truth linked commits. This is the single most load-bearing design choice for the central claim that \"set-recovery framing, not model capacity\" drives the 73.17% vs 48.39% F1 gap, for two compounding reasons.\n\nFirst, the pool is centered on the true resolution period using information unavailable at recovery time. A deployed system knows the issue creation/close dates but not the linked commits' timestamps. Because negatives are drawn from the same temporally concentrated window as positives, and because LinkRank's metadata group (f_time, f_dir relative to issue creation) is its largest ablation contributor (49.79% → 76.38% on PyTorch when metadata is added, Table 7), the temporal features are evaluated against negatives that are guaranteed to be temporally plausible — a harder-looking but actually pool-shaped setting whose difficulty profile a blind pool (e.g., windowed from issue dates, or the whole repo) may not reproduce.\n\nSecond, the threat-to-validity mitigation in §6.2 — \"verifying that more than 98% of ground-truth commits fall within it\" — is circular: since the window edges are derived from C_i itself, 100% of ground-truth commits fall within it by construction, regardless of δ. The stated check cannot detect anything, so the realism argument for RDS currently rests on no evidence.\n\nThis does not mean the direction of the result is wrong — the ranking metrics (MRR 87.19% vs 57.49%) are computed within the same pools for all methods, so the comparison is internally fair. The risk is external: whether the magnitude of the gain survives a blind candidate generator.","agreement_with_reader":"agree"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper reframes one-to-many issue–commit traceability as an issue-centric set-recovery problem rather than independent pairwise classification, and operationalizes this as LinkRank: LambdaMART ranking over 17 lexical/retrieval/semantic/metadata features within per-issue candidate pools, followed by an iterative pick–remove–renormalize selector with Known-K (oracle), ABS, and REL stopping rules. The authors construct a new dataset of 3,103 issues and 7,688 commits from six GitHub repositories (K=1..7), evaluate with 5-fold stratified CV, and report average Known-K F1 of 73.17% vs 48.39% for the strongest baseline (LinkAnchor), with Unknown-K F1 of 65.94% (ABS) and 62.19% (REL). The evaluation includes ranking metrics (MRR/NDCG@K/P@K), a feature ablation, threshold sensitivity analysis, a cost comparison, and Wilcoxon/Cliff's δ significance testing.","tokens_in":22366,"tokens_out":2045,"duration_ms":73514,"significance":"If the central comparison holds up under a blind candidate-pool construction, this is a useful contribution: the set-recovery reframing is clearly articulated, the dataset (3,103 issues / 7,688 commits, preserving raw artifacts and full per-issue commit sets, released via a public GitHub repository) fills a real gap left by tokenized releases such as EALink's, and the evaluation is unusually thorough for this literature — per-project tables with standard deviations, feature ablation showing metadata contributes the largest single gain (Table 7), threshold sensitivity plateaus (Fig. 4), a runtime comparison (Fig. 3), and effect-size reporting. The Unknown-K results with automatic stopping are the practically relevant numbers and remain strong. However, the headline 25-point F1 gap currently rests on a candidate-pool construction that uses ground-truth commit timestamps, so the magnitude of the improvement over baselines is not yet established in a deployment-realistic retrieval setting; the qualitative conclusion (issue-centric ranking beats pairwise classification) is likely robust, but this needs to be demonstrated rather than assumed.","major_comments":[{"comment":"Section 3.1.4, candidate pool definition P_i = {c : t_c in [t_min(C_i) - delta, t_max(C_i) + delta]}: the pool window is derived from the earliest and latest timestamps of the ground-truth linked commits, which is oracle information unavailable at recovery time. A deployed system knows the issue creation/close dates, not the linked commits' timestamps. This interacts directly with the paper's strongest feature signal: Table 7 shows metadata (f_time, f_dir relative to issue creation) contributes the largest ablation gain (49.79% -> 76.38% on PyTorch), and those temporal features are evaluated against negatives that are guaranteed to be temporally plausible because the pool is centered on the true resolution period. Part of the 73.17% vs 48.39% gap may therefore reflect pool geometry rather than the ranking formulation. The fix is within scope: re-run the main tables (4-6) with a blind RDS","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a competent empirical SE paper that actually moves the one-to-many issue–commit problem. The useful idea is not LambdaMART or SBERT—those are stock—but treating recovery as issue-level set recovery: rank inside a pool, then pick–remove–renormalize with Known-K / ABS / REL stops. They also ship a six-project dataset that keeps full commit multiplicity and large imbalanced pools. Under that protocol the light ranker clearly beats EALink, MPLinker, EasyLink, and an adapted LinkAnchor on set F1 and on MRR/NDCG/P@K, with ablation and cost numbers that make the story coherent.\n\nWhat they do well: stratified 5-fold CV, per-project means±std, Wilcoxon + Cliff’s δ, ranking metrics before selection, feature ablation (metadata is the big lift), threshold sweeps, and a runtime comparison. The central claim—that formulation beats raw model capacity here—is supported inside their setup. Writing is clear; related work is fair about pairwise assumptions in prior work.\n\nSoft spots, in proportion. The load-bearing design choice is RDS pools built from t_min/t_max of the ground-truth linked commits plus δ=365 days. A deployed system does not know those commit times; it knows issue dates. The “>98% of GT commits fall in the window” check is tautological by construction, so it does not validate realism. Comparisons stay internally fair (same pools for everyone), and ranking quality is strong, but part of the ~25-point Known-K gap may be pool shape rather than framing alone. Minor: abstract averages (74.54 / 68.84 / 67.02) do not match body Tables 4–5 (73.17 / 65.94 / 62.19). LinkAnchor is also softened (80-commit cap, different model), so the LLM baseline is not a full-strength reproduction. None of that kills the direction of the result.\n\nWho it’s for: MSR / traceability people who care about multi-commit links and dataset construction. Worth a serious referee. I’d engage—cite the framing and dataset if I work in this area—and I’d want one revision experiment with pools centered on issue open/close (or whole-repo retrieval) before trusting the magnitude of the gain in the wild.\n\nRecommendation: send to peer review; ask for blind-pool sensitivity and abstract/body number cleanup.","headline":"Solid MSR methods paper: set-recovery framing plus a real multi-commit dataset beat pairwise/LLM baselines under their protocol, but RDS pools are GT-time-anchored so the gap size needs a blind-pool check.","tokens_in":23024,"tokens_out":620,"would_cite":true,"duration_ms":18888,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"One-to-many issue–commit traceability works better as issue-level ranking and iterative set selection than as independent pairwise classification.","keywords":["issue–commit traceability","traceability link recovery","learning to rank","software maintenance","mining software repositories","one-to-many linking","iterative selection"],"falsifier":"Rebuild candidate pools without using ground-truth link timestamps (for example, from issue open time alone or a fixed repo-wide window), keep the same ranker and stopping rules, and check whether LinkRank still beats the strongest pairwise and LLM baselines on set-level F1 across the six projects.","tokens_in":22701,"feed_emoji":"🔗","tokens_out":896,"duration_ms":27632,"temperature":0.7,"pith_summary":"Software projects often fix a single issue with several commits, yet most automated link-recovery tools still treat each issue–commit pair in isolation and assume one commit per issue. This paper argues that the right task is set recovery: for each issue, rank a pool of candidate commits and iteratively pick the ones that jointly explain the fix. It introduces LinkRank, which scores candidates with a learning-to-rank model, then repeatedly selects the top commit, removes it, renormalizes the remaining scores, and stops by either a known count or an automatic threshold. On a new six-project dataset with large, imbalanced candidate pools, this formulation beats stronger neural and language-model baselines under both oracle and automatic stopping. A sympathetic reader cares because incomplete links break maintenance, debugging, and impact analysis; reframing the problem, not just scaling the model, is what closes the gap.","feed_headline":"Ranking beats pairing for multi-commit bug fixes","feed_subtitle":"Treating each issue as a set-recovery query lifts F1 well above one-to-one baselines on six projects.","key_machinery":"LinkRank: treat each issue as a query over a candidate commit pool, score with a learning-to-rank model, then iteratively pick the highest-scoring commit, remove it, renormalize remaining scores, and stop via Known-K, absolute threshold (ABS), or relative threshold (REL).","core_discovery":"The authors claim that one-to-many issue–commit recovery is fundamentally an issue-centric set-recovery problem, not pairwise binary classification. When candidates are ranked jointly inside each issue’s pool and selected with an iterative pick–remove–renormalize loop, a lightweight ranker recovers complete commit sets more accurately than heavier methods built for one-to-one linking, under both Known-K and automatic Unknown-K stopping.","pith_inferences":["The same pick–remove–renormalize pattern may transfer to other one-to-many software links (pull request to issues, requirement to tests) where pairwise classifiers currently dominate.","If blind candidate generation erodes the gains, hybrid pipelines that first filter with cheap retrieval then rank would become the practical deployment path the authors gesture toward in future work.","Explainable set recovery—short rationales for why each selected commit belongs—could be the product surface developers actually trust, beyond F1 alone."],"forward_implications":["Traceability tools should output a variable-size commit set per issue, not a single best commit or independent pair labels.","Ranking quality inside an issue-specific pool matters more for multi-commit recovery than raw model size or pairwise classifier accuracy.","Automatic ABS/REL stopping can approach oracle Known-K performance without knowing how many commits belong to an issue.","New benchmarks should preserve full per-issue commit multiplicity and large realistic candidate pools, not only balanced pair labels.","Metadata such as time proximity and file overlap should be first-class features alongside text and embedding similarity."],"fun_headline_variants":["Issue-centric ranking recovers multi-commit fixes better than pairs","LinkRank turns one-to-many traceability into iterative set recovery","Pick-remove-renormalize ranking lifts multi-commit link F1","Joint ranking per issue beats pairwise commit linking","Known-K and Unknown-K both favor rank-then-select over pairing"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"Candidate pools are built from a time window centered on the earliest and latest ground-truth linked commits for that issue, so the search space is already focused on the real resolution period rather than built blindly from the full history.","fun_headline_variants_meta":{"raw":{"variants":["Issue-centric ranking recovers multi-commit fixes better than pairs","LinkRank turns one-to-many traceability into iterative set recovery","Pick-remove-renormalize ranking lifts multi-commit link F1","Joint ranking per issue beats pairwise commit linking","Known-K and Unknown-K both favor rank-then-select over pairing"]},"model":"grok-4.5","effort":"low","cost_usd":0.004243,"raw_usage":{"total_tokens":1379,"prompt_tokens":899,"num_sources_used":0,"completion_tokens":74,"cost_in_usd_ticks":42428000,"prompt_tokens_details":{"text_tokens":899,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":406,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":899,"tokens_out":74,"duration_ms":7591,"temperature":1.0,"reasoning_tokens":406,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T17:40:44.802129+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Rebuild candidate pools without using ground-truth link timestamps (for example, from issue open time alone or a fixed repo-wide window), keep the same ranker and stopping rules, and check whether LinkRank still beats the strongest pairwise and LLM baselines on set-level F1 across the six projects.","supporting_citations":[],"review_version":1}