{"id":"14fe58d2-92dc-4958-8b88-78538ff07721","arxiv_id":"2412.01072","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":13,"one_line_summary":"Federated fine-tuning of six code LLMs on private bug-fix data improves program repair to near-centralized levels, with negligible impact from heterogeneous code.","lead":"This paper tests whether federated learning, a privacy-preserving training method, can let companies jointly fine-tune code-fixing AI models without sharing their private code. It finds the federated approach fixes bugs nearly as well as training on all data together, and that differing code styles across companies barely hurt performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning uses C++ TutorCode while evaluation is on Java EvalRepair-Java; the paper never justifies this cross-language transfer, so the claimed program-repair gains may reflect cross-lingual generalization rather than federated-learning-based repair learning.","rationale":"The reader's weakest_assumption is exactly the cross-language mismatch between the C++ fine-tuning dataset (TutorCode) and the Java evaluation benchmark (EvalRepair-Java). This is the most load-bearing concern because it threatens the external validity of every result in the paper: the absolute gains of federated fine-tuning, the comparison with centralized fine-tuning, and the heterogeneity findings. The internal comparisons among FL, local, and central are still meaningful for C++-trained models evaluated on Java, but the paper's broader claim about 'program repair capabilities' requires the training and evaluation languages to match or be explicitly validated as transferable. The paper never acknowledges this gap, and Section 7 omits it from the threats to validity. The concrete test I propose would settle the issue by either reproducing the results in a same-language setting or demonstrating that the observed Java improvements are indeed cross-lingual transfer, in which case the paper would need to be reframed. The reader's CONDITIONAL verdict remains appropriate, and my analysis does not change it; hence I set verdict_should_be to UNCHANGED.","tokens_in":40848,"tokens_out":6305,"duration_ms":52831,"concrete_test":"Re-run the RQ1 comparison (FL vs Local vs Central vs Original) with a Java bug-fix fine-tuning dataset of similar size (e.g., a Java subset of Defects4J or a Java tutor dataset) and evaluate on EvalRepair-Java; if the same ordering and improvement magnitudes appear, the cross-language transfer is not the driver. Alternatively, evaluate the current C++-fine-tuned models on a C++ repair benchmark (e.g., ManyBugs or IntroClass); if they show no improvement over the original models on C++ programs, then the Java gains are cross-lingual and the paper must reframe its claims and validate the transfer explicitly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that federated fine-tuning enhances program repair rests on fine-tuning LLMs on the private TutorCode dataset, which contains 1,239 buggy C++ programs (Sec 4.2), and evaluating on EvalRepair-Java, a Java benchmark (Sec 4.2). The paper provides no argument, pilot, or control experiment establishing that repair capabilities learned on C++ transfer to Java. This matters for two reasons. First, the headline improvement of FL over the original models (e.g., +16.57% Top@10 on Mistral-7B, Table 2) could be due to generic code-generation improvements from any fine-tuning, not to learning language-agnostic repair patterns; without a same-language baseline the causal attribution to 'program repair enhancement' is unfounded. Second, the key heterogeneity finding (RQ2) is constructed from C++ code features (coding style, complexity, embedding); the claim that 'the impact of heterogeneous code on LLM fine-tuning is negligible' may not hold when the fine-tuning data and evaluation share a language. The related-work discussion of cross-lingual repair (Ahmed et al., CIRCLE) concerns multi-lingual training, not this single-source-to-single-target setup. Section 7 (Threats to Validity) discusses feature coverage but never mentions the language mismatch, which is a notable omission. The reader's weakest_assumption correctly identifies this as the most load-bearing threat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of federated fine-tuning of code LLMs for automated program repair. It fine-tunes six LLMs (CodeLlama-13B, CodeLlama-7B, DeepSeekCoder-7B, WizardCoder-15B, Mistral-7B, CodeQwen-7B) using QLoRA adapters on a private industrial dataset (TutorCode, 1,239 buggy C++ programs) and evaluates the resulting models on the Java-based EvalRepair-Java benchmark. Three research questions are addressed: whether federated fine-tuning improves repair over original, local, and centralized fine-tuning (RQ1); whether code-feature-based heterogeneity (coding style, complexity, embedding) affects repair performance (RQ2); and how different federated algorithms (FedAvg, FedProx, FedSWA, FedOPT, pFedMe) compare (RQ3). The headline findings are that federated fine-tuning outperforms local fine-tuning and often rivals or exceeds centralized fine-tuning, that heterogeneous Non-IID data has negligible impact, and that FedAvg is the most robust algorithm overall.","tokens_in":41225,"tokens_out":4731,"duration_ms":43173,"significance":"If the findings hold, the paper would be a valuable early contribution on privacy-preserving federated fine-tuning for a generative software engineering task, with practical implications for industrial collaboration on proprietary code. The study has notable strengths: it uses a genuinely private industrial dataset (TutorCode), evaluates with an enhanced benchmark (EvalRepair-Java) that expands test cases to 583 per problem to mitigate patch overfitting, covers six LLMs of diverse architectures, and provides a public artifact. However, the central comparisons are currently undermined by a cross-language train/test mismatch (C++ fine-tuning vs. Java evaluation), by unequal training budgets between FL and the local/central baselines, and by missing statistical or sampling details (no n for Pass@k, no repeated runs). These issues are fixable, but without addressing them the paper's main claims are not yet fully supported.","major_comments":[{"comment":"The fine-tuning dataset TutorCode consists of 1,239 buggy C++ programs, whereas the evaluation benchmark EvalRepair-Java is Java-based. The paper provides no argument, pilot, or control experiment establishing that repair capabilities learned on C++ transfer to Java, and Section 7 (Threats to Validity) does not list this language mismatch as a threat. Consequently, the reported gains of federated fine-tuning over the original models (e.g., +16.57% Top@10 on Mistral-7B in Table 2) and the RQ2 claim that heterogeneity has negligible impact could reflect cross-lingual code-generation improvements rather than learning of repair patterns. The authors should either evaluate on a same-language benchmark (e.g., fine-tune on Java data or test on C++ repair tasks) or provide explicit evidence that the learned adapters improve Java repair through a mechanism beyond generic code generation.","section":"Section 4.2, Tables 2–6"},{"comment":"The experimental comparison does not control for the number of local update steps. The hyperparameter setup states a maximum of 30 epochs per client and 10 global rounds, while local and centralized fine-tuning are described as using the same 30-epoch budget with no rounds. If 30 epochs are executed per client per round, federated fine-tuning entails up to 300 epochs of local updates per client, i.e., 10 times more training than local or centralized fine-tuning; even with early stopping, the training-effort disparity could explain FL's apparent superiority. The authors should equalize the total number of optimizer steps (or report the effective step counts) across the three settings before attributing the improvement to federated aggregation.","section":"Sections 4.5 and 5.1 (Table 2)"},{"comment":"Equation (8) defines Pass@k in terms of n, the total number of generated samples, but n is never stated for any experiment; Top@k similarly depends on the number of sampled completions, which is not reported. Without this information the reported percentages cannot be reproduced or meaningfully compared (a Pass@10 computed from n=10 samples is not the same quantity as one computed from n=200). In addition, all results come from a single run, with no variance estimates or confidence intervals, so differences of 2–3% (e.g., CodeQWen-7B FL vs. Central in Table 2) are within the likely noise band and should not be over-interpreted.","section":"Section 4.4 and Tables 2–8"},{"comment":"The Wilcoxon signed-rank tests and Cliff's delta values in Table 7 pool across six LLMs and five metrics without stating the sample size or the unit of analysis. This creates non-independence among observations and raises multiple-comparison concerns. The authors should specify which observations enter each test (e.g., per-model, per-metric, per-problem results), the sample size, and whether any correction for multiple testing was applied; otherwise the reported p-values are difficult to interpret.","section":"Section 5.2, Table 7"}],"minor_comments":[{"comment":"The model name '\"Mitral-7B\"' appears to be a typo for '\"Mistral-7B\"'; please fix it in the abbreviated names list.","section":"Section 4.3"},{"comment":"The text describing Figure 3b refers to '\"TutorLLMCode\"', but the dataset is called TutorCode elsewhere; please use a consistent name.","section":"Section 5.2"},{"comment":"The conclusion states the dataset contains '\"1239 programming problems\"', while Section 4.2 reports 1,239 buggy programs addressing 35 distinct problems; please align the wording.","section":"Section 8"},{"comment":"The Venn diagram in Figure 2 has no legend or explanation of the region labels; please clarify what the numbers represent (e.g., numbers of bugs fixed by each model or overlaps).","section":"Figure 2"},{"comment":"RQ1's title includes '\"protecting data privacy\"', but the study does not evaluate privacy guarantees (e.g., differential privacy or leakage under inference attacks); consider rephrasing to avoid overclaiming, or add a discussion of the privacy model assumed.","section":"Section 4.1 and Section 7"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a standard empirical SE paper and is within scope. The artifact link is a positive point, but the missing Pass@k sample count and the lack of repeated runs are reproducibility risks that the editor may want to emphasize are prerequisites for publication. I do not see evidence of deliberate misrepresentation; the issues are methodological gaps rather than a fundamentally flawed design."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort take: this is the first empirical study of federated fine-tuning for LLM-based program repair, and it is a solidly built study with a genuinely new combination. But the headline claims are undercut by a train/eval language mismatch the paper never addresses, and the privacy framing is looser than the title suggests.\n\nWhat is actually new: the combination of federated fine-tuning with generative code repair, six LLMs of diverse architectures, and the construction of feature-skewed Non-IID data from coding style, complexity, and embeddings. The finding that Non-IID heterogeneity has negligible impact on repair performance is interesting and contrasts with typical classification-based FL results. Credit where due: they use a private industrial dataset, the enhanced EvalRepair-Java benchmark with 583 test cases per problem, proper statistical tests (Wilcoxon, Cliff's delta), and they ship an artifact.\n\nSoft spots, in descending order of seriousness. First, the fine-tuning data is 1,239 C++ programs from TutorCode, while evaluation is entirely on EvalRepair-Java. The paper never justifies why repair knowledge learned on C++ should transfer to Java. Without a same-language baseline or a control experiment, the claimed improvements over the original models could reflect generic code-generation gains from any fine-tuning, not repair-specific learning. This also weakens the RQ2 claim about heterogeneity being negligible, since the heterogeneity is constructed from C++ features but evaluated on Java. Second, there are no repeated runs or variance bars anywhere; all comparisons are single-run point estimates. The number of generated samples n for the Pass@k estimator is never stated, so the reported Pass@k values have unknown reliability. Third, the paper says federated learning preserves data privacy, but FL alone does not guarantee privacy; the paper mentions differential privacy and secure aggregation as available extensions but never implements them, so the privacy claim should be framed as a motivation, not a result.\n\nThese are all addressable, but they temper the strength of the conclusions. The core direction is sound and worth pursuing. The paper is for SE researchers working on APR and on federated learning for code; it deserves serious peer review, not desk rejection, provided the review asks for the missing control and reporting.\n\nI would bring it to reading group and cite it for the new combination, but I would treat its quantitative claims as provisional.\n\nRecommendation: send to peer review, with major-revision expectations.","headline":"Solid first empirical study of federated fine-tuning for LLM-based program repair, but a C++-to-Java train/eval mismatch and single-run results temper the headline claims.","tokens_in":41744,"tokens_out":2549,"would_cite":true,"duration_ms":22474,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Federated fine-tuning of code LLMs can match centralized fine-tuning for program repair while keeping raw code private, and data heterogeneity has negligible impact.","keywords":["federated learning","program repair","LLM fine-tuning","data heterogeneity","non-IID data","QLoRA","code privacy","EvalRepair-Java"],"falsifier":"Run the same federated fine-tuning pipeline on TutorCode's C++ data and evaluate on a held-out C++ repair benchmark (e.g., ManyBugs) in addition to EvalRepair-Java; if the gains on C++ are much larger than on Java or the Java gains disappear, the central claim depends on unverified cross-language transfer.","tokens_in":40678,"feed_emoji":"🔒","tokens_out":6852,"duration_ms":58459,"temperature":0.7,"pith_summary":"This paper argues that companies with proprietary bug-fix data can fine-tune large language models for automated program repair collaboratively, without ever sharing the code itself. In experiments on 1,239 real-world private C++ bug-fix pairs from TutorCode, it compares federated fine-tuning against local fine-tuning, centralized fine-tuning, and the original models across six code LLMs, evaluating on the augmented Java benchmark EvalRepair-Java. The central finding is that federated fine-tuning improves repair capability by up to 16.67% on Top@10 and 18.44% on Pass@10, and that it can match or even beat centralized fine-tuning, which is usually treated as the upper bound. A second finding is that heterogeneous code—varied coding style, complexity, and embeddings—has negligible impact, so even extreme non-IID data distributions still improve bug fixing. The paper reads these results as evidence that privacy-preserving collaboration is practical for LLM-based program repair.","feed_headline":"Federated fine-tuning rivals centralized LLM program repair","feed_subtitle":"Privacy-preserving collaboration improved bug fixing on six code LLMs; data heterogeneity barely mattered.","key_machinery":"The carrier of the argument is parameter-efficient federated fine-tuning: clients freeze 4-bit quantized pre-trained weights and train only low-rank QLoRA adapters, which are uploaded to a server, averaged (mainly by FedAvg), and redistributed, so raw code never leaves a client while communication cost is small. To create realistic data heterogeneity, the paper extracts 23 coding-style attributes, measures code complexity by the number of modified hunks between buggy and fixed code, and uses CodeBERT embeddings of natural-language-program pairs, then allocates data to clients via a Dirichlet distribution with concentration parameters 0.1, 0.01, and extreme (one cluster per client). This construction lets the authors compare IID, mild, medium, and extreme non-IID scenarios across six LLMs, and the combination of QLoRA adapters plus FedAvg aggregation is what lets them attribute observed repair gains to federated collaboration rather than to data pooling.","core_discovery":"On its own terms, the paper reports three findings. First, federated fine-tuning of code LLMs with QLoRA adapters enhances program repair relative to local fine-tuning and the original models, and it rivals centralized fine-tuning, despite never pooling raw data; across six LLMs it achieved maximal gains of 16.67% on Top@10 and 18.44% on Pass@10 on EvalRepair-Java. Second, feature-skewed non-IID data constructed from coding style, code complexity, and code embeddings does not degrade performance compared with IID data; the differences are mostly not statistically significant and, where significant, have negligible effect sizes, and in some cases non-IID distributions outperform IID. Third, among federated algorithms, FedAvg has the best overall Borda-count ranking, while client-side (FedProx), server-side (FedSWA), both-side (FedOPT), and personalized (pFedMe) optimizations each show model-specific strengths and weaknesses, with pFedMe lagging notably.","pith_inferences":["If heterogeneity is truly negligible, the same adapter-based federated fine-tuning should transfer to other generative code tasks such as comment generation, test generation, and code translation, where feature skew also dominates.","The experiments simulate clients on a small number of GPUs; real deployments with hundreds of companies would face partial participation, stragglers, and communication limits that the current study does not exercise.","Privacy here is architectural—raw data stays local—but the paper does not analyze whether model updates or adapters can leak training code; adding differential privacy or secure aggregation would be a natural stress test.","A same-language replication (C++ fine-tuning evaluated on C++ bugs) would separate genuine repair learning from cross-language transfer and would directly test the weakest assumption."],"forward_implications":["Private code owners can jointly fine-tune code LLMs through adapters only, gaining most of the benefit of centralized fine-tuning without sharing source code.","Diverse coding styles, complexity levels, and problem domains across clients do not block collaboration; even extreme non-IID splits improve repair capability.","Model choice matters: different LLMs fix different bugs (CodeQWen-7B fixed 7 unique bugs, CodeLlama-13B fixed 2), so federated systems may benefit from mixing architectures.","Federated algorithm selection should be model-aware: FedAvg is the safest default, but FedOPT and FedSWA beat it on specific LLMs.","Personalized federated learning, despite its value in conventional tasks, is currently the weakest option for LLM fine-tuning for program repair."],"supporting_citations":[{"why":"Defines federated averaging, the foundational aggregation algorithm that all comparisons in RQ3 build on.","marker":"[63]"},{"why":"QLoRA is the parameter-efficient 4-bit fine-tuning method used to train and exchange only adapters in the federated workflow.","marker":"[17]"},{"why":"Supplies the fine-tuning hyperparameter guidance and the evidence that fine-tuned code LLMs improve APR, the baseline this study extends.","marker":"[31]"},{"why":"TutorCode is the private industrial dataset of 1,239 buggy C++ programs used for fine-tuning.","marker":"[101]"},{"why":"EvalRepair-Java is the augmented program repair benchmark used for evaluation.","marker":"[102]"},{"why":"EvalPlus provides the expanded test cases that raise each EvalRepair problem to 583 tests, mitigating patch overfitting.","marker":"[55]"},{"why":"FedProx is one of the client-side optimization baselines compared in RQ3.","marker":"[48]"},{"why":"FedOPT is the both-side adaptive optimization baseline compared in RQ3.","marker":"[70]"},{"why":"FedSWA is the server-side optimization baseline compared in RQ3.","marker":"[10]"},{"why":"pFedMe is the personalized federated learning baseline compared in RQ3.","marker":"[77]"}],"fun_headline_variants":["Federated LLM repair: private data, public gains","Federated fine-tuning for LLMs: privacy preserved, performance matched","Federated learning fixes bugs with privacy, rivals central training","Federated tuning: private code, centralized-level bug fixing","LLM repair via federated learning: private data, equal performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fine-tuning data are 1,239 buggy C++ programs, while the evaluation benchmark is Java, so the paper must assume that repair skill learned on C++ transfers to Java; if that transfer is weak, the measured Java gains would not reflect the federated setup's true repair improvement.","fun_headline_variants_meta":{"raw":{"variants":["Federated LLM repair: private data, public gains","Federated fine-tuning for LLMs: privacy preserved, performance matched","Federated learning fixes bugs with privacy, rivals central training","Federated tuning: private code, centralized-level bug fixing","LLM repair via federated learning: private data, equal performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00035,"raw_usage":{"total_tokens":1954,"prompt_tokens":1035,"completion_tokens":919,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":830}},"tokens_in":651,"tokens_out":919,"duration_ms":7584,"temperature":1.0,"reasoning_tokens":830,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:42:43.878126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same federated fine-tuning pipeline on TutorCode's C++ data and evaluate on a held-out C++ repair benchmark (e.g., ManyBugs) in addition to EvalRepair-Java; if the gains on C++ are much larger than on Java or the Java gains disappear, the central claim depends on unverified cross-language transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TutorCode is the private industrial dataset of 1,239 buggy C++ programs used for fine-tuning."},{"cited_title":"Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan","cited_arxiv_id":null,"evidence_quote":"FedOPT is the both-side adaptive optimization baseline compared in RQ3."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"pFedMe is the personalized federated learning baseline compared in RQ3."}],"review_version":1}