{"id":"d17ef06b-faf3-45bf-b4d8-1c49f8b02bea","arxiv_id":"2412.16589","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuning FIM code models on curriculum examples with retrieved context improves completion quality and live acceptance, with the largest gains for small models.","lead":"This paper fine-tunes small code completion models on hard-to-complete snippets plus repository context, and reports gains in offline benchmarks and live A/B tests. It introduces a multi-line infilling benchmark from SWE-bench and finds smaller models benefit most, which matters for low-latency autocomplete.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ~200k fine-tuning samples are never deduplicated against the CCEval and SWE-bench-derived evaluation repositories, so the offline gains across all models may be memorization rather than generalization.","rationale":"I read the paper in good faith: it offers a practical fine-tuning recipe, a useful new multi-line benchmark, and a genuine online A/B experiment. The strongest evidence for the central claim is the online CAR/CPR improvement, which is not explained by train/eval contamination. The main weakness is that the offline generalization claim—CMFT helps across model families, sizes, and datasets—rests on benchmarks that may overlap with the training corpus. The paper's Section 4.2 step 6 leakage control is narrow and does not address this. Also, offline deltas are small (roughly 0.1–2.0 points) and no error bars are reported, so even modest contamination could account for them. This is not an accusation; it is an empirical possibility that the paper leaves untested. Since the concern is testable with a dedup/retrain check and the online result supports the direction of the claim, conditional acceptance remains the right call. I therefore recommend UNCHANGED relative to the reader's verdict.","tokens_in":16647,"tokens_out":7113,"duration_ms":62087,"concrete_test":"Build an n-gram/MinHash overlap index over all ~200k fine-tuning files and all files from the CCEval repositories plus the SWE-bench instances used for the Multi-Line benchmark. Flag any training file with a 10-gram overlap or >0.8 Jaccard similarity with an eval file. If overlap exists, drop all flagged training files, retrain the DeepSeek-Coder-1B and StarCoder2-7B CMFT variants with identical hyperparameters (§5.1.4), and re-run Table 3/Table 4. If the offline deltas disappear or shrink to noise, contamination is confirmed; if they persist on the deduplicated training set, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central offline claim (Table 3) is that CMFT improves EM/PM/Pass@1 across StarCoder2 and DeepSeek models on three benchmarks. Two of those benchmarks—CCEval and the new Multi-Line Infilling set—are built from public GitHub repositories, and the fine-tuning corpus is the-stack-dedup plus scraped GitHub (§4). No overlap analysis between the ~200k training samples and the evaluation repositories is reported. The paper's own leakage control (§4.2, step 6) only removes context snippets that originate from the ground-truth snippet itself; it does not exclude training files whose repository also appears in an eval benchmark. Since public code benchmarks are routinely contained in the-stack, this is a concrete risk. If CCEval or SWE-bench files were in the fine-tuning data, the offline deltas (e.g., DS-Coder-1B Python PM 31.5→33.3) could reflect memorization. The online A/B test (§5.3) is real evidence and is not subject to this contamination, but it covers only one 1.6B model, so it cannot by itself support the cross-architecture, cross-dataset generalization claim. Thus the load-bearing part of the paper's evidence is the offline benchmark table, and that is exactly the part threatened by the missing deduplication.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a fine-tuning recipe for Fill-in-the-Middle (FIM) code completion models. The authors build a 'curriculum' dataset by extracting AST node types that show low relative Completion Acceptance Rate (CAR) in production telemetry, and a 'context' dataset by retrieving symbol definitions and repository context via the TypeScript compiler API and BM25. They fine-tune StarCoder2 and DeepSeek models with LoRA, comparing Random Span Fine-Tuning (RSFT), Curriculum-Aware Fine-Tuning (CUFT), Context-Aware Fine-Tuning (COFT), and the combined method (CMFT). Offline evaluation is reported on the Single-Line Infilling benchmark, CCEval, and a new Multi-Line Infilling benchmark derived from SWE-bench. A two-week online A/B test on DeepSeek-Coder-1.6B reports significant improvements in CAR and Completion Persistence Rate (CPR) with maintained p75 latency.","tokens_in":16894,"tokens_out":8442,"duration_ms":70517,"significance":"The paper makes a practical, deployment-oriented contribution: it introduces a new public multi-line infilling benchmark, a real-world A/B test with roughly 350k suggestions per arm, and a training-data generation pipeline. If the offline results survive deduplication and statistical testing, the main finding---that context- and curriculum-based fine-tuning helps smaller FIM models without a latency penalty---would be useful both for practitioners and for the code-completion literature. The paper is also commendable for reporting a leakage-control step, though, as detailed below, that step is narrower than the main contamination risk. The central risk is that the offline benchmark table, which carries the cross-architecture and cross-dataset generalization claim, may be inflated by overlap between the ~200k fine-tuning samples and the public evaluation repositories.","major_comments":[{"comment":"The paper reports no overlap analysis between the fine-tuning corpus (the-stack-dedup plus scraped GitHub, §4) and the evaluation repositories used in CCEval, Single-Line Infilling, and the new SWE-bench-derived Multi-Line Infilling benchmark (§5.1.1, §5.1.2). The leakage control in §4.2 step 6 only removes context snippets that originate from the ground-truth snippet itself; it does not exclude training files whose repository also appears in an evaluation benchmark. Since CCEval and SWE-bench are built from public GitHub repositories and the-stack covers public code, training/eval overlap is a concrete risk. The offline gains in Table 3 (e.g., DeepSeek-Coder-base 1B Python PM 31.5→33.3) could reflect memorization rather than generalization. Please report file-level and repository-level overlap statistics, and ideally re-run the main comparisons on training data deduplicated against all evaluation repositories.","section":"§4 vs. §5.1"},{"comment":"The offline results are presented as point estimates without variance, confidence intervals, or significance tests. Many of the reported improvements are small, for example DeepSeek-Coder-base 1B Single-Line Python EM 73.4→73.9 and Starcoder2 7B Single-Line Python EM 66.1→66.4. Without error bars or repeated runs, the caption claim that CMFT 'impoves across all the models architectures, parameters and datasets' is not statistically established. Please add confidence intervals, paired tests, or at least explicit standard errors across multiple fine-tuning seeds.","section":"Table 3 and §5.2.1"},{"comment":"The abstract's claim of 'zero latency impact' is under-supported. The online evaluation section only states that the p75 latency metric was maintained, without reporting the actual latency values or a statistical comparison. Table 5 reports offline generation latency for a 256-token generation task, which is not the same as the user-facing p75 latency in the A/B test. Please report the online latency distributions (e.g., p50/p75/p95) for control and treatment arms, and state how 'zero impact' was determined.","section":"Abstract and §5.3.2"},{"comment":"The online A/B test uses DeepSeek-Coder-1.6B as the base model, but the offline experiments in Table 3 evaluate DeepSeek-Coder-base 1B, DeepSeek-Coder-v2-lite-base 2.4B, DeepSeek-Coder-base 7B, StarCoder2 3B, and StarCoder2 7B. The 1.6B model does not appear in the offline table. Consequently, the claimed 'directional alignment' between offline and online results is indirect: a different base model was used in the live experiment. Please add offline results for the 1.6B model or temper the generalization claim to reflect that the online validation covers a single, separately evaluated model.","section":"§5.3"}],"minor_comments":[{"comment":"The caption contains a typo: 'impoves' should be 'improves'.","section":"Table 3 caption"},{"comment":"There is a typo in 'We provider Prompts in the same format...'; it should be 'We provide prompts...'.","section":"§5.1.4"},{"comment":"In the Starcoder2 7B Python row, COFT reports PM 38.0 and ES 60.1 against CMFT's PM 37.9 and ES 60.0, so the statement that 'CMFT yields the best results across various model families and sizes' is not literally true in every cell; please soften or clarify.","section":"Table 4"},{"comment":"'Relative CAR' is used in Figures 2 and 7 but never formally defined; please specify the normalization (e.g., per-language average) and the denominator used.","section":"Figures 2 and 7"},{"comment":"The Pearson correlation between Prefix Match and CAR is reported as r=0.6, p<0.05, but the number of data points and the unit of analysis (model-language pairs, or something else) are not stated; please provide these details.","section":"§5.3.2"},{"comment":"The StarCoder2 family is described as including a 15B parameter model, but the evaluations only cover the 3B and 7B variants; please state why the 15B model was excluded.","section":"§5.1.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a credible industrial contribution with a real A/B test and a useful new benchmark, but the missing deduplication analysis is a serious threat to the offline generalization claim. If overlap is found, the offline conclusions could collapse; if not, the paper would be much stronger after adding statistical rigor to the offline tables and clarifying the online latency claim. The use of a different base model online is an additional concern that should be addressed either by adding offline results for the 1.6B model or by narrowing the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this paper up front: it ships a genuinely useful new benchmark and a real live A/B test, and it never checks whether its fine-tuning corpus overlaps its evaluation benchmarks. That second point is the thing to chase if you referee it.\n\nWhat's new: the CMFT recipe—curriculum examples extracted from AST node types that show low acceptance rates (call expressions, if statements, etc.) plus context retrieved via TSC compiler for TS/JS and BM25 for everything else—is a sensible integration of existing ideas. The ablations are well designed: they include a random-span fine-tuning (RSFT) control, and RSFT barely moves the needle, which makes the curriculum/context effects more believable. The offline gains are small but consistent across five base models and three benchmarks. The new Multi-Line Infilling benchmark derived from SWE-bench (375 Python problems, public on Hugging Face) addresses a real gap. And the online A/B, ~350k suggestions per variant with significance testing, shows CAR and CPR gains that align with the offline direction. That's real evidence.\n\nSoft spots, in order of severity. First, the leakage question. Training data is the-stack plus scraped GitHub; CCEval and the new SWE-bench benchmark are built from public repos. The paper reports no overlap analysis. Its leakage control (§4.2 step 6) only removes context snippets that come from the ground truth itself, not training files from the same repository as an eval example. Since the-stack contains huge amounts of public code, this is a concrete risk. If CCEval or SWE-bench repos appear in the 200k fine-tuning samples, the offline deltas could be partially memorization. The online A/B is not affected, but it only covers one 1.6B model, so the cross-architecture generalization claim rests on the offline table. Second, the 'first insights from real-world A/B testing' claim is contradicted by their own reference [29], Murali et al., which is an industry A/B study of code authoring. That should be softened. Third, the offline numbers have no error bars or significance tests; several deltas are within a point or less. Fourth, 'zero latency impact' is asserted without reported numbers—Table 5 shows some latency per model but the online p75 maintenance isn't quantified.\n\nOverall: the central claim is probably right in direction. The paper is a solid, practical contribution for people building low-latency code completion. It deserves a serious referee, and the main thing to demand is a deduplication analysis between the fine-tuning and evaluation data. If that comes back clean, this is a solid accept; if not, the offline claims need heavy qualification.","headline":"A practical fine-tuning study for FIM code completion with a useful new benchmark and a real A/B test, but the offline results are threatened by a missing train/eval dedup analysis.","tokens_in":17380,"tokens_out":2382,"would_cite":true,"duration_ms":20626,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training on hard examples and repo context improves code completion","keywords":["code completion","fill-in-the-middle","curriculum learning","context-aware fine-tuning","code language models","FIM","A/B testing","cross-file context"],"falsifier":"Compute the overlap between the training repositories and the CCEval and SWE-bench-derived evaluation repositories (e.g., by file-path or content hash), retrain CMFT on a version that explicitly excludes any overlapping repositories, and re-run the same benchmarks; if the Exact Match, Prefix Match, and Pass@1 gains shrink toward the RSFT baseline, the offline claim is explained by data leakage rather than by curriculum and context learning.","tokens_in":16441,"feed_emoji":"💻","tokens_out":5221,"duration_ms":39538,"temperature":0.7,"pith_summary":"Fill-in-the-middle (FIM) models power code autocomplete, but they are latency-limited to small sizes. This paper claims that fine-tuning such models on a deliberately hard curriculum—code spans at AST node types where completions fail most often—together with the repository context that defines the symbols in those spans, reliably improves completion quality. The gains appear across model families (StarCoder2 and DeepSeek, 1B–7B plus a 2.4B-active MoE), across three offline benchmarks, and in an online A/B test, with the largest relative improvements on the smallest models. If true, this gives latency-constrained products a way to buy accuracy through data curation instead of model size.","feed_headline":"Training on hard examples and repo context improves code completion","feed_subtitle":"Smaller models gain the most, and online A/B tests show higher acceptance with no added latency.","key_machinery":"CMFT (Curriculum and Context-aware Fine-Tuning). Curriculum extraction uses tree-sitter to parse code into ASTs, selects node types with low completion-acceptance rates (Call Expression, Function Parameters, If Statement, etc.), ranks them by the number of unique symbols in their subtrees, and samples one node as the completion ground truth. Context extraction then collects the definitions of the symbols in that ground truth—via the TypeScript Compiler API for TypeScript/JavaScript (with nested definitions resolved to depth 2 through a symbol graph) and via BM25 retrieval for other languages—and appends them to the prompt, after removing any snippet that could itself leak the ground truth. Models are LoRA fine-tuned on roughly 200k such examples using the Prefix-Suffix-Middle FIM format, matching each base model's pretraining prompt format.","core_discovery":"Fine-tuning FIM code models on a combined dataset of curriculum examples and context examples—denoted CMFT—improves Exact Match on single-line infilling, Prefix Match on CrossCodeEval, and Pass@1 on a new multi-line infilling benchmark, across all tested base models. The paper also reports that in a two-week A/B test on a DeepSeek-Coder-1.6B base, CMFT raised Completion Acceptance Rate by up to 4.24% and Persistence Rate at 30s by up to 2.53% depending on line group, with p75 latency unchanged. The central claim is that the specific composition of fine-tuning data—hard AST-node patterns plus the cross-file context needed to resolve their symbols—is what drives the improvement, not fine-tuning itself, since random-span fine-tuning (RSFT) shows little or no gain.","pith_inferences":["The paper never reports whether its ~200k fine-tuning samples were deduplicated against the CCEval and SWE-bench-derived evaluation repositories; if public code from those repos leaked into training, the offline gains could be inflated by memorization, and the online A/B results would then be the cleaner evidence.","Because context quality is the biggest lever (COFT > CUFT), the method likely transfers best to statically analyzable languages; for languages without a compiler-API equivalent, BM25 retrieval may supply weaker context and smaller gains.","The reported offline–online correlation of r=0.6 between Prefix Match and CAR suggests Prefix Match is a usable proxy, but the unexplained 40% of variance leaves room for other factors (e.g., suggestion display position) that offline benchmarks do not capture.","A natural testable extension is applying the same curriculum+context recipe to larger chat-based models or to next-line suggestions, where the curriculum node distribution could be re-weighted by online failure rates per language."],"forward_implications":["CMFT improves Exact Match, Prefix Match, and Pass@1 for StarCoder2 3B/7B, DeepSeek-Coder 1B/7B, and DeepSeek-Coder-v2-lite 2.4B across all three evaluation suites.","Smaller models gain the most: Prefix Match on CCEval rises ~6.25% relative for the 1B model versus ~1.25% for the 7B model, so latency-bound deployments can adopt smaller served models.","Curriculum and context are complementary: combining them (CMFT) beats either alone (CUFT, COFT), and COFT generally beats CUFT, meaning context utilization is the larger lever.","A two-week A/B test on DeepSeek-Coder-1.6B yields statistically significant CAR and CPR gains, with the largest CAR gains (4.24%) on 2–5 line completions and p75 latency maintained.","Fine-tuned 7B-scale models reach 13.9–15.5% Pass@1 on the new multi-line infilling benchmark in under 1.2s, versus 19.4–23.4% at 1.67–2.85s for GPT-4o and Claude-3.5-sonnet."],"supporting_citations":[{"why":"Establishes the fill-in-the-middle objective and PSM mode used in fine-tuning, and the prior result that random-span fine-tuning yields little gain.","marker":"[6]"},{"why":"The Stack dedup is the base dataset from which curriculum and context examples are extracted.","marker":"[19]"},{"why":"CrossCodeEval is both the source of the motivating analysis tuples and one of the three offline evaluation benchmarks.","marker":"[9]"},{"why":"SWE-bench is the source repository set from which the new Multi-Line Infilling benchmark is derived.","marker":"[18]"},{"why":"DeepSeek-Coder provides the base models fine-tuned and the filtering approach applied to the training data.","marker":"[13]"},{"why":"StarCoder2 provides additional base models fine-tuned and the repository-context pretraining approach the paper contrasts with.","marker":"[23]"},{"why":"SantaCoder's single-line infilling benchmark is one of the offline evaluation datasets.","marker":"[2]"},{"why":"LoRA is the parameter-efficient fine-tuning method used for all fine-tuned variants.","marker":"[16]"}],"fun_headline_variants":["Training on hard code patterns and context improves FIM completions","Curriculum and context data lift code completion, no latency hit","Smaller FIM models gain most from curriculum and context training","Fine-tune on hard examples and repo context for better completions","Code completion improved by combined curriculum and context fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the ~200k fine-tuning samples drawn from The Stack and scraped GitHub are effectively disjoint from the repositories used in CCEval and the new SWE-bench-derived benchmark; if those evaluation repos appear in the training data, the reported offline improvements could be memorization rather than genuine completion skill.","fun_headline_variants_meta":{"raw":{"variants":["Training on hard code patterns and context improves FIM completions","Curriculum and context data lift code completion, no latency hit","Smaller FIM models gain most from curriculum and context training","Fine-tune on hard examples and repo context for better completions","Code completion improved by combined curriculum and context fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00046,"raw_usage":{"total_tokens":2341,"prompt_tokens":1019,"completion_tokens":1322,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":1238}},"tokens_in":635,"tokens_out":1322,"duration_ms":9465,"temperature":1.0,"reasoning_tokens":1238,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:26:03.997071+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the overlap between the training repositories and the CCEval and SWE-bench-derived evaluation repositories (e.g., by file-path or content hash), retrain CMFT on a version that explicitly excludes any overlapping repositories, and re-run the same benchmarks; if the Exact Match, Prefix Match, and Pass@1 gains shrink toward the RSFT baseline, the offline claim is explained by data leakage rather than by curriculum and context learning.","supporting_citations":[],"review_version":1}