{"id":"c74ec7e2-525e-40ec-a098-44ce0fbe4d98","arxiv_id":"2509.07858","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Small code LLMs can be bootstrapped into effective instruction-data synthesizers via iterative self-distillation, producing code models that match or exceed baselines trained on proprietary-LLM data.","lead":"SCoder trains small 7B-14B open-source models to generate high-quality code instruction data, cutting the need for large amounts of expensive proprietary-LLM data. The method iteratively improves the data synthesizer using its own outputs, and the resulting code models match or beat larger baselines on several code benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scorer-weight vector w in Eq. (2) is fit on LiveCodeBench, which is also the headline Table 3 benchmark; the reported LiveCodeBench advantage is not a clean held-out result and may be inflated by test-set tuning.","rationale":"I agree with the reader's choice of the weakest assumption and elevate it to the primary gate on the headline claim. The paper's central empirical contribution is that SCoder's data-filtering pipeline turns small synthesizers into data of state-of-the-art quality. Eq. (2) determines a critical free parameter (per-aspect weights) using exactly the benchmark that Table 3 uses for the biggest claimed advantage. The term 'OOD test set' in §3.3 is contradicted by Appendix C. This is not an accusation of dishonesty; it is a standard selection-on-the-test-set risk. The HumanEval/MBPP results are not tainted this way, and BigCodeBench is not part of the w fit, so the overall equivalence claim likely survives; but the LiveCodeBench-specific 'surpasses baselines by 5.9%' must be re-derived with a clean validation target. A secondary concern I would not elevate: the theoretical Appendix E is a generic contraction argument whose assumptions (Lipschitz training/data generation) are neither verified nor used by the experiments; it does not support or undermine the main numbers. The right gate is the w/LiveCodeBench coupling; if the proposed re-fit test leaves the LCB gain intact, the paper's main claim is credible and could be accepted after release of code and data.","tokens_in":17528,"tokens_out":6115,"duration_ms":55871,"concrete_test":"Refit Eq. (2) with the same K=20 experiments but with y_k computed on HumanEval and MBPP (or on LiveCodeBench problems dated before 202410) instead of LiveCodeBench 202410-202501; freeze all other hyperparameters; regenerate the bootstrapped synthesizers and SCoder-Q14, and compare LiveCodeBench (Full) pass@1. If the LCB gap to WizardCoder-GPT-4-6.7B's 21.0 shrinks below about 1 point or reverses, the coupling concern lands. Also report the fitted w under both evaluation targets and the correlation between them.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing weakness is the coupling between model selection and evaluation. In §3.3, the multi-aspect scoring weights w are estimated by ridge regression (Eq. 2) over K=20 candidate data-selection experiments. The paper calls this an 'out-of-distribution (OOD) test set,' but Appendix C identifies it as LiveCodeBench (202410-202501), and the main results in Table 3 report LiveCodeBench (Full) on the same benchmark family. The full pipeline—multi-aspect scoring, and hence the weight vector w—is therefore tuned directly on the benchmark used for the headline claim. The effect is visible in Table 4: removing multi-aspect scoring drops LiveCodeBench from 21.4 to 19.9 for SCoder-Q7, so the component is a major contributor to the reported gain. Because w is a free parameter of the data-selection procedure, the LiveCodeBench advantage of SCoder-Q14 over the best baseline (22.2 vs 21.0) may partly reflect overfitting to that benchmark rather than a genuine improvement in instruction-data quality. The HumanEval/MBPP results are less affected, and BigCodeBench was not used to fit w, so the central 'matching/outperforming baselines' claim does not collapse; but the specific LiveCodeBench superiority is not a clean test.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCoder, a method for bootstrapping small open-source LLMs (7B-14B) into effective synthesizers of code instruction data, thereby reducing reliance on proprietary-LLM distillation. The synthesizers are first trained on 10K GPT-4o distilled samples and then improved through two iterations of self-distillation. At each iteration, candidate instruction samples are generated by multi-checkpoint sampling, scored by a multi-aspect scorer with ridge-regression weights w, and filtered by gradient-based influence estimation against the proprietary seed data. The resulting instruction data are used to fine-tune DeepSeek-Coder-6.7B-Base (after a shared 110K evol-codealpaca-v1 stage). The main results (Table 3) show that SCoder-Q14-DS-6.7B outperforms or matches open-source baselines on HumanEval, MBPP, LiveCodeBench, and BigCodeBench; ablations (Table 4) show that each proposed component contributes; data-scaling experiments (Figure 3) and transfer to other base models (Table 8) support the generality of the approach.","tokens_in":17803,"tokens_out":4197,"duration_ms":36974,"significance":"If the results hold, this is a valuable empirical contribution: it demonstrates that small open-source models can produce instruction data competitive with proprietary-LLM-distilled data, at substantially lower API cost. The paper is careful to control for the shared 110K evol-codealpaca-v1 base across baselines, and it provides ablations for all three proposed components, a data-scaling study, a cost analysis, and transfer experiments on additional base models. These are concrete strengths. The main caveat is that the multi-aspect scoring weights are fitted on LiveCodeBench, which is also a headline evaluation benchmark, so the reported LiveCodeBench advantage is not a clean held-out result; the other benchmark gains are less affected.","major_comments":[{"comment":"The multi-aspect scoring weights w in Eq. (2) are estimated by ridge regression on K=20 experiments evaluated on LiveCodeBench (202410-202501), as stated in Appendix C. LiveCodeBench then appears as a headline benchmark in Table 3 (LCB Full and its sub-splits). Consequently, the reported LiveCodeBench advantage of SCoder-Q14 over the best open-source baseline (22.2 vs 21.0, claimed as a 5.9% gain in §4.4) is not a clean held-out comparison: part of the gap may reflect tuning w to that benchmark. The ablation in Table 4 shows that removing multi-aspect scoring drops LiveCodeBench from 21.4 to 19.9, so this component is a major contributor to the reported gain. I request that the authors either refit w on a benchmark not used in Table 3 and demonstrate that the LiveCodeBench gains survive, or report LiveCodeBench only as a development-set result, or show that a fixed a-priori weight vector yields similar LiveCodeBench performance. The HumanEval, MBPP, and BigCodeBench comparisons are not affected by this coupling and should be emphasized accordingly.","section":"§3.3, Eq. (2); Appendix C; Table 3"},{"comment":"The convergence analysis rests on the contraction condition L_T L_G < 1 stated as Assumption (A3), but no evidence is provided that this condition holds for the actual training operator T and data generation map G used in the experiments; the Lipschitz constants are not estimated. As written, the result is a conditional statement about an abstract operator, not a proof that the iterative self-distillation procedure used in the paper converges. Please either remove or substantially soften the theoretical claim, or provide an empirical check (e.g., estimate the relevant Lipschitz constants on the actual models) and state clearly that the contraction condition is an unverified assumption.","section":"Appendix E, Eqs. (7)-(9)"}],"minor_comments":[{"comment":"The abstract and introduction state that SCoder achieves 'state-of-the-art code generation capabilities' without qualifying that the comparison is against open-source models; Table 3 shows that GPT-o1-Preview substantially outperforms SCoder on several benchmarks. Please qualify the claim.","section":"Abstract and §1"},{"comment":"The sentence 'surpasses the best open-source baselines by 5.9% and 9.7% on average' should specify whether the percentages are relative or absolute, and it should identify which sub-benchmarks are being averaged, since Table 3 reports multiple LiveCodeBench and BigCodeBench columns.","section":"§4.4"},{"comment":"Calling LiveCodeBench an 'out-of-distribution (OOD) test set' is confusing because the same benchmark family is used for the main evaluation in Table 3; unless the exact problem split used for weight fitting is disjoint from the evaluated split, a term such as 'development set' would be more accurate.","section":"§3.3"},{"comment":"The legend '60K null; 40K 40K; 20K 20K; 40K 20K' is not self-explanatory; please clarify what the two numbers denote (e.g., amount of self-distilled data used in iteration 1 and iteration 2).","section":"Figure 4"},{"comment":"The reference-model sensitivity study reports only HumanEval and LiveCodeBench; since LiveCodeBench is the benchmark coupled to weight fitting, please also report MBPP and BigCodeBench for these sensitivity runs.","section":"Appendix G and Table 7"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Mark,\n\nThe short version: this is a real advance in cost-efficient code instruction data synthesis, and the empirical story mostly holds, but the headline LiveCodeBench result is not a clean test because the scorer weights were tuned on that benchmark. The rest of the evidence is strong enough that I'd send it out, with a request to fix or reframe the LiveCodeBench evaluation.\n\nWhat's new: the iterative bootstrap—train a small synthesizer on 10K proprietary samples, then generate its own training data through multi-checkpoint sampling, multi-aspect scoring, and gradient influence filtering, and repeat. Each piece is borrowed, but the combination for code instruction data is new and clearly effective. The ablations show all three components matter, the data scaling curve improves with more synthesized data, and the final models beat or match baselines that use 75K–200K proprietary-distilled samples. The cost analysis (roughly $263 to train the synthesizer vs. API costs) is a useful concrete point.\n\nThe big soft spot: the multi-aspect scorer's weight vector w is fit by ridge regression on K=20 experiments evaluated on LiveCodeBench (202410–202501), and Table 3 reports LiveCodeBench (Full) on that same benchmark family. The ablation shows w contributes ~1.5 points on LCB; the gap to the best baseline is 1.2 points. So the specific 'outperforms on LiveCodeBench' claim likely includes test-set tuning. The authors are transparent about this in Appendix C, but they still call it an 'OOD test set' in the main text, which is misleading. The other benchmarks (HumanEval, MBPP, BigCodeBench) were not used to fit w, and the gains there are credible.\n\nMinor: no code or data released, which matters because the method depends on a 10K GPT-4o seed. The theoretical analysis in Appendix E is filler—Lipschitz assumptions and a contraction condition that are unverified; it adds nothing to the empirical claims. The 'self' in self-distillation is also anchored to a proprietary seed, so it's really 'reduced-reliance distillation', though the authors don't overclaim that beyond the title.\n\nBottom line: the central claim holds on the benchmarks that aren't used for tuning; the LiveCodeBench advantage is not trustworthy as stated. I'd cite this for the method, not for the LCB number.\n\nFor peer review: send it out, but ask for (1) an honest separation of tuned vs. untouched benchmarks, or re-fitting w on a disjoint split; (2) release of the generated datasets and scorer; (3) either remove or heavily reframe the theoretical section.","headline":"A real step forward in cheap code instruction data synthesis, but the LiveCodeBench headline is fitted, not tested.","tokens_in":18332,"tokens_out":3477,"would_cite":true,"duration_ms":29987,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that small open-source code LLMs, bootstrapped by iterative self-distillation from just 10K proprietary samples, can generate instruction data that lets a 6.7B code model match or beat models trained on far larger…","keywords":["code generation","instruction tuning","data synthesis","self-distillation","small language models","influence estimation","data selection","synthetic training data"],"falsifier":"Hold out a newer or disjoint set of code-generation problems (e.g., later LiveCodeBench months or another contest benchmark), refit the scoring weights without looking at it, regenerate the synthesizer, and rerun the main comparison; if the LiveCodeBench and BigCodeBench advantages vanish while HumanEval/MBPP gains persist, the headline numbers partly measure test-set tuning.","tokens_in":17318,"feed_emoji":"🤖","tokens_out":6780,"duration_ms":52967,"temperature":0.7,"pith_summary":"This paper tries to show that small open-source code LLMs (7B, 8B, 14B) can be turned into powerful generators of code instruction data with only a small seed of proprietary samples, eliminating the need to distill large volumes of data from costly APIs. The authors propose an iterative self-distillation loop in which the synthesizer is repeatedly trained on its own most influential self-generated samples, selected by multi-checkpoint sampling, multi-aspect scoring, and gradient-based influence estimation. Using 10K GPT-4o samples as a seed and two bootstrap iterations, the resulting synthesizers generate 60K instruction data on which a DeepSeek-Coder-6.7B-Base model is fine-tuned. The resulting SCoder models match or beat open-source baselines that were trained on more and larger proprietary-distilled datasets on HumanEval, MBPP, LiveCodeBench, and BigCodeBench. If this holds, high-quality code instruction data no longer requires expensive proprietary distillation.","feed_headline":"Bootstrapped 7B models beat costly LLM data distillation","feed_subtitle":"A 6.7B code model trained on 60K self-distilled samples matches models trained on 75K+ GPT-4 samples.","key_machinery":"The load-bearing mechanism is the iterative self-distillation loop that alternates data generation with synthesizer training. Each iteration starts with M×N candidate problem-solution pairs sampled from M checkpoints of the current synthesizer per code snippet; a multi-aspect scorer evaluates each candidate across ten aspects and aggregates them into a weighted score, where the aspect weights w are fit by ridge regression over K=20 fine-tuning experiments evaluated on an out-of-distribution test set. The selected candidates are then filtered by gradient-based influence estimation: a LoRA reference model trained on the 10K proprietary samples is used to compute projected gradients for each self-distilled sample (via a Rademacher projection), and only samples whose gradients align with the average proprietary-sample gradient are kept. These influential samples train the synthesizer itself for the next iteration, and the final bootstrapped synthesizer generates the 60K instruction data used to fine-tune the target code model. The machinery's job is to keep the self-distillation loop from drifting toward low-quality or repetitive data while minimizing the number of proprietary samples needed.","core_discovery":"The central claim is that instruction-data synthesis is not a capability reserved for large proprietary models: a 7B-14B open-source model, trained once on 10K proprietary synthesis samples, can bootstrap itself through iterative self-distillation into a synthesizer whose output trains a 6.7B target model to state-of-the-art or matching performance. In the paper's experiments, two bootstrap iterations with 20K and then 40K self-distilled samples raise the quality of the synthesizer's data on all four benchmarks; the best model, SCoder-Q14-DS-6.7B, surpasses the best open-source baseline by 5.9% on LiveCodeBench and 9.7% on BigCodeBench on average. The paper further claims that all three selection components are necessary: removing multi-checkpoint sampling, multi-aspect scoring, or gradient-based influence estimation degrades target-model performance by 4.8-8.1 points on the reported benchmarks. It also reports that synthesized data scores higher than the 110K evol-codealpaca-v1 dataset on all ten quality aspects judged by a GPT-4o-based evaluation.","pith_inferences":["The method's convergence argument (Banach fixed point, Nash equilibrium framing) is domain-agnostic; the same loop should transfer to math, general instruction, or other structured generation tasks if a seed set of high-quality samples and an appropriate snippet pool exist.","The gradient-influence filter's anchor is the proprietary seed set, not the test set; if the anchor were replaced by any small high-quality reference set, the pipeline could become fully open-source without relying on proprietary models at all.","The cost analysis implies that the one-time 10K API seed is the dominant proprietary cost; after training, the synthesizer generates instruction data at a marginal cost far below API distillation, so the method's advantage grows with scale.","Because the multi-aspect scoring weights are fitted on LiveCodeBench, the method's reported advantage on that benchmark may be optimistic; on benchmarks not used in weight fitting (HumanEval and MBPP), the gains are real but smaller."],"forward_implications":["With only 10K proprietary samples as seed, two bootstrap iterations let a Qwen2.5-Coder-14B synthesizer lift a DeepSeek-Coder-6.7B target from 65.3 to 70.1 HumanEval and 73.7 to 76.5 MBPP.","The final SCoder models, trained on 60K synthesizer-generated data plus the common 110K evol-codealpaca-v1 warm-up, match or beat open-source baselines that use larger proprietary-distilled datasets (75K-110K+).","Ablations show that each selection component contributes: removing multi-checkpoint sampling, multi-aspect scoring, or gradient-based influence estimation costs at least 4.8 points on HumanEval and 8.1 points on LiveCodeBench.","Data scaling holds: increasing synthesized data size improves the target model across benchmarks, surpassing DeepSeek-Coder-6.7B-Instruct on most benchmarks.","The bootstrap stabilizes after two iterations; a third iteration yields no further gains, suggesting a natural stopping point."],"supporting_citations":[{"why":"Supplies DeepSeek-Coder-6.7B-Base, the target code model fine-tuned in all experiments.","marker":"(Guo et al., 2024)"},{"why":"Magicoder's OSS-Instruct motivates the code-snippet-to-instruction synthesis setup and the ten task categories used for diversified snippet sampling; also a baseline.","marker":"(Wei et al., 2024)"},{"why":"Supplies the 110K evol-codealpaca-v1 dataset used as common warm-up data, and WizardCoder as a baseline.","marker":"(Luo et al., 2024)"},{"why":"StarCoder2's preprocessing pipeline is the basis for the clean code snippet pool C.","marker":"(Lozhkov et al., 2024)"},{"why":"LESS provides the gradient-based influence selection approach adapted here to filter self-distilled samples.","marker":"(Xia et al., 2024)"},{"why":"Gives the gradient-influence formulation Inf(d,t) on which the filtering score is based.","marker":"(Pruthi et al., 2020)"},{"why":"TRAK supplies the Rademacher projection used to reduce gradient dimensionality with Johnson-Lindenstrauss guarantees.","marker":"(Park et al., 2023)"},{"why":"The Johnson-Lindenstrauss Lemma justifies that projected gradients preserve distances for the cosine-similarity influence estimate.","marker":"(Johnson et al., 1984)"},{"why":"Best-of-N, the single-checkpoint baseline that multi-checkpoint sampling is designed to improve upon.","marker":"(Stiennon et al., 2022)"}],"fun_headline_variants":["Small code models self-distill into SOTA synthesizers","Iterative self-distillation turns 7B models into top data generators","Self-distilled data: 6.7B model rivals GPT-4 distilled training","Bootstrapped 7B synthesizers cut need for proprietary LLM data","SCoder: iterative self-distillation boosts small code LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The scoring weights that decide which self-distilled samples are kept are tuned on LiveCodeBench, and the final models are then compared on LiveCodeBench, so some of the reported gain on that benchmark may be tuning to the test set rather than a general improvement in data quality.","fun_headline_variants_meta":{"raw":{"variants":["Small code models self-distill into SOTA synthesizers","Iterative self-distillation turns 7B models into top data generators","Self-distilled data: 6.7B model rivals GPT-4 distilled training","Bootstrapped 7B synthesizers cut need for proprietary LLM data","SCoder: iterative self-distillation boosts small code LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000179,"raw_usage":{"total_tokens":1333,"prompt_tokens":1012,"completion_tokens":321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":226}},"tokens_in":628,"tokens_out":321,"duration_ms":3111,"temperature":1.0,"reasoning_tokens":226,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:08:31.171367+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a newer or disjoint set of code-generation problems (e.g., later LiveCodeBench months or another contest benchmark), refit the scoring weights without looking at it, regenerate the synthesizer, and rerun the main comparison; if the LiveCodeBench and BigCodeBench advantages vanish while HumanEval/MBPP gains persist, the headline numbers partly measure test-set tuning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the gradient-influence formulation Inf(d,t) on which the filtering score is based."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Johnson-Lindenstrauss Lemma justifies that projected gradients preserve distances for the cosine-similarity influence estimate."}],"review_version":2}