{"id":"b94efd1a-0ff1-412b-be23-e681b9250f59","arxiv_id":"2412.15151","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"LANCE lets a 7B language model generate, review, filter, and annotate its own training data over four iterations, improving average benchmark scores by up to 3.64 points.","lead":"LANCE is a training loop where one language model writes new practice tasks, scores them, filters them, and builds preference pairs, then fine-tunes on the resulting data over several rounds. The authors report average benchmark gains of +3.64 on Qwen2-7B and +1.75 on Qwen2-7B-Instruct, with the largest jump in GSM8K math accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The review-score reliability of the self-generated constitution-based rewards is the load-bearing premise; the ablation in Table 3 shows removing DPO yields comparable or higher scores in early iterations, so the claimed necessity of the full pipeline and preference-pair construction is not…","rationale":"The reader's weakest_assumption correctly identifies that the self-review scores in Sections 3.3.1–3.3.3 lack validation against human judgments or an independent reward model. My stress-test concurs and sharpens the point using the paper's own ablation evidence: Table 3 shows that removing the DPO stage (which relies on the self-generated preference pairs) yields higher or comparable average scores in the first two iterations (66.89/67.08 vs 65.58/65.65), and in Iter3 the gap is only 0.07. This suggests the preference-pair construction and the reward scores behind it are not demonstrably the operative mechanism for the reported gains; the SFT-data generation alone may account for most of the improvement. A second, separate issue is the paper's autonomy framing: Section 4.1 states that Llama3-70B produced the seed review rationales/scores, so the first stage of the self-review pipeline is bootstrapped from an external model. This does not invalidate the method, but it weakens the claim of eliminating external model reliance and should be reported transparently. Additional statistical concerns (no seeds, no error bars, best-iteration selection in Table 1) mean the headline 3.64/1.75 gains are not yet robustly established. These are exactly the kind of addressable issues that warrant a CONDITIONAL verdict rather than ACCEPT or REJECT. I agree with the reader's assessment and recommend keeping CONDITIONAL. I do not see an internal inconsistency that would require REJECT: the method is coherent, the ablations are directionally plausible (full LANCE wins at Iter4), and the appendix includes per-iteration and per-step tables that partially mitigate cherry-picking concerns. The single most load-bearing uncertainty remains the reliability of the self-reward as a quality signal, and the concrete test above (human/independent-model agreement plus multi-seed ablations) would settle whether the gains are attributable to the claimed mechanism.","tokens_in":21424,"tokens_out":2244,"duration_ms":17042,"concrete_test":"Run the LANCE loop on Qwen2-7B with a held-out validation of the self-reward: (a) take a random sample of the generated instruction and preference pairs from each iteration, have human annotators or an independent judge (e.g., GPT-4o or a strong open reward model like ArmoRM) score them, and compute agreement with the Mt scores used for filtering and pairing; if agreement is below, say, 0.6 (Spearman or Cohen's kappa), the self-reward is miscalibrated and the filtering/preference signal is not quality. (b) Additionally, run the full LANCE pipeline and the w/o-dpo ablation with 3 seeds each and report means and standard deviations; if the w/o-dpo ablation matches or beats full LANCE within noise at the final iteration, the claimed necessity of the DPO/preference-pair component, and thus the value of the self-generated preference annotations, is not established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that LANCE continuously improves models through self-generated, self-reviewed data rests on Equation 4: the model Mt scores instruction-response pairs against a hand-designed constitution, and these scores drive both filtering (V=7) and preference-pair construction (Eq. 7). The paper never validates these self-rewards against human judgments or an independent reward model, so the loop may be self-confirming: the model can prefer responses that merely match its own stylistic biases rather than actual quality. The decisive internal evidence against the reward being the operative mechanism is Table 3 (Ablation Studies, Section 4.4) and Table 9 (Appendix F): LANCE w/o dpo reaches 66.89 (Iter1), 67.08 (Iter2), 67.85 (Iter3), 67.79 (Iter4), whereas full LANCE reaches 65.58, 65.65, 67.92, 68.24. In the first two iterations, removing DPO—and therefore removing the preference-pair construction that depends on the self-scores—outperforms the full pipeline; even in Iter3 the difference is only 0.07. The headline claim that preference learning stabilizes long-term improvements is inferred from a single trajectory without error bars or seeds, and the 'optimal iteration' selection in Table 1 further weakens it. The paper's autonomy claim is also overstated: Section 4.1 states that Llama3-70B generated the initial reward rationales and scores for the review seed data, so the self-reward mechanism is bootstrapped from an external model, which contradicts the abstract's positioning of eliminating reliance on external models. None of this is fatal by itself, but it means the central mechanism (self-reward quality) and the headline gains (3.64 / 1.75) are not yet robustly supported; the verdict CONDITIONAL is appropriate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LANCE, an iterative post-training paradigm in which an LLM carries out the full data-construction cycle for its own continued training. Starting from a small seed set of instruction-response pairs plus a separately annotated review set, the model M_t (i) reviews each seed example against a hand-written 'constitution' and produces 0-10 scores with rationales (Eq. 4); (ii) for items scoring below threshold V=7, generates K new instructions and responses via few-shot and chain-of-thought prompting, and for items at or above V, generates deliberately flawed responses; (iii) filters generated data by length and ROUGE-L similarity; (iv) retains instruction data whose review score exceeds V and constructs preference pairs by comparing self-scores of the original and flawed responses (Eq. 7); and (v) trains by SFT followed by DPO, repeating for four iterations. LANCE is evaluated on Qwen2-7B and Qwen2-7B-Instruct over six Open LLM Leaderboard tasks plus MATH, Olympiad Bench, MGSM, and Minerva Math, against SFT, Self-Instruct, SPIN, and I-SHEEP. The reported results show average gains of 3.64 and 1.75 points over the SFT baseline at the best (final) iteration, concentrated in mathematical reasoning, e.g., GSM8K +21.15 on Qwen2-7B.","tokens_in":21729,"tokens_out":24689,"duration_ms":165359,"significance":"Should the empirical claims withstand the robustness checks described below, the paper would be a useful data point for the self-evolving LLM literature: the idea of closing the loop over generation, cleaning, review, and preference construction is coherent, the fully reported per-iteration trajectories (Tables 2, 7, 8) and released code make the results checkable, and the math-reasoning gains (GSM8K +21.15, MATH +6.38, cross-lingual MGSM transfer from an English-only general seed) are genuinely interesting. The paper also names its own limitations explicitly in Section 6: knowledge gains are bounded by the model's existing knowledge, and the computational overhead is significant. The significance is tempered, however, by three addressable gaps: the review signal that drives the whole pipeline is never validated against human judgments or an independent reward model; the ablation evidence indicates the SFT-only variant is competitive or better in the first iterations, so the preference-learning component's contribution is not established; and all conclusions rest on single runs without error bars.","major_comments":[{"comment":"The ablation evidence does not support the conclusion that the DPO/preference component is necessary for the reported gains. Taking the average rows at face value, removing DPO yields higher scores than the full pipeline in Iterations 1 and 2 (66.89 vs 65.58; 67.08 vs 65.65), a near-tie in Iteration 3 (67.85 vs 67.92), and the full pipeline's only clear advantage is 0.45 points at Iteration 4; the claimed 'decline' of the SFT-only variant at Iteration 4 is a change of 0.06 points (67.85 to 67.79), which is within run-to-run noise. Table 8 likewise shows the DPO stage lowering the average relative to the preceding SFT stage in three of four iterations (66.89 to 65.58; 67.94 to 65.65; 68.09 to 67.92), with a +0.04 gain in Iteration 4. Consequently the headline average score enhancement of 3.64 cannot be attributed to the preference-learning component, and the claims that DPO accelerates performance gains and is essential for sustaining long-term improvements require multi-seed comparisons showing a statistically reliable advantage at the later iterations before they can be accepted.","section":"Section 4.4, Tables 3 and 8"},{"comment":"The constitution-based review score is the pipeline's only quality signal: it determines the V=7 filter that admits generated instruction data (Section 3.3.3) and orders the preference pairs in Eq. (7). The paper never validates this signal against human judgments or an independent reward model, and the initial review capability is bootstrapped from an external model plus human labels (Section 4.1: Llama3-70B generates the reward rationales and scores for the review seed, retaining only items consistent with the human-labeled OpenAssistant scores). The Introduction's and Abstract's framing about eliminating the need for human involvement or external models is therefore stronger than the setup warrants, and the loop is at risk of self-confirmation: the same model generates, scores, filters, and is trained on its own data, so high benchmark scores could in principle reflect selection for the model's own stylistic preferences rather than for data quality. The paper would be substantially strengthened by a concrete validation: report per-iteration agreement between M_t's scores and held-out human scores or an independent judge, and ablate the reward by replacing the V=7 filter with volume-matched random selection to show that the review signal, rather than the extra training volume, is what drives the gains.","section":"Sections 3.3.1-3.3.3 and 4.1, Eqs. (4)-(7)"},{"comment":"The central empirical claims rest on single runs without seeds, error bars, or significance tests, and several adjacent-iteration differences are extremely small: on Qwen2-7B the Iteration-1-to-2 average gain is 0.07 (65.58 to 65.65), and on Qwen2-7B-Instruct the average declines in Iteration 2 (68.72 to 68.36), so the caption claim that LANCE consistently shows performance gains across iterations holds only for the base model. The comparison with baselines is also endpoint-dependent: on Qwen2-7B, LANCE's average is below SPIN's in Iterations 1 and 2 (65.58 and 65.65 vs 68.00 and 67.86) and only overtakes SPIN from Iteration 3, so 'outperforming other self-evolution methods' describes the best and final iterations, not the trajectory as a whole. Per-benchmark, ARC-C remains below the SFT starting point at every LANCE iteration (51.11 to 50.68, Table 7), which also contradicts the Section 1 claim that individual metrics remain stable or show improvement. At minimum, please report seeds and error bars for the four-iteration trajectories and the main ablation, and report the variance associated with the GSM8K jump from 66.64 at Iteration 2 to 80.14 at Iteration 3, since a single-iteration jump of this size currently carries much of the headline result.","section":"Section 4.2, Tables 1 and 7, Figure 3"},{"comment":"The ablation tables contain inconsistencies that prevent an unambiguous reconstruction of the ablation trajectory. Table 3 lists the SFT starting-point average as 61.42, whereas Tables 1, 7, and 9 all report 64.60. In Table 8, the SFT rows' per-benchmark scores are swapped between the two models (the Qwen2-7B row carries the Instruct model's benchmark scores and vice versa), and the SFT-stage averages in Table 8 (66.89, 67.94, 68.09, 68.20) do not match the corresponding 'w/o dpo' averages in Table 9 (66.89, 67.08, 67.85, 67.79) from Iteration 2 onward, even though the two tables are meant to describe the same SFT-only stages. In Table 9, the LANCE Iter1 row appears to have the GSM8K and Winogrande columns swapped relative to Tables 2 and 7 (GSM8K shown as 72.38 instead of 67.32). These numbers need to be reconciled before the ablation-based conclusions in Section 4.4 can be evaluated.","section":"Appendix E (Table 8), Appendix F (Table 9), Section 4.4 (Table 3)"}],"minor_comments":[{"comment":"The text states that a regular expression f extracts the scores from the review output, but the review format and the regular expression are never specified; please provide both for reproducibility.","section":"Section 3.3.1"},{"comment":"The sentence 'we conducted four rounds of iterative experiments with N = 5' is ambiguous; given that the methodology defines the loop from t=0 to t=N, please clarify the intended N and whether the stopping point was fixed in advance.","section":"Section 4.2"},{"comment":"The column headers are garbled ('MA TH', 'MGSM_latin', 'Minerva Mathde sw fr average'); please reformat the table and state precisely which subtasks the MGSM and Minerva columns contain and how the Average column is computed.","section":"Table 2"},{"comment":"Labels such as 'LANCEIter1' lack spaces, and more importantly, the per-iteration tables (Tables 7, 8, and 9) should use one fixed column order to avoid the column-swap confusion noted in the major comments.","section":"Tables 2 and 7"},{"comment":"The Abstract claims LANCE significantly reduces the time and cost of post-training data construction, but no wall-clock, GPU-hour, or cost measurements are provided anywhere; given the Limitation section's acknowledgement of significant computational overhead, please either quantify the cost or qualify the claim.","section":"Abstract and Section 6"},{"comment":"Please report the number of generated instructions, the number of preference pairs, and the pass rates of the length, similarity, and V=7 filters at each iteration, so that readers can assess whether gains scale with data volume or with the filtering mechanism.","section":"Sections 3.3.3 and 4.1"},{"comment":"The curves and axis labels are not legible at print size in the submitted rendering; please regenerate the figure at higher resolution.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The headline results (Tables 1, 2, and 7) are internally consistent, which suggests the discrepancies are concentrated in the tables that support the ablation discussion (Tables 3, 8, and 9) rather than in the main results; still, the discrepancies are too numerous to be dismissed as trivial typos, and I would want the corrected tables verified before acceptance. My other substantive worry is framing: the no-external-models and no-human-involvement narrative is in tension with the Llama3-70B-initialized review data and the human-labeled OpenAssistant scores, and the authors should either provide an initialization-free variant or explicitly reposition the contribution as reducing rather than eliminating external supervision. I do not think the self-reward circularity, by itself, warrants rejection: the benchmark improvements are measured on held-out external benchmarks, so the claim is falsifiable in principle; the missing step is validation that the self-reward is the operative component, which the reward-agreement and filter-ablation experiments would provide. If multi-seed experiments are infeasible, the editor should weigh whether the single-run evidence, with all trajectory tables, meets the journal's bar."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: LANCE is a clean integration of known self-training pieces, and the claimed benchmark gains are real enough to take seriously, but the paper's own evidence doesn't yet carry the central mechanism. The specific design—reward-thresholded generation that writes new instructions for low-scoring seeds and deliberately flawed responses for high-scoring seeds, then filters and builds DPO pairs via self-scores—is not in the cited baselines. The iterative results, especially the GSM8K jump from 60.96 to 82.11 for Qwen2-7B, are the kind of thing that makes you want to try it. The paper is honest in places: per-iteration numbers are reported, ablations are included, and the authors admit that DPO caused temporary degradation in early iterations. The limitation section correctly notes that knowledge-dependent tasks don't improve.\n\nThe soft spots are real but addressable. The self-reward scores are never validated against human judgments or an independent reward model; the only ground truth is a seed review set scored once by Llama3-70B. That undercuts the 'eliminating external models' framing in the abstract, though it does not destroy the empirical claim. More worrying is Table 3: removing DPO beats the full pipeline in the first two iterations (66.89 vs 65.58, 67.08 vs 65.65), and the full pipeline only pulls ahead at Iter4 by 0.45 points. With single runs and no error bars, that difference is not robust. Table 1 also reports each method's best iteration, which flatters any method whose trajectory is noisy. The SFT baseline trained on only 8.8k examples is weak, so part of the gain may be from starting low.\n\nI think the verdict is conditional rather than reject. The paper deserves a serious referee, but the referee should push for multiple seeds, error bars, and some external validation of the self-scores. The authors should also soften the 'no external model' claim to 'no external model in the loop.' If you work on self-improving LLMs, this is a useful data point; I wouldn't cite it as established, but it is a fair piece of work with an honest limitation section.","headline":"A plausible self-training pipeline with modest gains, undercut by unvalidated self-rewards and an ablation that doesn't cleanly support the full design.","tokens_in":22373,"tokens_out":2931,"would_cite":false,"duration_ms":26542,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A language model can act as its own data engineer, generating, reviewing, and preference-annotating new training data in a loop, and in experiments this lifts Qwen2-7B's average benchmark score by 3.64 points and Qwen2-7B-Instruct's by…","keywords":["LANCE","self-evolving language models","synthetic data generation","data filtering","direct preference optimization","instruction tuning","LLM-as-judge","self-training"],"falsifier":"Take a held-out sample of LANCE-generated instruction and preference pairs, have human experts rate them, and compare those ratings with the constitution-based self-review scores; then train one model on data selected by the self-review scores and another on data selected by the human ratings. If the self-review selection does not beat random selection, or agrees only weakly with human ratings, the claimed self-evolution signal is not doing the work.","tokens_in":21188,"feed_emoji":"🤖","tokens_out":7833,"duration_ms":64603,"temperature":0.7,"pith_summary":"The paper proposes LANCE, a training paradigm in which the language model itself generates, cleans, reviews, and preference-annotates its own post-training data, and shows on Qwen2-7B and Qwen2-7B-Instruct that iterating this cycle improves average benchmark scores by 3.64 and 1.75 points respectively. The point of the claim is that high-quality expert-labeled data is the limiting resource for further LLM progress; if a model can make its own data, the bottleneck moves from annotation cost to the model's own generative and evaluative capacity. The experiments run four iterations and compare LANCE with self-instruction, self-play, and self-alignment baselines, finding continuous gains where the baselines stall or decline. A distinctive result is that the gains concentrate in mathematical reasoning, including transfer to Latin-script multilingual math, even though the seed data is general-purpose.","feed_headline":"Model that writes its own training data gains 3.64 points","feed_subtitle":"LANCE lets an LLM score and filter its own data, then fine-tune on it for steady benchmark gains.","key_machinery":"The load-bearing object is the review score $\\bar{S}_i$ that model $M_t$ assigns to each data item against a hand-written constitution (clarity, usefulness, challenge, safety, professionalism, guidance). Items with $\\bar{S}_i < V = 7$ are treated as low-quality and trigger few-shot chain-of-thought generation of new instruction-response pairs for SFT; items with $\\bar{S}_i \\geq V$ are treated as high-quality and trigger generation of deliberately flawed responses, which are scored and paired with the original as preference data. Cleaning uses length constraints and ROUGE-L similarity filtering, and training alternates negative log-likelihood loss with the preference-driven likelihood ratio loss of DPO, where the implicit reward is $\\hat{r}(x,y) = \\beta \\log \\frac{M^\\theta_t(y\\mid x)}{M^S_t(y\\mid x)}$, after which the fine-tuned model becomes $M_{t+1}$. The same model thus serves as generator, judge, cleaner, and annotator, closing the loop.","core_discovery":"On the paper's own terms, the central discovery is that a single LLM can run the full post-training data construction lifecycle without human experts or an external reward model: it reviews seed data against a constitution, splits examples by a threshold, generates new instruction data for weak spots and adversarial preference pairs for strong examples, filters and re-scores them, then fine-tunes itself with SFT followed by DPO. In the reported experiments this loop produces monotone or near-monotone average gains over four iterations on Qwen2-7B and a late-iteration jump on Qwen2-7B-Instruct, with the largest gains in math (GSM8K up 21.15 on Qwen2-7B), and it is the only tested method that improves on every iteration round. The paper interprets this as evidence that self-evolution can extend beyond the seed distribution rather than collapsing toward it, and that SFT and DPO play synergistic roles: SFT stabilizes, while DPO accelerates and sustains improvement.","pith_inferences":["Beyond the paper: a direct stress test would replace the model's self-review scores with an independent reward model on the same generated data and rerun the loop; if gains persist, the constitution-based self-judgment is not essential, and if they vanish, it is doing the causal work.","Beyond the paper: the reported multilingual math transfer covers Latin-script languages, so a natural extension is to test non-Latin scripts such as Chinese or Arabic, where the paper makes no claim of transfer.","Beyond the paper: since the paper says knowledge-dependent abilities are not improved, a promising extension would inject retrieval or fact-checking into the data-generation step, letting the model verify content before scoring it.","Beyond the paper: the threshold $V=7$ and the hand-written constitution are free parameters; varying the threshold across a grid and measuring downstream benchmark performance would show how sensitive the loop is to the review calibration."],"forward_implications":["If LANCE's results generalize, self-evolution can push a model beyond its original seed-data distribution, unlike methods that the paper says converge back toward the seed distribution.","The average benchmark gains persist across four iterations on Qwen2-7B, with the final iteration on Qwen2-7B-Instruct reaching the highest performance, suggesting further iterations could yield more.","The largest measured gains are in math reasoning (GSM8K up 21.15 on Qwen2-7B), and the gains transfer to Latin-script multilingual math even though the seed data is English-only and general-purpose.","Ablations indicate that neither SFT nor DPO alone suffices: removing SFT makes iteration unstable, while removing DPO slows improvement and lets it decline by the fourth iteration.","The paper's own limitation is that knowledge-dependent tasks improve only slightly, because self-evolved data cannot introduce knowledge beyond what the model already knows."],"supporting_citations":[{"why":"Supplies the Self-Instruct baseline and the core idea of using the LLM itself to construct instruction data, which LANCE extends into a full lifecycle.","marker":"Wang et al. (2023a)"},{"why":"Supplies the SPIN self-play baseline that LANCE must beat, and the convergence argument LANCE contrasts with its own data-expanding generation.","marker":"Chen et al. (2024b)"},{"why":"Supplies the I-SHEEP self-alignment baseline, the closest comparison where the model generates and assesses its own training data.","marker":"Liang et al. (2024)"},{"why":"Provides the DPO objective and implicit reward formulation used as the PLR training step after SFT.","marker":"Rafailov et al. (2024)"},{"why":"Inspires the hand-written constitution of review principles used to score seed and generated data.","marker":"Bai et al. (2022)"},{"why":"Supports the premise that an LLM can judge response quality with high agreement with human evaluations.","marker":"Zheng et al. (2023)"},{"why":"Supplies part of the seed instruction data, sampled from UltraChat.","marker":"Ding et al. (2023)"},{"why":"Supplies the OpenAssistant seed conversations with human-labeled scores used to build the review portion of the seed dataset.","marker":"Köpf et al. (2024)"},{"why":"Supplies the Qwen2-7B and Qwen2-7B-Instruct backbone models on which all experiments are run.","marker":"Yang et al. (2024)"}],"fun_headline_variants":["LLM as its own data engineer: 3.64-point gain","Self-evolving LLM improves via self-made data","LLM writes, scores, and learns from its own data","Autonomous self-data loop lifts LLM by 3.64","LLM without experts builds its own training data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole loop depends on the model's own review scores genuinely separating good data from bad; if those scores are miscalibrated, filtering and preference-pairing would amplify the model's existing mistakes instead of correcting them.","fun_headline_variants_meta":{"raw":{"variants":["LLM as its own data engineer: 3.64-point gain","Self-evolving LLM improves via self-made data","LLM writes, scores, and learns from its own data","Autonomous self-data loop lifts LLM by 3.64","LLM without experts builds its own training data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000828,"raw_usage":{"total_tokens":3643,"prompt_tokens":993,"completion_tokens":2650,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":2567}},"tokens_in":609,"tokens_out":2650,"duration_ms":21981,"temperature":1.0,"reasoning_tokens":2567,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:34:55.181656+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out sample of LANCE-generated instruction and preference pairs, have human experts rate them, and compare those ratings with the constitution-based self-review scores; then train one model on data selected by the self-review scores and another on data selected by the human ratings. If the self-review selection does not beat random selection, or agrees only weakly with human ratings, the claimed self-evolution signal is not doing the work.","supporting_citations":[],"review_version":1}