{"id":"7fe85948-7c16-4746-bf78-0e4f6187d5e7","arxiv_id":"2509.04796","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Under recursive synthetic training, a 1B-parameter LLM loses factual accuracy while preserving fluent, confident output, with collapse timing depending on prompt format and domain-aligned training.","lead":"This paper shows that an AI model repeatedly trained on its own writing can lose factual accuracy while still sounding fluent and confident. It also reports that question phrasing changes how fast this happens, and that subject-focused synthetic training slows the decline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy metric computed from highest-probability answer token (Table 2, A.6) may measure surface-form selection rather than factual recall; every stage definition and the 15x mitigation claim rest on this score.","rationale":"The reader's weakest-assumption analysis identifies the accuracy metric, and my reading of the paper agrees that this is the load-bearing point. The three-stage framework, the instruction-format dependence, and the 15x mitigation claim are all derived from slopes and thresholds of this score, so a scoring artifact would not just weaken a secondary result; it would change the central claim. The paper has real strengths: a controlled recursive-training setup, public datasets, clearly specified hyperparameters, and concrete qualitative examples of confident wrong answers. Those strengths make the paper worth conditional consideration, but they do not substitute for a validated measure of factual recall. The checklist also claims a subject-sensitivity result (F(4,1960)=5.92, p<0.001) that appears nowhere else in the paper; while not part of the abstract's central claims, it is a further sign that the evaluation pipeline is under-documented. If the accuracy re-analysis requested above is run and the curves are robust, the conditional verdict should stand; if the curves diverge, the quantitative claims would need to be re-estimated from a generative accuracy score.","tokens_in":17380,"tokens_out":6409,"duration_ms":57101,"concrete_test":"Recompute the accuracy curves for the alpha=0.50 instruction-format experiment (Figure 3) and the domain-mitigation comparison (Figure 4) on the same checkpoints using two scorers: (a) the Table 2 highest-probability-token rule, and (b) greedy free-form decoding scored by exact match or the paper's judge/entailment metrics, with options omitted from the prompt. If per-generation accuracy differs by more than about five points, or if the Stage A-to-B transition shifts by more than one generation, the paper's accuracy metric is not a faithful measure of factual recall. As a secondary check, compare single-token scoring against the full-option log-probability score s(o) defined in A.5; if a nontrivial fraction of items change their top option, the reported metric is tokenization-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claims are all built on one accuracy score: the Stage A/B/C boundaries in Figure 2, the instruction-format ordering in Figure 3, and the 15x decay-rate comparison in Figure 4 and Table 4. The only description of how that score is computed is Table 2's caption: 'evaluation was computed from the highest-probability answer token.' Appendix A.6 says MMLU items were reformatted into 'short-answer' prompts that list all option texts explicitly, and that responses were standardized to 1-2 sentences. These two descriptions are in tension. If accuracy is whether the single highest-probability token is the first token of the correct option, then multi-token options are reduced to their first token and the model is being scored on surface-form selection, not knowledge retrieval. If instead the top-1 generated token is used, a fluent correct answer such as 'Expanding territories under Spanish control' can be scored wrong while a wrong phrase copied from the prompt can be scored right. Either way, the observed decline could reflect format adaptation or tokenization artifacts rather than the three-stage epistemic collapse the paper claims. The judge and entailment metrics in A.5 could have validated the accuracy score, but they are never used to define stages or to compute the decay rates. The reported ANOVAs do not resolve this because they test the same potentially artifact-laden score.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies recursive synthetic training of a 1B instruction-tuned language model (GEMMA 3 1B IT) and proposes a three-stage phenomenon called \"knowledge collapse\": Stage A (knowledge preservation), Stage B (confidently wrong answers with preserved fluency), and Stage C (instruction-following collapse). It reports that the trajectory and timing of collapse depend on instruction format (zero-shot, few-shot, short-answer), and that domain-specific synthetic training—built by semantically filtering WikiText for a target MMLU subject—delays accuracy decay by roughly 15× relative to general synthetic training. The evaluation combines model-centric signals (perplexity, entropy, gibberish score) with task-centric signals (accuracy, greedy rate, max option frequency, judge and entailment scores) across synthetic ratios α ∈ {0.25, 0.50, 1.0} over multiple recursive generations. The paper includes detailed appendices on dataset construction, training configuration, statistical analysis, and qualitative response trajectories, plus a NeurIPS-style checklist claiming reproducibility and statistical validation.","tokens_in":17610,"tokens_out":3817,"duration_ms":36693,"significance":"If the central claims hold, the paper makes a timely empirical contribution: it identifies a dangerous intermediate regime (Stage B) where models produce fluent but factually incorrect outputs under recursive synthetic training, and it argues that this regime is instruction-format-dependent rather than a global, prompt-agnostic drift. The proposed domain-specific mitigation, if not an artifact of evaluation leakage, would be practically useful for knowledge-intensive applications. Strengths of the manuscript include the reproducible experimental framework (fixed seeds, detailed hyperparameters, public datasets), the use of complementary model- and task-centric metrics, and the qualitative examples that illustrate the claimed degradation trajectories. The main weaknesses are the ambiguous definition of the accuracy score, the absence of error bars and independent runs, and the potentially circular construction of the mitigation corpus. These issues are load-bearing because all stage boundaries, format comparisons, and the 15× mitigation claim are computed from that single accuracy score.","major_comments":[{"comment":"The accuracy metric is not defined precisely enough to support the paper's central claims. Table 2's caption states that \"evaluation was computed from the highest-probability answer token,\" while Appendix A.6 says responses were reformatted to open-ended short answers of 1–2 sentences. These two descriptions are in tension. If the score is based on the single highest-probability token, then multi-token correct answers such as \"Expanding territories under Spanish control\" are scored on their first token only; if the score is based on the generated text, the decoding procedure must be specified. Because the option texts are listed inside the prompt, the score may measure surface-form selection or format adaptation rather than factual recall. Since every Stage A/B/C boundary in Figure 2, the instruction-format ordering in Figure 3, and the decay rates in Figure 4/Table 4 are derived from this score, the authors should state the exact matching rule (e.g., exact match on the full option string, first-token match, token-level probability of the correct option), and show that the judge and entailment metrics of Appendix A.5, which are only reported descriptively, agree with the accuracy-based stage assignments.","section":"Appendix A.6, Table 2, and Table 5"},{"comment":"The statistical support for the quantitative claims is fragile. The paper appears to report a single training run per condition, with no error bars, confidence intervals, or statement of the number of seeds. The two-way ANOVA in Table 3 reports residual df = 270, which is unexplained: with 3 instruction formats × 10 generations there are 30 cells, so a residual df of 270 implies 10 observations per cell, but the paper never states what these replicates are (10 random seeds? 10 evaluation questions? bootstrap resamples?). Similarly, Table 4 reports df = 1 for Generation and for the interaction, suggesting Generation was treated as a continuous covariate rather than a factor, while the text describes a two-way ANOVA; the design and replication structure should be clarified. Without independent runs, the 15× decay-rate comparison (−0.00054 vs −0.00837 per generation) rests on a single trajectory per condition and may not be statistically distinguishable from noise.","section":"Section 3, Figures 2–4, and Tables 3–4"},{"comment":"Table 5 contains a factual inconsistency that undermines the qualitative evidence for the claimed trajectories. The 100% synthetic column lists \"Escaping oppression at home [0.907]\" at Gen 0, which is the wrong answer, whereas the 25% and 50% columns show the correct answer \"Expanding territories under Spanish control [0.999]\" at Gen 0, and the main text and Figure 2 present Gen 0 as a shared baseline. If the 100% synthetic run already answers incorrectly with high confidence at Gen 0, then the observed trajectory is not solely a consequence of recursive synthetic training. The authors should correct this entry and re-check all qualitative tables for consistency with the reported baseline.","section":"Appendix C.1, Table 5"},{"comment":"The domain-specific mitigation experiment is partially circular. The corpus is constructed by semantically filtering WikiText segments against MMLU World Religions questions (Appendix A.7, using Sentence-BERT and a cross-encoder), and the evaluation is performed on the same MMLU World Religions subject (Figure 4 and Table 4). The improved collapse resistance may therefore reflect simple train/evaluation distributional similarity rather than a general property of domain-aligned synthetic training. To support the mitigation claim, the paper should either evaluate on a held-out subject whose questions were not used in corpus filtering, or demonstrate that the benefit transfers beyond the exact subject used to build the corpus.","section":"Appendix A.7 and Figure 4"},{"comment":"The Stage A/B/C definitions are not operationalized. The text states that Stage B is where \"factual accuracy deteriorates while task format adherence persists\" and Stage C begins when \"accuracy approaches random baselines (≤0.28)\", but no quantitative criterion is given for the Stage A→B transition, the Stage B→C transition, or where the vertical arrows in Figure 2 are placed. Without an explicit, pre-specified threshold or algorithm, the stage boundaries are not reproducible and the claim of a distinct three-stage phenomenon cannot be independently verified.","section":"Section 3, Figure 2"}],"minor_comments":[{"comment":"The checklist claims a \"subject sensitivity\" result with F(4,1960) = 5.92, p < 10⁻³, but this analysis does not appear in the main text or appendices; either add the corresponding analysis or remove the claim.","section":"NeurIPS Checklist, Item 1"},{"comment":"The caption of Figure 6 says judge scores are on a 1–5 scale, while Appendix A.5 defines the Judge Score as 1–3; this inconsistency should be corrected.","section":"Figure 6 and Appendix A.5"},{"comment":"Please correct the spelling \"ANOV A\" to \"ANOVA\" in the section header and body text.","section":"Appendix B.2, Table 3 and surrounding text"},{"comment":"The GEMMA 3 1B IT model is referenced as a model name without a citation; please cite the model card or technical report. Also, the appendix cites Sentence-BERT and the cross-encoder (all-MiniLM-L6-v2, all-roberta-large-v1) without references.","section":"Appendix A.1 and references"},{"comment":"The limitations paragraph mentions \"domain sensitivity\" and \"five MMLU subjects\" but does not explicitly state that all experiments used a single model scale and, apparently, a single seed per condition; this should be acknowledged in the limitations discussion.","section":"Limitations section"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and relevant problem, and the qualitative trajectory descriptions are suggestive. However, the central quantitative claims depend on an under-specified accuracy metric and on statistical evidence with unexplained replication structure. I would encourage the editor to require the authors to release the exact scoring code and to provide multi-seed error bars before considering the paper for publication. The mitigation experiment's circular construction is the most serious concern and should be addressed with an out-of-domain evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core phenomenon here is real and worth attention: recursively training a small instruction-tuned model on its own synthetic text can make it confidently wrong before it becomes incoherent. The paper also does something genuinely new by showing that the collapse trajectory depends on prompt format, with short-answer prompts staying stable longer than few-shot ones. That finding, plus the qualitative examples of degradation, is the best part of the paper. The domain-specific mitigation idea is sensible, and the appendices give enough detail that a determined reader could reproduce the setup.\n\nThe soft spots are concentrated where the quantitative claims live. The accuracy metric is underspecified: Table 2's caption says it was computed from the highest-probability answer token, but Appendix A.6 says MMLU items were reformatted into short-answer prompts that list full option texts. If the score is top-1 token matching, multi-word options are being scored on their first token, and the decline could be format adaptation rather than knowledge loss. The stress-test note is correct to call this load-bearing, because every stage boundary and the 15x mitigation ratio is computed from this score.\n\nOther problems: only one seed, no error bars anywhere; the ANOVAs have residual degrees of freedom that imply replication but the paper never says there were multiple runs; Table 5 shows the 100% synthetic column already wrong at Gen 0, which is inconsistent given Gen 0 is the same base model; and the NeurIPS checklist promises a subject-sensitivity analysis (F(4,1960)=5.92) that never appears in the main text or appendix. That is a real discrepancy. The 15x claim also comes from a single subject (World Religions) where the training corpus was semantically filtered against the evaluation questions, which is fine as a proof of concept but not as a general result.\n\nNone of this kills the paper's central idea. The three-stage pattern shows up across multiple metrics—perplexity, entropy, judge scores—not just the questionable accuracy number, so the qualitative trajectory is on solid ground. The flaws are fixable: release code and data, rerun with multiple seeds or bootstrap confidence intervals, clarify how accuracy is scored, and remove or substantiate the missing subject-sensitivity claim. If the authors do that, the format-dependence and mitigation findings would be a solid contribution to the synthetic-data literature.\n\nI'd send this to serious referees. A good reviewer could push the authors to make the empirical claims match the evidence, and the topic is important enough to justify the effort. My own verdict is that the paper is not acceptable as-is, but it deserves a chance to be revised rather than a desk reject.","headline":"A plausible and interesting study of recursive synthetic training causing confident factual errors, but the headline numbers rest on a fragile accuracy metric and a single seed; deserves serious refereeing, not acceptance as-is.","tokens_in":712,"tokens_out":1016,"would_cite":false,"duration_ms":34250,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Recursive training on synthetic data drives language models through a three-stage 'knowledge collapse' where factual accuracy erodes while fluency and confidence persist — and domain-matched corpora slow the decay about 15-fold.","keywords":["knowledge collapse","model collapse","synthetic data","recursive training","factual accuracy","instruction following","prompt format","domain-specific training"],"falsifier":"Re-run the same recursive-training loop but score every generation twice: exactly as the paper does (open-ended rewrite, highest-probability answer token) and on the original lettered multiple-choice items with full greedy decoding graded by exact match. If the lettered accuracy stays roughly flat while the open-ended score falls, the 'knowledge collapse' is at least partly an artifact of the reformatting; if both decline together while fluency metrics stay flat, the three-stage phenomenon is real.","tokens_in":17128,"feed_emoji":"📉","tokens_out":19463,"duration_ms":160209,"temperature":0.7,"pith_summary":"Recursive training on model-generated text, the paper argues, produces a failure mode it names knowledge collapse: factual accuracy erodes while fluency, formatting, and confidence survive, so models emit well-formed, confident, factually wrong answers. The claim is that collapse passes through three stages — knowledge preservation, then a dangerous 'confidently wrong' phase, then instruction-following breakdown — and that the speed and shape of the slide depend on the prompt format as well as the synthetic-data fraction. The paper further reports that restricting synthetic training to a domain-matched corpus delays factual decay by about 15-fold (accuracy falling at $-0.00054$ per generation versus $-0.00837$, with $p<0.001$ for the interaction) at no extra compute cost. If these claims hold, synthetic-data pipelines must monitor factual accuracy separately from fluency, and corpus alignment gives them a cheap lever to postpone the failure.","feed_headline":"Facts fail before fluency under recursive AI training","feed_subtitle":"Three-stage study pins down the 'confidently wrong' phase; domain-aligned data cuts the accuracy slide by ~15x.","key_machinery":"The load-bearing apparatus is the recursive fine-tuning loop together with a two-family evaluation that defines the stages. Each generation builds a training corpus from $(1-\\alpha)$ real 64-token prompts and $\\alpha$ model-generated continuations, applies a light half-epoch update, and evaluates the new checkpoint. Model-centric signals — perplexity, token entropy $H(T)=-\\sum_i p_i\\ln p_i$, and a gibberish-versus-coherent text score — measure surface fluency; task-centric signals — accuracy, greedy rate (the fraction of responses in which every token matches the model's most likely choice), and maximum option frequency (drift toward always picking one answer) — measure whether the model still knows the facts. Stage B is defined by the divergence between the two families: fluency metrics stay flat while accuracy falls and option bias climbs, which is why the phase is easy to miss. The mitigation experiment adds a third component: a domain-matched corpus built by embedding text segments, matching them to the evaluation subject's topics, and reranking the candidates, which narrows the distribution gap between training and evaluation for that subject.","core_discovery":"On the paper's own terms, the discovery is that knowledge collapse is a measurable phenomenon distinct from classical model collapse and from catastrophic forgetting. Using a 1-billion-parameter instruction-tuned model fine-tuned for half an epoch per generation on corpora that mix real text with synthetic continuations at rates $\\alpha \\in \\{0.25, 0.50, 1.0\\}$, the authors observe three stages: Stage A preserves factual accuracy and instruction adherence; Stage B keeps task format and confidence intact — greedy rate stays high and option bias climbs — while factual accuracy decays; Stage C collapses both, with accuracy near random and outputs incoherent. The trajectory differs by synthetic ratio, with 25% sustaining Stage A longest and 100% rushing through to Stage C, and it differs by instruction format, with few-shot prompts failing around generation 6, zero-shot around generation 7, and short-answer prompts holding to generation 8 (interaction $F(18,270)=12.67$, $p<0.001$); the authors also claim, in summarizing their contributions, that interpretive knowledge domains collapse faster than static factual ones (subject-generation interaction $F(4,1960)=5.92$, $p<10^{-3}$). A subject-matched synthetic corpus then anchors the distribution: accuracy decays at $-0.00054$ per generation versus $-0.00837$ for general training, perplexity grows to roughly 35 rather than 170, and entropy stays flat — evidence the authors read as distributional anchoring that preserves long-tail knowledge in the matched domain.","pith_inferences":["If distributional anchoring is the mechanism behind the 15-fold delay, the same benefit should reappear in any self-contained factual domain — medicine, law, finance, code — where a topic-matched synthetic corpus can be built; the paper tests only one subject, so this transfer is a prediction beyond its results.","The stage framework implies that model-reported confidence and lexical fluency are actively misleading during Stage B; deployments that gate on confidence or well-formedness will be blind to the exact failure mode the paper names, and periodic in-domain accuracy probes would be the safer monitoring choice.","The instruction-format data supports a default pipeline rule the paper stops short of stating: when recursive synthetic training is unavoidable, generate with constrained short-answer-style prompts and avoid few-shot exemplar templates, because exemplars add surface structure that accelerates overfitting.","The paper documents a spurious late rise in entailment-style similarity scores from verbose gibberish under 100% synthetic training; a practical corollary is that evaluation suites for recursively trained models should distrust semantic-overlap metrics once the instruction-following stage begins and rely on exact-grounded accuracy instead."],"forward_implications":["Higher synthetic fractions move the model through the stages faster: at 100% synthetic the model rushes through the confidently-wrong phase into incoherence, while at 25% it spends many generations fluently wrong, so a moderate mixing ratio does not avoid knowledge collapse; it only changes when and how fast the stages appear.","Prompt format is a dial on collapse timing: few-shot templates fail around generation 6, zero-shot around generation 7, and short-answer prompts around generation 8 under identical training, so evaluating a recursively trained model with a single prompt format will misstate how much knowledge remains.","Domain-matched synthetic corpora delay factual decay by about 15-fold within the matched subject, capping perplexity growth near 35 versus 170 and preserving vocabulary diversity, and the benefit comes from distributional alignment rather than from extra data or compute.","Fluency-oriented metrics such as perplexity, coherence, and the gibberish score can look healthy throughout Stage B, so they are not early-warning signals for factual erosion; task-level accuracy on a fixed probe set is the reliable detector among the metrics tested.","The concrete stage boundaries reported — accuracy near or below 0.28 for Stage C, stable greedy rate with rising option bias during Stage B — give other labs a recipe for reproducing and detecting knowledge collapse in their own recursive-training runs."],"supporting_citations":[{"why":"It establishes the baseline model-collapse phenomenon — recursive training converging toward repetitive outputs — that knowledge collapse is defined against.","marker":"[16]"},{"why":"It shows strong collapse can occur even at small synthetic fractions, anchoring the paper's theoretical framing of why synthetic recursion is dangerous.","marker":"[2]"},{"why":"It treats collapse as a change of scaling laws, cited for the claim that adding synthetic data eventually stops improving performance.","marker":"[3]"},{"why":"It relates distribution shift to the synthetic-data fraction, motivating the paper's sweep over mixing ratios.","marker":"[15]"},{"why":"It supplies the multi-subject multiple-choice benchmark from which the five factual-recall evaluation domains are drawn.","marker":"[6]"},{"why":"It supplies the encyclopedia-style text corpus used to build the real prompts and the synthetic continuations.","marker":"[11]"},{"why":"It provides the mechanism by which few-shot exemplars become corrupted under recursive training, plus the instruction-fidelity baseline the mitigation is compared with.","marker":"[10]"},{"why":"It offers earlier evidence that domain-specific synthetic training preserves accuracy in specialized areas, the prior result the mitigation experiment extends.","marker":"[22]"}],"fun_headline_variants":["Knowledge collapse: facts fade as fluency persists","Three-stage collapse: LLMs turn confidently wrong","Prompt style shifts when AI knowledge collapse hits","Domain-matched data slows AI knowledge decay ~15x","Synthetic training ratio steers LLM knowledge collapse"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument hinges on the rewritten questions' accuracy score actually measuring factual knowledge, because the benchmark items are converted from lettered multiple choice into open-ended form and scored from a single most-probable answer token — if that rewriting changes the difficulty of the questions, the measured decline could be format adaptation rather than knowledge collapse.","fun_headline_variants_meta":{"raw":{"variants":["Knowledge collapse: facts fade as fluency persists","Three-stage collapse: LLMs turn confidently wrong","Prompt style shifts when AI knowledge collapse hits","Domain-matched data slows AI knowledge decay ~15x","Synthetic training ratio steers LLM knowledge collapse"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1280,"prompt_tokens":1029,"completion_tokens":251,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":645,"completion_tokens_details":{"reasoning_tokens":180}},"tokens_in":645,"tokens_out":251,"duration_ms":3124,"temperature":1.0,"reasoning_tokens":180,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:27:34.982548+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same recursive-training loop but score every generation twice: exactly as the paper does (open-ended rewrite, highest-probability answer token) and on the original lettered multiple-choice items with full greedy decoding graded by exact match. If the lettered accuracy stays roughly flat while the open-ended score falls, the 'knowledge collapse' is at least partly an artifact of the reformatting; if both decline together while fluency metrics stay flat, the three-stage phenomenon is real.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It establishes the baseline model-collapse phenomenon — recursive training converging toward repetitive outputs — that knowledge collapse is defined against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It treats collapse as a change of scaling laws, cited for the claim that adding synthetic data eventually stops improving performance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the multi-subject multiple-choice benchmark from which the five factual-recall evaluation domains are drawn."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the mechanism by which few-shot exemplars become corrupted under recursive training, plus the instruction-fidelity baseline the mitigation is compared with."},{"cited_title":"Which term refers to enlightened beings in Buddhism? (A) Arhats (B) Bodhisattvas (C) Mahayana (D) TheravadaAnswer:","cited_arxiv_id":null,"evidence_quote":"It offers earlier evidence that domain-specific synthetic training preserves accuracy in specialized areas, the prior result the mitigation experiment extends."}],"review_version":2}