{"id":"07dcd597-a096-4183-992c-4ba54d99c5cc","arxiv_id":"2608.04268","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Fairness collapse: gender-occupation bias worsens in the first iterations of recursive synthetic-data training, ahead of visible model-collapse indicators.","lead":"Models that are repeatedly trained on text they generated themselves become more gender-biased in early rounds, before standard quality scores clearly drop. The paper names this effect 'fairness collapse' and demonstrates it on a small language model using a professional biography dataset.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'fairness before collapse' claim rests on single-run EO-GAP changes and on training-set perplexity that must improve; Table 1's few-shot EO GAP ends back at baseline, so 'consistent amplification' is not established.","rationale":"I read the paper as a controlled empirical study whose central claim is that fairness degradation is an earlier and separate failure mode than model collapse. The strongest evidence would be a consistent ordering at early iterations across multiple settings. The authors disclose limitations about dataset, model scale, generation strategies, and metric scope, but not about seed variance or held-out perplexity. The reader's weakest assumption, that single-run noise could explain the 2 to 7 point EO GAP changes, is valid and I agree it is load-bearing. However, I think two additional issues are at least as important: the in-distribution training-set PPL cannot play the role the argument assigns to it, and the paper's own Table 1 few-shot recursive trajectory contradicts 'consistently amplifies' by ending at the baseline EO GAP value. In good faith, the paper has real strengths: the generation-quality analysis, profession-level tables, and appendices are detailed, and the phenomenon may well exist. But the central claim is not yet established by the reported single-run results, so the conditional verdict remains appropriate rather than acceptance or rejection.","tokens_in":25074,"tokens_out":6139,"duration_ms":59301,"concrete_test":"Re-run the seeded-recursive and few-shot-recursive conditions with at least 10 random seeds; for each iteration report mean and 95% confidence interval for EO GAP, NLL GAP, MMLU, and perplexity on the held-out human test split. Test whether the Iteration-1 EO GAP increase over Iteration 0 is significant with a paired test, and test whether held-out human PPL has already increased at that same iteration. If the EO GAP increase is not significant, or if held-out human PPL rises at Iteration 1, the 'fairness degrades before standard metrics' claim as stated in Section 1 is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.3 reads the aggregate EO GAP trajectories as showing that fairness degradation 'precedes classical model collapse,' but two of the three load-bearing supports are not secure. First, Table 1 (few-shot recursive) is internally inconsistent with the 'consistently amplifies' claim: EO GAP is 13.18 at Iteration 0, rises to 16.36/18.82, then falls to 13.45 at Iteration 5, ending at the baseline value. The headline pattern therefore rests on the early-iteration subset, not on a consistent trajectory. Second, the reported PPL 'improvement' (16.07 to 10.55 in seeded recursive training) is computed on the training data used at the corresponding iteration, as stated in the Table 1 and Table 2 captions; that means the PPL is evaluated on model-generated text from the model being trained. A decreasing training-set PPL is the expected memorization effect and cannot be cited as evidence that standard language-modeling metrics are blind to the change. Third, every condition appears to be run once: no random seeds, standard deviations, or significance tests are reported, so the 2 to 3 percentage point Iteration-1 EO GAP increases (for example, 13.18 to 15.84 in seeded recursive training) are not distinguishable from run-to-run variance. The central claim in the abstract, that bias increases 'silently before strong indicators of model collapse become apparent,' requires held-out human perplexity and replicated runs to discriminate fairness collapse from ordinary distributional collapse of minority demographic subgroups.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether continued pretraining on synthetic text amplifies social biases before standard measures of model collapse degrade. Using Bias in Bios, the authors construct balanced human-written data, generate synthetic biographies by seeded continuation and few-shot prompting with Qwen2.5-0.5B, and train under iterative and recursive feedback regimes. They report EO GAP, NLL gaps, CrowS-Pairs, SoFA, MMLU, and training-set perplexity across five iterations. The central claim is that fairness degradation emerges before severe language-modeling degradation, and that this 'fairness collapse' is a distinct early failure mode. The paper includes controlled comparisons, external fairness benchmarks, held-out evaluation, and anonymous code.","tokens_in":25400,"tokens_out":3271,"duration_ms":31100,"significance":"If the central claim were established, the paper would make a useful contribution: it identifies a failure mode that standard perplexity and accuracy monitoring would miss, and it connects the model-collapse literature with fairness evaluation. The controlled setup, the use of external benchmarks (CrowS-Pairs, SoFA, MMLU, held-out Bias in Bios test data) rather than fitted parameters, and the clear separation of iterative versus recursive regimes are strengths. The core evidence, however, is currently too fragile to support the strength of the claims: the tables report single runs without error bars, the few-shot recursive trajectory is nonmonotonic and ends at baseline, and the perplexity metric is computed on the training distribution, so it cannot establish that standard metrics are blind to the degradation. The phenomenon may well be real, but the manuscript needs additional statistical support and a revision of the central claims to match the evidence.","major_comments":[{"comment":"All conditions appear to be run once, with no random seeds, standard deviations, or significance tests. The EO GAP changes that drive the paper's conclusion are 2–7 percentage points, and some trajectories are nonmonotonic. In particular, Table 1 (few-shot recursive) shows EO GAP rising from 13.18 at Iteration 0 to 16.36 at Iteration 1 and 18.82 at Iteration 2, then falling back to 13.45 at Iteration 5, essentially the baseline value. This directly contradicts the 'consistently amplifies' claim in Section 1 and Section 5.3. The authors should report multiple seeds with confidence intervals or at minimum show that a single-run variance estimate cannot explain the ordering they claim.","section":"§5.3, Tables 1–2"},{"comment":"The perplexity values are computed on the training data used at the corresponding iteration, as the captions state. For recursive training this is model-generated text from the model being trained, so a decreasing training-set PPL from 16.07 to 10.55 is the expected memorization effect and cannot be cited as evidence that standard language-modeling metrics fail to detect the change. The claim that bias increases 'silently before strong indicators of model collapse become apparent' requires a held-out metric, such as perplexity on human-written text or on a fixed validation split, reported across iterations for both synthetic and human-only regimes.","section":"Tables 1–2, captions; §5.3"},{"comment":"The temporal claim that fairness degradation precedes classical model collapse is not supported by the reported aggregate trajectories. In Table 1, MMLU drops from 42.14 to 32.08 under seeded recursive training and from 42.14 to 24.63 under few-shot recursive training by Iteration 5, which is a substantial general-capability decline, not a stable baseline. Moreover, the human-only baseline in Table 15 also shows accuracy declining from 80.05 to 74.28 and NLL GAP shifting from -0.093 to -0.103. Without confidence intervals, it is unclear whether the synthetic-data regimes produce effects beyond the human-only training drift or beyond run-to-run variation. The authors should define the 'before' comparison quantitatively, for example, by testing whether fairness metrics change significantly at Iterations 1–2 while held-out language-modeling metrics do not.","section":"§5.3, Table 1 vs. Table 15"}],"minor_comments":[{"comment":"There are several typographical artifacts, including 'V olume' in references, 'T oken Length' in Figure 5, and 'F oods' in Table 14; these should be corrected.","section":"Throughout"},{"comment":"The caption is grammatically overloaded; splitting it into separate labeled panels would make the left, center, and right subplots easier to read.","section":"Figure 3 caption"},{"comment":"The choice of seed length 30 and temperature 0.9 is motivated by the generated-data evaluation, but the paper does not report whether the qualitative fairness-collapse results are sensitive to these generation hyperparameters; a brief sensitivity discussion would strengthen the claims.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would be suitable for the journal if the authors can provide replicated runs or otherwise bound the single-run variance, and if they replace training-set perplexity with a held-out metric. The core idea is worth publishing, but the current evidence is not yet strong enough for the abstract's 'silently before strong indicators' claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper names a real, testable phenomenon—fairness collapse—and gives it a plausible mechanism, but the headline claim that bias \"silently\" rises before model collapse is not yet supported by the evidence as reported. The central numbers come from single runs, and one of the two recursive conditions contradicts the \"consistently amplifies\" claim.\n\nWhat's new and good: the setup is clean. Balanced Bias-in-Bios subset, controlled iterative vs. recursive regimes, seeded vs. few-shot generation, and evaluation on external benchmarks (CrowS-Pairs, SoFA, MMLU) that are not used for training. The likelihood-based NLL gap and SoFA do rise fairly monotonically in the recursive conditions, which is more convincing than the EO GAP story. The paper is also honest about its scope in the limitations section. If the effect is real, it is practically important: standard perplexity/accuracy monitoring would miss it.\n\nSoft spots: first, every condition appears to be run once. No seeds, no error bars, no significance tests. The 2–3 point EO GAP increases in early iterations are within plausible run-to-run noise for this metric. Second, the reported PPL \"improvement\" (16.07 to 10.55) is computed on the training data at each iteration, i.e., on the model's own generated text. Decreasing training-set PPL is expected memorization; it cannot be used to claim that standard LM metrics are blind. Third, Table 1's few-shot recursive condition has EO GAP end at 13.45, essentially back to the iteration-0 baseline (13.18), after peaking at 18.82 at iteration 2. That directly contradicts \"consistently amplifies.\" The seeded recursive condition does show a sustained rise (13.18 to 19.38), so the central pattern survives in one regime, but the abstract overgeneralizes. Fourth, the paper cites Xu et al. (2025) showing collapsed models promote fairness but never directly reconciles that opposite result; saying \"it remains unclear whether exposure amplifies or attenuates\" is not enough when your own data all point one way.\n\nBottom line: this is a worthwhile hypothesis-generating study, not a confirmed phenomenon. The fix is standard: multiple seeds, held-out perplexity, and a direct comparison to Xu et al. I would send it to peer review—the question matters, the experimental design is mostly sound, and the limitations are disclosed—but I would expect reviewers to require re-runs before acceptance.\n\nRecommendation: engage with it, but as a revise-and-resubmit, not as a citable result yet.","headline":"Fairness collapse is a plausible and cleanly framed empirical claim, but the single-run evidence and in-distribution perplexity cannot yet support the 'silent before collapse' headline; worth refereeing with re-runs required.","tokens_in":25919,"tokens_out":2249,"would_cite":false,"duration_ms":19808,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Recursive training on synthetic text widens gender-occupation bias before standard quality metrics show trouble.","keywords":["fairness collapse","model collapse","synthetic data","gender bias","bias amplification","recursive training","language models","Bias in Bios"],"falsifier":"Repeat the recursive seeded training protocol with, say, ten random seeds per iteration and compute confidence intervals on the aggregate EO GAP; if the spread across seeds at any iteration exceeds the reported gap increase (roughly 13 to 19 percentage points by iteration 5), then fairness collapse is not distinguishable from noise. A second check: train the same model on human-written data alone with the same number of repeated runs and show the EO GAP stays within a narrower band than the synthetic conditions.","tokens_in":24864,"feed_emoji":"⚖️","tokens_out":6083,"duration_ms":48765,"temperature":0.7,"pith_summary":"This paper proposes and tests a failure mode it calls fairness collapse: when a language model is repeatedly retrained on text it generated itself, gender-occupation associations grow stronger before conventional signs of model collapse appear. Using a balanced subset of the Bias in Bios biographies and a 0.5B-parameter model, the authors run five iterations of continued pretraining under four regimes—seeded or few-shot generation, each in iterative or recursive form—and track fairness metrics (equal-opportunity gaps, likelihood asymmetries, stereotype benchmarks) alongside standard quality metrics (perplexity, MMLU, classification accuracy). They find that the equal-opportunity gap rises from 13.18 to 19.38 under recursive seeded training while perplexity falls from 16.07 to 10.55, meaning the model gets better at predicting its training text at the same time as its demographic fairness worsens. The claim matters because synthetic data is increasingly common in web-scale corpora, and if bias amplifies silently, standard monitoring would not catch it.","feed_headline":"Synthetic-data training widens gender bias before models falter","feed_subtitle":"Fairness collapse sets in while perplexity keeps improving, so standard quality checks miss the damage.","key_machinery":"The load-bearing object is the recursive training loop the authors call fairness collapse: a model generates synthetic biographies (either continuing a human-written seed or following a few-shot prompt), the next model is trained on those generations, and the process repeats. The argument is carried by the aggregate Equality of Opportunity gap (EO GAP), the root-mean-square across professions of the difference in true-positive rates for male versus female biographies in a pairwise occupation-prediction task, together with signed gender-conditioned negative log-likelihood gaps and external stereotype benchmarks (CrowS-Pairs and SoFA). These fairness measures move in the direction of greater demographic asymmetry across iterations while perplexity on the training text keeps improving, which is the evidential core that separates fairness collapse from model collapse. The key comparison is between the recursive regime, where the same checkpoint is retrained and regenerates its own data, and the iterative regime, where each new model starts from the human-trained checkpoint; the recursive loop amplifies bias more strongly.","core_discovery":"The central claim, stated in the introduction and supported by Tables 1 and 2, is that fairness degradation emerges before severe language-model degradation. In controlled continued-pretraining experiments on the Bias in Bios corpus, exposure to synthetic data consistently amplifies gender-occupation bias—measured by the aggregate equal-opportunity gap rising from 13.18 to 19.38 in recursive seeded training—while perplexity simultaneously improves from 16.07 to 10.55 and MMLU accuracy declines only gradually, from 42.14 to 32.08. The same pattern appears in the few-shot variant and in the iterative regime, though weaker and less monotonic, with SOFA bias scores increasing from 0.509 to 1.020 under recursive seeded training. The authors interpret this as evidence of a distinct early-stage failure mode, fairness collapse, in which biased associations are progressively reinforced by recursive training on model-generated text, independent of and earlier than classical model collapse.","pith_inferences":["If fairness collapse generalizes beyond Bias in Bios, a testable prediction is that synthetic-data pipelines trained on other stereotype-laden domains (race, age, language) will show measurable bias growth before distributional collapse; the paper does not test this.","The results imply that data curation for recursive training should include fairness filtering or balanced regeneration, not just quality or diversity filtering—an extension the authors do not pursue.","A direct experimental extension would be to check whether retaining a fixed fraction of human-written data, which prior work shows slows model collapse, also slows or halts fairness collapse; the paper does not run this condition.","Because each condition appears to be run once, the quantitative gaps in the tables should be treated as indicative; re-running with multiple seeds is the obvious next step before using these numbers as benchmarks."],"forward_implications":["Standard language-modeling monitoring—perplexity, MMLU, downstream accuracy—can improve or stay flat while demographic bias is growing, so these metrics are not sufficient early-warning indicators.","Recursive or self-consuming training loops are more dangerous for fairness than iterative regeneration from a human checkpoint, because cumulative parameter updates compound the bias.","Synthetic data contamination of real training corpora may silently reinforce gender-occupation stereotypes before any model-collapse symptoms are visible.","Fairness metrics such as EO GAP and likelihood asymmetry should be tracked alongside quality metrics when training on data that may contain model-generated text.","The amplification is directional: female-associated professions like nurse drift toward more negative EO gaps and male-associated professions like professor toward positive ones, showing existing corpus associations are strengthened rather than new noise introduced."],"supporting_citations":[{"why":"Supplies the Bias in Bios corpus and the gender-occupation fairness task that all training and evaluation build on.","marker":"De-Arteaga et al., 2019"},{"why":"Defines model collapse, the phenomenon this paper extends by asking whether fairness degrades first.","marker":"Shumailov et al., 2023"},{"why":"Provides MMLU, the general language-modeling benchmark used to argue fairness collapse precedes classic degradation.","marker":"Hendrycks et al., 2021"},{"why":"Supplies CrowS-Pairs, one of the external stereotype benchmarks showing likelihood asymmetry rises.","marker":"Nangia et al., 2020"},{"why":"Supplies SoFA, the social-fairness probe whose bias score increases across recursive iterations.","marker":"Marchiori Manerba et al., 2024"},{"why":"Prior finding that synthetic-text training reduces linguistic diversity, extended here to fairness.","marker":"Guo et al., 2024"}],"fun_headline_variants":["Bias grows before models break in synthetic-data training","Fairness collapse: gender bias spikes while perplexity improves","Synthetic data silently amplifies bias before model collapse","Gender bias amplifies first, then model collapse follows"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 2–7 percentage point movements in the equal-opportunity gap are genuine signal and not run-to-run noise; each training condition appears to have been run once, with no random seeds or significance tests reported, so if single-run variance is that large the claimed ordering could disappear.","fun_headline_variants_meta":{"raw":{"variants":["Bias grows before models break in synthetic-data training","Fairness collapse: gender bias spikes while perplexity improves","Synthetic data silently amplifies bias before model collapse","Gender bias amplifies first, then model collapse follows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1215,"prompt_tokens":924,"completion_tokens":291,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":227}},"tokens_in":540,"tokens_out":291,"duration_ms":3161,"temperature":1.0,"reasoning_tokens":227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:40:48.823718+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the recursive seeded training protocol with, say, ten random seeds per iteration and compute confidence intervals on the aggregate EO GAP; if the spread across seeds at any iteration exceeds the reported gap increase (roughly 13 to 19 percentage points by iteration 5), then fairness collapse is not distinguishable from noise. A second check: train the same model on human-written data alone with the same number of repeated runs and show the EO GAP stays within a narrower band than the synthetic conditions.","supporting_citations":[],"review_version":1}