Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a network of LLMs sharing a retrieval-augmented memory converges to nearly identical outputs, and proves the exact analogue for a system of Gaussian mixture models.

desk verdict The LWD framework is a useful, cheap testbed for network-level model collapse, but the claimed theoretical guarantee is a sketch with real gaps and the LLM result is a single run without error bars. read the letter →

arxiv 2506.15690 v3 pith:FZU5CMSS submitted 2025-05-26 cs.LG cs.AIcs.SIstat.ME

Tianyu Wang , Akira Horiguchi , Lingyou Pang , Carey E. Priebe This is my paper
classification cs.LGcs.AIcs.SIstat.ME
keywords modelcollapselargelanguagemodelsretrieval-augmentedgenerationGaussianmixturepairwisedistancematrixsyntheticdatanetworkdynamicsembeddingsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies model collapse at the level of a whole network of language models rather than a single model being retrained on its own output. It sets up a shared, growing 'Internet' of sentences: at every step each of several LLMs samples a fixed fraction of that database uniformly at random, answers a fixed query, and posts its answer back to the database. The paper measures collapse by embedding all answers and tracking the Frobenius norm of the pairwise distance matrix across models. It reports that this norm falls from roughly 20 to about 5 in a Bitcoin-query experiment with three models, and it proves for an analogous system of Gaussian mixture models that the norm converges to zero in expectation and in probability. If correct, the result means that a community of RAG-augmented LLMs fed on their own synthetic output will drift to a single consensus response, regardless of how different the models started.

What carries the argument

The load-bearing object is the n by n pairwise distance matrix D(t) whose (i,j) entry is the Euclidean distance between the mean embeddings of model i's and model j's responses at time t; its Frobenius norm is the collapse metric. The dynamics are driven by a shared, monotonically growing retrieval set A(t) in which each model samples k_t = floor($\beta$ times |A(t)|) entries uniformly at random, so the contexts available to all models become progressively more identical as synthetic answers accumulate. For the GMM proof, the machinery is a recursive mixture-weight update with ownership probabilities; taking alpha_t = 1/k_t makes the expected difference between two models' weights shrink by a factor (1 - alpha_t)^{k_t} tending to $e^{{-1}}$ per time step, so the product over t tends to zero. The proof also relies on a well-separated-components approximation that replaces the likelihood ratio by an indicator at the component means.

What would settle it

Run the same LWD experiment with retrieval changed from uniform random sampling of a fixed beta fraction to top-k relevance retrieval (for example k = 5 to 10) over the same growing database, keeping all other settings fixed; if the Frobenius norm of the pairwise distance matrix plateaus above zero or stops decreasing, the claimed convergence is an artifact of the uniform-sampling update. For the GMM proof, the analogous check is to make the component means overlap or the variances large, breaking the well-separated-components approximation in the proof; if pairwise weight differences then fail to vanish, the approximation is the step that carries the theorem.

Watch

Extended reading notes

Core claim

The central claim is that network-level model collapse is real, measurable, and provable in a simplified setting. In the LWD framework, n LLMs with different pretraining backgrounds repeatedly retrieve from and write to a shared text database; the authors conjecture that the distributions of the models' responses to a fixed query become increasingly similar, and that the Frobenius norm of the pairwise Euclidean distance matrix of embedded average responses converges in probability to a small nonnegative constant. For the equivalent GMM system, with components having fixed means and covariances and only mixture weights evolving by a recursive ownership update, the paper proves the stronger statement that pairwise weight differences vanish, and therefore the distance-matrix norm converges to zero in expectation and in probability, for any initial weights strictly between 0 and 1. The LLM version is left as a conjecture rather than a theorem because a transformer is argued to occupy an effectively infinite-dimensional function space.

Load-bearing premise

The load-bearing premise is that each model retrieves a fixed fraction beta of the entire shared database uniformly at random at every step, so all models read increasingly identical contexts; real retrieval is relevance-ranked and typically reads a small window, and if that premise gives way the convergence may weaken or disappear.

Editorial extensions

If this is right

  • A network of RAG-augmented LLMs sharing one memory will lose cross-model diversity: after enough synthetic self-posting, all agents answer the same query with nearly the same sentence.
  • The Frobenius norm of the pairwise embedding distance matrix gives an API-only, model-agnostic monitor of ecosystem-level collapse that does not require retraining or fine-tuning any model.
  • The GMM theorem says collapse in this setup is inevitable for any starting mixture weights between 0 and 1, so diversity at initialization does not protect the network once a shared synthetic memory dominates.
  • The GMM proxy can be used to explore collapse scenarios cheaply before running expensive LLM experiments, with larger component counts giving trajectories closer to LLM behaviour.
  • Because convergence to an information-neutral equilibrium is not labelled good or bad, the metric can be used as a neutral stability certificate for whether further synthetic training changes communication dynamics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's setup, relevance-ranked retrieval with a small k is the realistic regime; one testable extension is to run LWD with top-k retrieval and predict that the convergence rate slows and the limiting norm stays above zero.
  • The proof's separation approximation and the simplifying assumption of an empty initial database together suggest the GMM guarantee may fail when components overlap or when a substantial pool of human-written text remains in A(t); relaxing these is a natural stress test.
  • The distance-matrix norm as defined averages L responses per model, so it measures both cross-model convergence and within-model loss of diversity at once; separating these two components would tell whether models converge to a shared point or merely each collapse to their own narrow region.
  • If the conjecture extends beyond three models, the framework implies a feedback loop: synthetic text on the Internet does not just bias individual models, it pushes entire heterogeneous ecosystems toward one dominant style and factual frame.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes LLM Web Dynamics (LWD), a network of n LLMs that share a retrieval-augmented memory: at each step every model retrieves a fraction beta of the shared database uniformly at random, generates L responses to a fixed query, and posts one response back into the database. The authors define a pairwise embedding-distance matrix D(t) and conjecture that its Frobenius norm decreases to a small value, indicating collapse of response diversity. To give theoretical support, they introduce an analogous system of Gaussian mixture models with fixed component parameters and time-varying mixture weights, conjecture that ||D(t)||_F converges to 0 in probability (Eq. (5)), and provide a proof sketch in Appendix A. Experiments show one LLM run (three models, one query, T=60) with a norm decrease from roughly 20 to roughly 5, and GMM simulations with B=2 (five replicates) and B=11 (one replicate) that converge to near zero. The paper explicitly lists limitations and positions the GMM system as a cost-efficient proxy.

Significance. If the norm decrease is real and the GMM convergence can be rigorously established, LWD would be a useful low-cost testbed for studying synthetic-data feedback loops in LLM networks without iterative retraining. The choice of a fixed query and a shared RAG memory is transparent, and the GMM analogue has a clean mathematical structure worth analyzing. The paper is also honest about its limitations. However, the significance is currently limited by two substantial issues: the LLM experiment is a single run with no statistical support, and the Appendix A proof of the central GMM claim is an uncontrolled sketch. The framework is promising but, as it stands, neither the empirical nor the theoretical central claim is established.

major comments (5)
  1. [Section 4.1, Figure 2] The central empirical claim that the Frobenius norm of the LLM distance matrix decreases from approximately 20 to approximately 5 is based on a single run with one query, n=3, and no error bars or multiple seeds. Figure 2 alone does not rule out sampling noise or prompt-specific behavior, and the paper's own Section 6 concedes that the results need to be transformed into a hypothesis-testing framework. Please report repeated runs with confidence intervals and a formal statistical test, or explicitly label the LLM result as an illustrative anecdote in the abstract and conclusions.
  2. [Appendix A, Eqs. (7)-(9)] The proof of Eq. (5) replaces the likelihood ratio g_{i'}^{(t')}(u)/g_i^{(t)}(u) by Dirac masses at component means, justified only by an informal 'small variance'/'well-separated' assumption, while the model fixes a nondegenerate covariance matrix Sigma. No error bound is given, and the approximated expected update is exactly what produces the contraction factor (1-alpha_t)^{k_t}. Since the covariance is fixed, this approximation error need not vanish, and the theorem as stated does not apply to Algorithm 2. The proof also explicitly assumes A(0) is empty and omits the positivity/normalization step (Algorithm 2, lines 7-14), both of which are present in the simulations of Section 4.2. This is load-bearing because Eq. (5) is the only theoretical guarantee advertised in the abstract.
  3. [Appendix A, Eq. (10)] The step from lim_t E[pi_{i,b}^{(t)} - pi_{i',b}^{(t)}] = 0 to lim_t E|pi_{i,b}^{(t)} - pi_{i',b}^{(t)}| = 0 is invalid for signed differences; a random variable can have expectation approaching zero while its absolute value stays bounded away from zero. The subsequent bound on E||D^{(t)}||_F^2 relies on this step, so the convergence in probability of ||D^{(t)}||_F is not established. A repair would require a mean-square contraction or an explicit variance bound, not just convergence of the signed expectation.
  4. [Abstract and Section 3.2] The statement that LWD provides 'theoretical guarantees for this convergence' overstates the results. Section 3.1 explicitly leaves the LLM norm decrease as a conjecture, and the only attempted proof concerns the GMM analogue; moreover, Section 5 acknowledges that the LLM limit is a small positive c while the GMM limit is zero. The claims should be reworded so that the GMM result is presented as a heuristic or provisional analysis and the LLM behavior remains conjectural.
  5. [Section 3.1, retrieval model] The uniform sampling of kt = floor(beta |A(t)|) sentences with beta=0.5 means each model sees a large random fraction of the entire shared memory, so the contexts of all models become increasingly similar by construction. This is not how RAG systems retrieve in practice, which use top-k relevance and small context windows. The paper should discuss this modeling restriction when claiming LWD mirrors real-world Internet dynamics, and the sensitivity of the collapse pattern to beta and to the retrieval rule should be investigated.
minor comments (5)
  1. [Equation (6)] The notation is inconsistent: the numerator uses c_B while the denominator uses c_{t,B}, and the ownership term o_{i,b}^{(t)}(u) is not defined consistently with Algorithm 2, where the ownership uses the partially updated pi^{(t+1)}. Please harmonize the notation.
  2. [Figure 2] The figure lacks axis labels and a legend; the caption should state what is plotted and for which model configuration, and the 'norm' should be identified as the Frobenius norm of D(t).
  3. [Appendix B] The example posts are labeled t=1 and t=60, but the text refers to them as 'initial' and 'at T'; since retrievals begin at t=0, t=1 is already after one update, so the wording should be adjusted.
  4. [Section 2 and Acknowledgments] There are several typographical errors, including 'Artifical Intelligence' in the acknowledgments and 'V AEs' in Section 2; the Dubey et al. reference also truncates the author list as 'and 1 others'.
  5. [Section 4.2, Figure 6] The B=11 result in Figure 6 is a single replicate, and the sudden fall around t=160 is discussed as if it were generic; please clarify that this is one trajectory and not a replicated finding.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the GMM convergence is an approximate derivation from the update rule, and the LLM claim is an explicitly conjectural analogy, not a renamed input.

full rationale

The central formal claim, Eq. (5), is derived in Appendix A from the Algorithm 2 update rule by taking conditional expectations and exhibiting a contraction in pairwise weight differences; the proof does not assume its conclusion. The uncontrolled aspects are all rigor/correctness issues rather than circularity: the appendix is candidly titled "Sketch of Proof," assumes A(0) is empty, replaces a smooth likelihood ratio by a Dirac sum without a quantitative error bound, and passes from convergence of expectations of signed differences to L1 convergence without justification. These would be objections to the validity of the proof, not evidence that the conclusion is built into the premises. The LLM convergence result is explicitly stated as a conjecture in Section 3.1 and is not derived from the GMM theorem; the GMM system is presented as an analogue, so there is no reduction-by-construction from LLM outputs to the theoretical result. The only self-citations (Wang et al. 2025 as motivation for the GMM proxy and Helm et al. 2024 for DKPS dimension reduction) are background or methodological and are not load-bearing for the proof in Appendix A. Section 6 also candidly limits the LLM findings to pattern learning and calls for further statistical inference, which is consistent with the claim being empirical and conjectural rather than circular. Overall, no load-bearing circular step is present; the score reflects only the presence of minor self-citations that do not support the derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on several assumptions that are either chosen by hand (beta, GMM component separation) or asserted by analogy (GMM proxy for LLMs, uniform retrieval as a stand-in for RAG). The proof also assumes an empty initial pool, which differs from the LLM experiment.

free parameters (3)
  • beta (retrieval fraction) = 0.5
    Chosen fixed hyperparameter in both LLM and GMM experiments. Controls how much of the shared database each model reads each round. The convergence result depends on this being a positive constant; changing it changes the speed and possibly the existence of collapse.
  • L (responses per model per round) = 40 (LLM), 3 (GMM)
    Number of responses generated per model per time. Design choice for estimating the response distribution; not fitted, but affects the variance of the distance matrix estimates.
  • GMM component parameters (means, variances) = mu = +/-5, variances = 1
    Fixed for the B=2 simulation. The theoretical proof assumes these components are well separated, so the values are chosen to satisfy the approximation, not learned from data.
assumptions (5)
  • ad hoc to paper Well-separated mixture components with small variances
    Invoked in Appendix A: 'if the mixture variances are small enough ... then we can make the approximation g_i'/g_i ≈ ...'. The theorem's contraction argument relies on this approximation; it is not proven to hold for general component overlap.
  • ad hoc to paper Empty initial real-data pool A(0)
    The proof assumes A(0) is empty (Appendix A) so that every retrieved point comes from a known GMM. The LLM experiment starts with 20 human posts, so the theorem's setting does not exactly match the experiment.
  • domain assumption Uniform random retrieval approximates RAG
    Section 3.1: 'we draw kt sentences uniformly at random from this set each time'. Real RAG retrieves by relevance; the uniform assumption is what makes all models' contexts become shared and drives convergence.
  • domain assumption Embedding distances reflect semantic similarity
    The metric uses nomic-embed-v1.5 distances as a proxy for semantic similarity of LLM outputs. The paper does not validate this against human judgments or downstream measures.
  • domain assumption LLM response distributions can be approximated by GMMs with fixed components and only weights updated
    Section 3.2 states this analogy; the theoretical guarantee applies to the GMM proxy, and the transfer to LLMs is by analogy, not proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs." pith.science (2026). https://pith.science/paper/FZU5CMSS

@misc{pith2026250615690,
  author       = {Pith},
  title        = {Pith review of: LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FZU5CMSS}},
  note         = {Machine review of arXiv:2506.15690}
}
read the original abstract

The increasing use of synthetic data from the public Internet has enhanced data usage efficiency in large language model (LLM) training. However, the potential threat of model collapse remains insufficiently explored. Existing studies primarily examine model collapse in a single model setting or rely solely on statistical surrogates. In this work, we introduce LLM Web Dynamics (LWD), an efficient framework for investigating model collapse at the network level. By simulating the Internet with a retrieval-augmented generation (RAG) database, we analyze the convergence pattern of model outputs. Furthermore, we provide theoretical guarantees for this convergence by drawing an analogy to interacting Gaussian Mixture Models.

Figures

Figures reproduced from arXiv: 2506.15690 by the authors.

Figure 1
Figure 1. Experiment in Section 3.1, from t to t + 1. At time t, three different LLMs generate responses based on augmentation contexts fetched from a shared text database. They then post their responses to the text database, based on which they generate the next-time responses. fined as D (t) ij := [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. As t approaches 60, the norm of the distance matrix in our LLM experiment, i.e [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. As t increases, the norm of the distance matrix in our GMM experiment (with B = 2), i.e [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: Scatter plots of the embedded responses from [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]
Figure 6
Figure 6. Figure 6: As [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 5
Figure 5. Figure 5: Histograms with density curves for three [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Epistemic diversity across language models mitigates knowledge collapse

    cs.LG 2025-12 reject novelty 5.0 of 10

    In repeated self-training loops on Wikitext2, ecosystems of four small language models show lower average perplexity than one, two, or sixteen models, but the paper's broader claims about monotonic optima, robustness,...

Reference graph

Works this paper leans on

42 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Baraniuk

    Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. 2023. https://arxiv.org/abs/2307.01850 Self-consuming generative models go mad . Preprint, arXiv:2307.01850

    arXiv 2023

  4. [4]

    Badr AlKhamissi, Muhammad ElNokrashy, Mai AlKhamissi, and Mona Diab. 2024. https://arxiv.org/abs/2402.13231 Investigating cultural alignment of large language models . Preprint, arXiv:2402.13231

    arXiv 2024

  5. [5]

    Estevão Filho, Todd Hendry, Daniel Holstein, Jennifer Marsman, Nick Mecklenburg, Sara Malvar, Leonardo O

    Angels Balaguer, Vinamra Benara, Renato Luiz de Freitas Cunha, Roberto de M. Estevão Filho, Todd Hendry, Daniel Holstein, Jennifer Marsman, Nick Mecklenburg, Sara Malvar, Leonardo O. Nunes, Rafael Padilha, Morris Sharp, Bruno Silva, Swati Sharma, Vijay Aski, and Ranveer Chandra. 2024. https://arxiv.org/abs/2401.08406 Rag vs fine-tuning: Pipelines, tradeof...

    arXiv 2024

  6. [6]

    Quentin Bertrand, Avishek Joey Bose, Alexandre Duplessis, Marco Jiralerspong, and Gauthier Gidel. 2024. https://arxiv.org/abs/2310.00429 On the stability of iterative retraining of generative models on their own data . Preprint, arXiv:2310.00429

    arXiv 2024

  7. [7]

    Matyas Bohacek and Hany Farid. 2025. https://arxiv.org/abs/2311.12202 Nepotistically trained generative-ai models collapse . Preprint, arXiv:2311.12202

    arXiv 2025

  8. [8]

    Ali Borji. 2024. https://arxiv.org/abs/2410.12954 A note on shumailov et al. (2024): `ai models collapse when trained on recursively generated data' . Preprint, arXiv:2410.12954

    arXiv 2024

Show all 42 references
  1. [9]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

    2020 arXiv

  2. [10]

    Mingyue Cheng, Yucong Luo, Jie Ouyang, Qi Liu, Huijie Liu, Li Li, Shuo Yu, Bohou Zhang, Jiawei Cao, Jie Ma, Daoyu Wang, and Enhong Chen. 2025. https://arxiv.org/abs/2503.10677 A survey on knowledge-oriented retrieval-augmented generation . Preprint, arXiv:2503.10677

    2025 arXiv

  3. [11]

    DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, and 69 others. 2024. https://arxiv.org/abs/2401.02954 Deepsee...

    2024 arXiv

  4. [12]

    Apratim Dey and David Donoho. 2024. https://arxiv.org/abs/2410.22812 Universality of the ^2/6 pathway in avoiding model collapse . Preprint, arXiv:2410.22812

    2024 arXiv

  5. [13]

    Elvis Dohmatob, Yunzhen Feng, Arjun Subramonian, and Julia Kempe. 2024. https://arxiv.org/abs/2410.04840 Strong model collapse . Preprint, arXiv:2410.04840

    2024 arXiv

  6. [14]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  7. [15]

    Pejman Ebrahimi. 2024. https://huggingface.co/datasets/arad1367/Crypto_Semantic_News Crypto semantic news

    2024

  8. [16]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. https://arxiv.org/abs/2312.10997 Retrieval-augmented generation for large language models: A survey . Preprint, arXiv:2312.10997

    2024 arXiv

  9. [17]

    Roberts, Diyi Yang, David L

    Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Henry Sleight, John Hughes, Tomasz Korbak, Rajashree Agrawal, Dhruv Pai, Andrey Gromov, Daniel A. Roberts, Diyi Yang, David L. Donoho, and Sanmi Koyejo. 2024. https://arxiv.org/abs/2404.01413 Is model collap...

    2024 arXiv

  10. [18]

    Nate Gillman, Michael Freeman, Daksh Aggarwal, Chia-Hong Hsu, Calvin Luo, Yonglong Tian, and Chen Sun. 2024. https://arxiv.org/abs/2402.07087 Self-correcting self-consuming loops for generative model training . Preprint, arXiv:2402.07087

    2024 arXiv

  11. [19]

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org

    2016

  12. [20]

    Shailja Gupta, Rajesh Ranjan, and Surya Narayan Singh. 2024. https://arxiv.org/abs/2410.12837 A comprehensive survey of retrieval-augmented generation (rag): Evolution, current landscape and future directions . Preprint, arXiv:2410.12837

    2024 arXiv

  13. [21]

    Vansh Gupta, Sankalan Pal Chowdhury, Vilém Zouhar, Donya Rooein, and Mrinmaya Sachan. 2025. https://arxiv.org/abs/2504.17720 Multilingual performance biases of large language models in education . Preprint, arXiv:2504.17720

    2025 arXiv

  14. [22]

    Hengzhi He, Shirong Xu, and Guang Cheng. 2025. https://arxiv.org/abs/2502.18049 Golden ratio weighting prevents model collapse . Preprint, arXiv:2502.18049

    2025 arXiv

  15. [23]

    Hayden Helm, Brandon Duderstadt, Youngser Park, and Carey Priebe. 2024. https://aclanthology.org/2024.emnlp-main.90 Tracking the perspectives of interacting language models . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1508-...

    2024

  16. [24]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  17. [25]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://arxiv.org/abs/2001.08361 Scaling laws for neural language models . Preprint, arXiv:2001.08361

    2020 arXiv

  18. [26]

    Donoho, and Sanmi Koyejo

    Joshua Kazdan, Rylan Schaeffer, Apratim Dey, Matthias Gerstgrasser, Rafael Rafailov, David L. Donoho, and Sanmi Koyejo. 2025. https://arxiv.org/abs/2410.16713 Collapse or thrive? perils and promises of synthetic data in a self-generating world . Preprint, arXiv:2410.16713

    2025 arXiv

  19. [27]

    Ying Li, Zhidi Lin, Feng Yin, and Michael Minyi Zhang. 2024. https://arxiv.org/abs/2404.01697 Preventing model collapse in gaussian process latent variable models . Preprint, arXiv:2404.01697

    2024 arXiv

  20. [28]

    Keith Man and Javaan Chahl. 2022. https://doi.org/10.3390/jimaging8110310 A review of synthetic image data and its use in computer vision . Journal of Imaging, 8(11)

    2022 doi

  21. [29]

    Gonzalo Martínez, Lauren Watson, Pedro Reviriego, José Alberto Hernández, Marc Juarez, and Rik Sarkar. 2023. https://arxiv.org/abs/2303.01255 Combining generative artificial intelligence (ai) and the internet: Heading towards evolution or degradation? Preprint, arXiv:2303.01255

    2023 arXiv

  22. [30]

    Morris, Brandon Duderstadt, and Andriy Mulyar

    Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. 2024. https://arxiv.org/abs/2402.01613 Nomic embed: Training a reproducible long context text embedder . Preprint, arXiv:2402.01613

    2024 arXiv

  23. [31]

    Jorge Pérez, Javier Marinković, and Pablo Barceló. 2019. https://arxiv.org/abs/1901.03429 On the turing completeness of modern neural network architectures . Preprint, arXiv:1901.03429

    2019 arXiv

  24. [32]

    Rylan Schaeffer, Joshua Kazdan, Alvan Caleb Arulandu, and Sanmi Koyejo. 2025. https://arxiv.org/abs/2503.03150 Position: Model collapse does not mean what you think . Preprint, arXiv:2503.03150

    2025 arXiv

  25. [33]

    Mohamed El Amine Seddik, Suei-Wen Chen, Soufiane Hayou, Pierre Youssef, and Merouane Debbah. 2024. https://arxiv.org/abs/2404.05090 How bad is training on synthetic data? a statistical analysis of language model collapse . Preprint, arXiv:2404.05090

    2024 arXiv

  26. [34]

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. 2024. https://arxiv.org/abs/2305.17493 The curse of recursion: Training on generated data makes models forget . Preprint, arXiv:2305.17493

    2024 arXiv

  27. [35]

    Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. 2025. https://arxiv.org/abs/2501.09136 Agentic retrieval-augmented generation: A survey on agentic rag . Preprint, arXiv:2501.09136

    2025 arXiv

  28. [36]

    Ananda Theertha Suresh, Andrew Thangaraj, and Aditya Nanda Kishore Khandavally. 2024. https://arxiv.org/abs/2412.17646 Rate of model collapse in recursive training . Preprint, arXiv:2412.17646

    2024 arXiv

  29. [37]

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2024. https://arxiv.org/abs/2211.04325 Will we run out of data? limits of llm scaling based on human-generated data . Preprint, arXiv:2211.04325

    2024 arXiv

  30. [38]

    Wang, Tianyu Wang, Avanti Athreya, Vince Lyzinski, and Carey E

    Edward L. Wang, Tianyu Wang, Avanti Athreya, Vince Lyzinski, and Carey E. Priebe. 2025. https://arxiv.org/abs/2506.00077 Gaussian mixture models as a proxy for interacting language models . Preprint, arXiv:2506.00077

    2025 arXiv

  31. [39]

    Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235

    2023 arXiv

  32. [40]

    Sierra Wyllie, Ilia Shumailov, and Nicolas Papernot. 2024. https://arxiv.org/abs/2403.07857 Fairness feedback loops: Training on synthetic data amplifies bias . Preprint, arXiv:2403.07857

    2024 arXiv

  33. [41]

    Greg Yang. 2020. https://arxiv.org/abs/2006.14548 Tensor programs ii: Neural tangent kernel for any architecture . Preprint, arXiv:2006.14548

    2020 arXiv

  34. [42]

    Zoran Zivkovic and Ferdinand van der Heijden. 2004. Recursive unsupervised learning of finite mixture models. IEEE Transactions on pattern analysis and machine intelligence, 26(5):651--656

    2004

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.