Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models
Reviewed by Pithpith:4CFSWDKOopen to challenge →
read the original abstract
Diffusion probabilistic models have become a cornerstone of modern generative AI, yet the mechanisms underlying their generalization remain poorly understood. In fact, if these models were perfectly minimizing their training loss, they would just generate data belonging to their training set, i.e., memorize, as empirically found in the overparameterized regime. We revisit this view by showing that, in highly overparameterized diffusion models, generalization in natural data domains is progressively achieved during training before the onset of memorization. Our results, ranging from image to language diffusion models, systematically support the empirical law that memorization time is proportional to the dataset size. Generalization vs. memorization is then best understood as a competition between time scales. We show that this phenomenology is recovered in diffusion models learning a simple probabilistic context-free grammar with random rules, where generalization corresponds to the hierarchical acquisition of deeper grammar rules as training time grows, and the generalization cost of early stopping can be characterized. We summarize these results in a phase diagram. Overall, our results support that a principled early-stopping criterion - scaling with dataset size - can effectively optimize generalization while avoiding memorization, with direct implications for hyperparameter transfer and privacy-sensitive applications.
This paper has not been read by Pith yet.
Forward citations
Cited by 6 Pith papers
-
Learn from your own latents and not from tokens: A sample-complexity theory
Latent prediction SSL recovers latent trees from PCFG data with sample complexity constant in hierarchy depth L (up to logs), unlike exponential for token-level or supervised methods.
-
The Interplay of Data Structure and Imbalance in the Learning Dynamics of Diffusion Models
Higher-variance classes are learned first in diffusion models; strong class imbalance reverses the order and imposes distinct delayed learning times on minority classes.
-
Evaluating the Representation Space of Diffusion Models via Self-Supervised Principles
Introduces the Invariant Contamination Ratio (ICR), a Fisher-based metric, to evaluate how diffusion models balance invariant representations with residual variation and to detect the onset of memorization during training.
-
Memorisation, convergence and generalisation in generative models
Linear generative models memorize at small data loads but converge continuously once samples scale linearly with dimension; this convergence is insensitive to sharp recovery of principal latent factors.
-
diffGHOST: Diffusion based Generative Hedged Oblivious Synthetic Trajectories
diffGHOST is a conditional diffusion model that segments learned latent space to identify and mitigate memorization of critical trajectory samples, aiming to deliver privacy guarantees alongside data utility.
-
Understanding diffusion models requires rethinking (again) generalization
Diffusion models require new generalization frameworks because memorization and novel generation are incompatible, so research should focus on what models learn before memorization begins.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.