From optimal score matching to optimal sampling
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:FYCHWX7Brecord.jsonopen to challenge →
read the original abstract
The recent, impressive advances in algorithmic generation of high-fidelity image, audio, and video are largely due to great successes in score-based diffusion models. A key implementing step is score matching, that is, the estimation of the score function of the forward diffusion process from training data. As shown in earlier literature, the total variation distance between the law of a sample generated from the trained diffusion model and the ground truth distribution can be controlled by the score matching risk. Despite the widespread use of score-based diffusion models, basic theoretical questions concerning exact optimal statistical rates for score estimation and its application to density estimation remain open. We establish the sharp minimax rate of score estimation for smooth, compactly supported densities. Formally, given \(n\) i.i.d. samples from an unknown \(\alpha\)-H\"{o}lder density \(f\) supported on \([-1, 1]\), we prove the minimax rate of estimating the score function of the diffused distribution \(f * \mathcal{N}(0, t)\) with respect to the score matching loss is \(\frac{1}{nt^2} \wedge \frac{1}{nt^{3/2}} \wedge (t^{\alpha-1} + n^{-2(\alpha-1)/(2\alpha+1)})\) for all \(\alpha > 0\) and \(t \ge 0\). As a consequence, it is shown the law \(\hat{f}\) of a sample generated from the diffusion model achieves the sharp minimax rate \(\bE(\dTV(\hat{f}, f)^2) \lesssim n^{-2\alpha/(2\alpha+1)}\) for all \(\alpha > 0\) without any extraneous logarithmic terms which are prevalent in the literature, and without the need for early stopping which has been required for all existing procedures to the best of our knowledge.
This paper has not been read by Pith yet.
Forward citations
Cited by 7 Pith papers
-
Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
Helmholtz-Hodge decomposition of score errors shows only the gradient component affects marginal Fokker-Planck dynamics in diffusion models, yielding an impossibility result for L2 error bounding divergences and a tra...
-
Diffusion Models Observe Only Gradients: A Geometric Perspective on Score Matching Errors
Only the gradient component of score errors affects marginal distributions in diffusion models, so L2 error can be arbitrarily large with perfect match; this yields an impossibility result, a gradient-only KL bound, a...
-
Intrinsic Wasserstein Rates for Score-Based Generative Models on Smooth Manifolds
Score-based generative models attain intrinsic Wasserstein-1 sample rates of order n to the power of -(beta+1)/(d+2beta) on d-dimensional smooth manifolds with beta-Holder densities.
-
On the Robustness of Distribution Support under Diffusion Guidance
Guided diffusion generates samples near the target distribution support under exact score access, explaining its empirical success in producing plausible outputs.
-
Diffusion Model for Manifold Data: Score Decomposition, Curvature, and Statistical Complexity
Diffusion models on manifold-supported data admit score decompositions whose statistical rates are controlled by intrinsic dimension and curvature.
-
Diffusion Models for Adaptive Sequential Data Generation
Introduces a sequential forward-backward diffusion framework that generates adapted time series by conditioning on prior history, with a parallelizable score-matching objective and statistical guarantees for ReLU networks.
-
On the Robustness of Distribution Support under Diffusion Guidance
Establishes robustness of distribution support for guided diffusion processes under exact score access across DDIM, DDPM, and exponential integrator discretizations.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.