{"id":"a4dce534-5d72-40fe-a411-cfc5fa4d0209","arxiv_id":"2411.12523","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Diffusion models tolerate pruning up to 90% of training data without FID degradation, and cluster-center selection in CLIP/DINO embedding space beats established gradient-based pruning methods.","lead":"This paper tests whether removing data before training a diffusion model hurts it, and finds that large fractions can be pruned with little or no loss in quality, sometimes even improving it. It also shows that a simple clustering method picks better subsets than more complex scoring methods, and can rebalance skewed training sets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ImageNet pruning-tolerance claim is confounded by fixed-step training: at PR=0.9 the pruned model gets ~256 epochs while unpruned gets ~26, so the apparent benefit may be extra epochs, not data redundancy.","rationale":"The reader's weakest assumption is the full-data VQ-VAE. That is a genuine limitation: the autoencoder is trained on the unpruned dataset and confers a shared latent prior on all conditions, so the DM-only comparison may overstate how much of the data distribution is recoverable after pruning. The authors themselves flag this in Section 4.2. However, because the same VQ-VAE is used for the unpruned and all pruned conditions, it is held constant across the method comparison; it cannot by itself explain why ClusterC beats random at fixed steps. The more serious confound is the fixed-step training protocol. The full text reports a fixed 120k iterations on ImageNet and 220k on CelebA-HQ, regardless of pruning ratio. Since batch size is constant, the number of epochs grows by a factor of 1/(1-PR). At PR=0.9 on ImageNet, the pruned model is trained for 256 epochs while the unpruned model is trained for 25.6 epochs. The headline claim \"no decline in FID after 90% pruning\" is therefore comparing a heavily trained small set to an undertrained full set. If the authors had trained the unpruned model for the same number of epochs, the full data would likely achieve a lower FID, and the conclusion would shift from \"data redundancy\" to \"more repetition helps\". This is not a case of disagreement with consensus; it is an internal control that the paper does not include. The paper's ablative inverse experiments are a strength, and the qualitative results are suggestive, but they do not resolve the epoch imbalance. The appropriate verdict remains CONDITIONAL: the claims are plausible, and the requested same-epoch baselines are feasible to run; adding them would settle whether the central effect is real. The code is stated to be available, which makes this check tractable.","tokens_in":1116,"tokens_out":1274,"duration_ms":118698,"concrete_test":"Re-run the ImageNet PR=0.9 ClusterC and unpruned conditions with matched epoch counts instead of matched step counts: unpruned at 1.2M iterations and ClusterC at 120k iterations (both about 256 epochs at batch 256). If unpruned FID is then lower than ClusterC, the \"90% pruning without decline\" claim is an artifact of the 10x epoch imbalance. A cheaper proxy on CelebA-HQ: compare unpruned (220k iterations, about 251 epochs) with PR=0.9 clustering at 22k iterations (about 251 epochs on the 10% subset); if FID degrades, the tolerance depends on repeated epochs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is the ImageNet tolerance result (Section 1: \"we can prune as much as 90% of the data without any decline in performance\"; Section 4.2: \"Across all PRs, ClusterC/D yields the lowest FID ... At PR=0.9 the gap between unpruned and ClusterC is approximately 5\"). The evaluation fixes the number of optimizer steps rather than the number of epochs. Section 4 states: \"we use a batch size of 256 ... a total of 120k iterations\" on ImageNet. One epoch on 1.2M images at batch 256 is 4,688 iterations, so the unpruned baseline is trained for only about 25.6 epochs. At PR=0.9 the training set has 120k images, one epoch is 469 iterations, so the same 120k steps equal roughly 256 epochs. The pruned model therefore receives about 10x more gradient updates per image than the unpruned model. A 10% subset trained for 10x more epochs can match an undertrained full-data model without any of the discarded 90% being redundant. Random pruning shares the inflated epoch count, so comparing ClusterC to random does not control for this; the unpruned condition is the only one without the boost. This confound is not acknowledged in the paper. The VQ-VAE concern raised by the reader is real but is partly acknowledged in Section 4.2 and does not explain the relative ordering of pruning methods under fixed steps. The training-budget confound directly affects the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies data pruning for generative diffusion models. The authors implement several pruning criteria (random, loss monotonicity, GraNd, EL2N, MoSo, and clustering in CLIP/DINO embedding spaces) and evaluate them on CelebA-HQ and ImageNet using a DiT/SiT flow-matching diffusion model operating on a VQ-VAE latent space. They also run inverse-selection ablations and a balanced-sampling variant of clustering. The main claims are that diffusion models tolerate pruning up to 90% on ImageNet without FID degradation, that cluster-center selection in a large visual model's embedding space outperforms the unpruned model, and that balanced clustering can mitigate skewed distributions.","tokens_in":14351,"tokens_out":7560,"duration_ms":74600,"significance":"The paper addresses a timely and practically important question and assembles a broad set of comparisons; the inverse-selection ablations are a thoughtful way to test whether a scoring criterion carries signal, and the code release is a strength. If the headline results were established, the finding that simple CLIP/DINO clustering beats more expensive gradient-based methods on ImageNet would be a useful practical result. However, the central quantitative claims are currently undermined by a training-budget confound and a full-data VQ-VAE confound, and they rest on single-run evaluations. The balanced-sampling application is interesting but also currently supported mainly by qualitative evidence.","major_comments":[{"comment":"The ImageNet comparison fixes the number of optimizer steps (120k iterations, batch size 256) rather than the number of epochs. The unpruned model is trained for 120k / (1.2M / 256) ≈ 25.6 epochs, while a PR=0.9 model is trained for 120k / (120k / 256) ≈ 256 epochs, so each retained sample is presented to the optimizer roughly ten times more often. This unequal training effort per sample can explain the Section 4.2 result that PR=0.9 matches or improves on the unpruned baseline without implying that the discarded 90% is redundant. Please report results with equal epoch budgets, or with multiple compute budgets, or explicitly reframe the claim as a fixed-step result rather than a data-redundancy result.","section":"Section 4, Training"},{"comment":"The VQ-VAE is trained on the full unpruned dataset, and the paper itself identifies the autoencoder as 'another pivotal factor in preserving the performance.' Because the latent encoder/decoder has seen all images, information from pruned-out samples may be encoded in the latent geometry, so the experiment does not cleanly isolate the diffusion model's tolerance to data pruning. Please add an ablation with a VQ-VAE trained only on the pruned subset (or another control that removes the full-data representation confound) and qualify the headline claim accordingly.","section":"Section 4.3"},{"comment":"All FID, F-score, Inception, and Vendi results appear to be single runs with no error bars or confidence intervals. The headline comparison (e.g., 'at PR=0.9 the gap between unpruned and ClusterC is approximately 5') and the relative ordering of ClusterC/D versus random rest on differences that could be within run-to-run noise. Please report means and standard deviations over at least three seeds, and state the number of seeds used.","section":"Sections 4.1-4.2"},{"comment":"The text asserts that a VAE and a GAN degrade 'instantaneously' when trained on a randomly pruned subset, but no quantitative results, figure, or experimental details are given. This claim is load-bearing for the conclusion that DMs are unique in their pruning tolerance, so it needs to be either reported with evidence or removed.","section":"Section 4.3, Comparison with other generative models"}],"minor_comments":[{"comment":"Numerous typos and grammatical errors (e.g., 'compehensive', 'monotoncity', 'otpimization', 'mdeosl', 'instantateous') make the text harder to read; a careful proofread is needed.","section":"Throughout"},{"comment":"The pruning ratio is never formally defined; please state explicitly that PR denotes the fraction of data removed (so PR=0.9 leaves 10% of the data).","section":"Section 3.2 and Section 4"},{"comment":"The notation for clustering variants is inconsistent ('Cluster backbone', 'Cluster mid backbone', 'Cluster −1 backbone' vs 'ClusterC/D' vs table entries 'clusterC'/'clusterD'); unify the notation.","section":"Section 3.2"},{"comment":"The choice of k (24 for CelebA-HQ, 1000 for ImageNet) is reasonable but sensitivity to k is not reported; since clustering performance may depend on this hyperparameter, a small sensitivity analysis would strengthen the claims.","section":"Section 4"},{"comment":"The column layout is crowded and values run together (e.g., '7.74345.5829'), making the table difficult to read; fix the formatting.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper has enough empirical merit to support revision rather than rejection, but the headline claim must be re-tested under equal epoch budgets before publication. The main risk is that the 90% tolerance result is an artifact of the unpruned baseline being trained for only about 26 epochs while the pruned model is trained for about 256 epochs; if that is the case, the paper would need substantial reframing. I would encourage the editor to seek a revision that adds the epoch-controlled comparison and the VQ-VAE ablation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. It's the most thorough empirical comparison of data pruning methods for diffusion models I've seen, and the inverse-selection ablations are a genuinely good design choice. But the headline claim—that ImageNet diffusion models tolerate 90% pruning without FID loss—is confounded by the training schedule. They run a fixed 120k iterations for all conditions. At batch 256 on 1.2M images, that's ~26 epochs for the unpruned model. At PR=0.9, the same 120k steps give the pruned model ~256 epochs, ten times more updates per image. So the pruned model isn't just seeing less data; it's seeing the retained data many more times. The unpruned baseline is undertrained, and the comparison doesn't isolate data redundancy from extra per-sample training. This isn't a minor detail; it's the load-bearing result of Sections 1 and 4.2.\n\nWhat's actually new: applying pruning to generative diffusion models is relatively underexplored, and the systematic comparison of gradient-, loss-, and cluster-based methods at two scales is useful. The inverse-selection experiments are a smart sanity check—e.g., GraNd's inverse degrading as expected gives some confidence in the method. The balanced-cluster experiment for fair sampling is a nice side contribution. The finding that simple clustering in CLIP/DINO space outperforms expensive gradient methods is plausible and worth taking seriously.\n\nThe soft spots beyond the training-budget confound: single runs, no error bars or multiple seeds anywhere, which matters for FID differences of ~2–5. The VQ-VAE trained on the full dataset is a real confound, and the authors acknowledge it in Section 4.3 but don't control for it—no experiment retrains the autoencoder on pruned data. The clustering uses k=1000 on ImageNet, equal to the number of classes; that choice may leak label information, and they don't test sensitivity to k.\n\nWho's this for: researchers working on data-efficient generative training, especially those deciding whether to invest in pruning strategies. They'll find a broad map of what works and what doesn't, but they should treat the quantitative tolerance claims as upper bounds until fixed-epoch baselines are added. I'd send it to a serious referee—the topic is timely, the experiments are extensive, and the confounds are addressable. The authors should be asked to rerun or analyze fixed-epoch conditions and to report variance across seeds. If they can show the tolerance survives those controls, it becomes a strong paper.","headline":"The paper's broad pruning survey is useful, but its central 90%-tolerance claim is confounded by fixed-step training that gives pruned models ~10x more epochs per sample.","tokens_in":14844,"tokens_out":2537,"would_cite":false,"duration_ms":24515,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Diffusion models tolerate losing 90% of their ImageNet training data, and a simple clustering rule turns the pruned set into a better generator than the full dataset.","keywords":["data pruning","diffusion models","generative models","coreset selection","embedding clustering","dataset balancing","ImageNet","CelebA-HQ"],"falsifier":"Train the VQ-VAE on exactly the pruned subset (or on a disjoint half of the data) and repeat the ImageNet 90% pruning run; if the FID advantage over the unpruned baseline disappears or turns negative, the reported tolerance is an artifact of the full-data autoencoder rather than a property of diffusion training.","tokens_in":13827,"feed_emoji":"✂️","tokens_out":6980,"duration_ms":60386,"temperature":0.7,"pith_summary":"The paper asks whether data pruning, discarding part of the training set, can help or hurt generative diffusion models, which must model a whole distribution rather than a decision boundary. It argues that pruning is not only possible but often beneficial: on ImageNet, removing up to 90% of the data leaves FID unchanged, and a simple clustering rule that keeps samples closest to cluster centers in a pretrained model's embedding space beats both the unpruned model and more expensive gradient-based pruning methods. On the smaller CelebA-HQ dataset, random selection is competitive, and the authors show how per-cluster balanced sampling can rebalance a skewed distribution so underrepresented groups are generated reliably. The significance is practical: if correct, large diffusion models can be trained on a fraction of the data, and the same mechanism offers an unsupervised handle on fairness in generation.","feed_headline":"Clustering pick of 10% of ImageNet beats training on all of it","feed_subtitle":"Simple embedding-space clustering picks the core data, keeping FID as good as or better than full training.","key_machinery":"The central device is embedding-space clustering as a data-selection rule. A pretrained visual model extracts a feature vector for each image; k-means clusters those embeddings; and the pruning rule keeps, from each cluster, the samples closest to the cluster center (or, in the ablations, furthest or in-between). The nearest-to-center samples are treated as typical representatives of each mode, so aggregating them covers the distribution's dense regions. The other load-bearing component is the latent-space setup: a VQ-VAE trained on the full unpruned dataset encodes images into a compact latent space, and the diffusion model is trained in that space with a flow-matching objective, allowing the autoencoder to reinterpret the model's output even after most training samples are gone.","core_discovery":"Diffusion models have a high tolerance threshold for pruning. On ImageNet, pruning 90% of the training data causes no decline in FID, and selecting samples nearest the centroids of clusters formed in CLIP or DINO embedding space yields better FID than training on the full dataset. On CelebA-HQ, no pruning strategy reliably beats random sampling, indicating that for smaller, more homogeneous datasets the main requirement is coverage of the distribution. The authors also show that taking an equal number of samples from each cluster, rather than a number proportional to cluster size, produces generated data with more balanced group representations, at little or no FID cost on ImageNet.","pith_inferences":["A testable corollary is that the tolerance should persist in pixel-space diffusion models if the mechanism is iterative denoising; if it only appears with a full-data autoencoder, the latent space, not denoising, is what preserves quality.","The nearest-to-centroid rule can be viewed as an ordering of samples by typicality; this suggests data-scaling laws for generative models might be expressible in terms of effective cluster mass rather than raw sample count.","The success of CLIP/DINO-based clustering likely depends on how well the backbone's embeddings match the target distribution's modes; on out-of-distribution domains the recipe may need a backbone fine-tuned on that domain.","A practical workflow emerges from the paper's ablations: embed with a frozen foundation model, cluster, keep centroid-near samples for dense coverage, and switch to equal-per-cluster sampling when demographic balance matters more than raw FID."],"forward_implications":["On ImageNet-scale data, around 90% of training samples can be discarded for latent diffusion training, so compute, storage, and energy budgets could shrink by roughly an order of magnitude without hurting FID.","A frozen pretrained embedding backbone plus k-means is a cheaper and more effective pruning recipe than gradient-norm-, loss-, or forgetting-based scoring on large diverse datasets.","Balanced per-cluster sampling can steer a diffusion model toward underrepresented groups in an unsupervised way, with a small FID penalty on skewed small datasets and almost none on ImageNet.","For small datasets such as CelebA-HQ, random pruning is the practical choice at high pruning ratios, because expensive scoring strategies do not beat it and can even hurt.","Diffusion models' step-by-step denoising makes them more resilient to data removal than single-pass generative models such as GANs and VAEs, which degrade immediately on pruned data."],"supporting_citations":[{"why":"Supplies the forgetting-count idea that the paper adapts into the loss-monotonicity pruning baseline.","marker":"[55]"},{"why":"Provides the GraNd and EL2N gradient- and loss-based scoring baselines.","marker":"[38]"},{"why":"Provides the Moving-one-Sample-out (MoSo) baseline and its gradient-agreement rationale.","marker":"[51]"},{"why":"Supplies the CLIP embedding backbone used for clustering and for memorization-distance checks.","marker":"[40]"},{"why":"Supplies the DINO embedding backbone used for clustering.","marker":"[37]"},{"why":"Provides the DiT transformer architecture on which the diffusion training is implemented.","marker":"[39]"},{"why":"Supplies the VQ-VAE that maps images into the latent space; the paper trains its own VQ-VAE on the full dataset.","marker":"[56]"},{"why":"Supplies the flow-matching objective used to reverse the diffusion process during training and sampling.","marker":"[29]"}],"fun_headline_variants":["Cluster-selected 10% of ImageNet beats full set for diffusion","Pruning 90% of ImageNet improves diffusion model FID","Simple clustering outperforms full ImageNet for diffusion training","Less data, better diffusion: clustering picks the core","Data pruning with clustering beats full training on ImageNet"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The autoencoder that defines the latent space is trained on the full unpruned dataset, so information from pruned-away samples may still be encoded and available to the diffusion model.","fun_headline_variants_meta":{"raw":{"variants":["Cluster-selected 10% of ImageNet beats full set for diffusion","Pruning 90% of ImageNet improves diffusion model FID","Simple clustering outperforms full ImageNet for diffusion training","Less data, better diffusion: clustering picks the core","Data pruning with clustering beats full training on ImageNet"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000322,"raw_usage":{"total_tokens":1761,"prompt_tokens":845,"completion_tokens":916,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":833}},"tokens_in":461,"tokens_out":916,"duration_ms":8875,"temperature":1.0,"reasoning_tokens":833,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:24:59.447148+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the VQ-VAE on exactly the pruned subset (or on a disjoint half of the data) and repeat the ImageNet 90% pruning run; if the FID advantage over the unpruned baseline disappears or turns negative, the reported tolerance is an artifact of the full-data autoencoder rather than a property of diffusion training.","supporting_citations":[{"cited_title":"Deep learning on a data diet: Finding im- portant examples early in training","cited_arxiv_id":null,"evidence_quote":"Provides the GraNd and EL2N gradient- and loss-based scoring baselines."},{"cited_title":"Data pruning via moving-one-sample-out","cited_arxiv_id":null,"evidence_quote":"Provides the Moving-one-Sample-out (MoSo) baseline and its gradient-agreement rationale."},{"cited_title":"Learning transferable visual models 14 from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP embedding backbone used for clustering and for memorization-distance checks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DINO embedding backbone used for clustering."}],"review_version":1}