{"id":"0a7141d1-4fa8-4bf5-9a99-a597a2436058","arxiv_id":"2411.12841","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"D2M distills a dataset's knowledge into the parameters of a pre-trained GAN, enabling flexible, architecture-general synthetic training data with state-of-the-art classification accuracy.","lead":"D2M replaces the distilled image set with a fine-tuned generative model: after training, the model can produce any number of training images per class. It reports top accuracy on CIFAR-10/100, TinyImageNet, and high-resolution ImageNet subsets, and needs no re-distillation when the images-per-class ratio changes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 1's 'pre-trained generative model' is never specified for non-ImageNet targets; if it was trained on the full target dataset, D2M is fine-tuning an already-distilled generator and the reported costs omit the dominant pretraining expense.","rationale":"The paper's contribution is an empirical framework with strong headline numbers (Tables 1-5) and a plausible loss design (embedding matching plus prediction matching). My concern is not about the loss formulation or the internal math of Eqs. 2-5; those are clear. The vulnerability is external: the input to Algorithm 1 is a pre-trained generator, and the paper does not disclose its provenance. In dataset distillation, the total cost must include everything needed to produce the distilled set. Prior pixel methods train from the target dataset alone; if D2M requires a GAN already trained on the target dataset, its 'distillation cost' is only a small fine-tuning tail. This also weakens novelty: the generator already encodes class structure and image statistics, so Eq. 5 is refining known knowledge rather than distilling it. The Appendix B note about the latency-matched evaluation protocol is a separate issue; it is secondary in my assessment because it can be addressed by reporting the exact number of images and epochs used, and it may even be conservative if generation time eats into training time. The lambda discrepancy (100 in Sec. 4.1 vs. 10 in Sec. 4.3) is a reproducibility bug but not the central scientific risk. I did not find evidence of fraud or fabrication; the paper provides code links, multiple generator ablations, and a diffusion-model extension, which are positive signs. The right response is a conditional acceptance requiring the GAN provenance and cost audit, exactly as the reader recommended.","tokens_in":25909,"tokens_out":8738,"duration_ms":84624,"concrete_test":"Request or obtain the exact GAN checkpoints and training scripts for all 15 datasets, then run the control: use the public ImageNet BigGAN checkpoint (not trained on CIFAR-10/100 or DermaMNIST), apply the authors' class-embedding remapping, and run D2M's Eq. 5 optimization to reproduce Table 1 (CIFAR-10 IPC1/10/50: 50.2/67.8/74.4). If accuracy falls materially below these numbers, the reported results depend on GAN pretraining on the full target dataset, and Table 4 must include that pretraining cost; if accuracy matches, the concern is resolved. Also recompute Table 4 with GAN pretraining GPU hours included to check the claimed re-distillation savings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 and Algorithm 1 take as input a 'pre-trained generative model G,' but the paper never states how BigGAN is obtained or adapted for the non-ImageNet datasets in Tables 1, 6, and 7. BigGAN is a 1000-class ImageNet generator; CIFAR-10/100 (32x32, 10/100 classes), TinyImageNet (200 classes), and DermaMNIST (7-class medical images) are outside its native class and resolution range. The paper says only 'we deploy the pre-trained BigGAN with the default hyperparameters' (Sec. 4.1). If the generator was fine-tuned or trained from scratch on each full target dataset before distillation, then its parameters already encode the target classes and statistics, so the 'distillation' in Eq. 5 is fine-tuning a model that already contains the distilled knowledge, and the reported cost (Table 4: 4.9 GPU hours total for CIFAR-10) omits the dominant pretraining expense. If instead an off-the-shelf ImageNet checkpoint is used without target-domain training, the mechanics of class-conditional generation for CIFAR-100 or DermaMNIST are unexplained, since class-embedding remapping is never described. This is the load-bearing assumption: the central claim of distilling a dataset into a generative model, and the efficiency comparison against pixel-space distillation, both depend on what G was pre-trained on and at what cost.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Data-to-Model Distillation (D2M), a framework that distills a large training dataset into the parameters of a pre-trained generative model (default: BigGAN) by matching channel attention maps and softened logits between real and generated images under a randomly sampled network from a model pool. After distillation, the generator can produce training images for arbitrary images-per-class (IPC) settings without re-distillation, and is claimed to achieve state-of-the-art results across 15 datasets, including 128×128 ImageNet-1K, while maintaining constant storage complexity. The paper also reports re-distillation efficiency, cross-architecture generalization, and a neural architecture search application.","tokens_in":26193,"tokens_out":4813,"duration_ms":46762,"significance":"If the central assumptions are clarified and the evaluation is made directly comparable, D2M is a conceptually interesting step: replacing pixel-space synthetic data with a tunable generator could genuinely address the re-distillation and scalability limitations of prior dataset distillation methods. The paper's strengths include extensive experiments on many datasets, a clear and reproducible algorithmic skeleton (Algorithm 1), public code, and ablations over temperature, task balance, batch size, and generator backbones. The cross-architecture generalization results (Table 5) and the NAS correlation experiment (Table 8) are valuable empirical contributions. However, the significance of the main efficiency and performance claims hinges on the provenance and cost of the pre-trained generator, which the manuscript leaves underspecified.","major_comments":[{"comment":"The pre-trained generative model G is never specified for the non-ImageNet target datasets. BigGAN is a 1000-class ImageNet generator, while CIFAR-10/100 (10/100 classes, 32×32), TinyImageNet (200 classes), and DermaMNIST (7-class medical images) are outside its native class and resolution range. The manuscript only states 'we deploy the pre-trained BigGAN with the default hyperparameters' (Section 4.1). If G was trained on the full target dataset before distillation, then its parameters already encode the target classes and statistics, and the reported distillation cost in Table 4 (4.9 GPU hours total for CIFAR-10) omits the dominant pretraining expense. If instead an off-the-shelf ImageNet checkpoint is used without target-domain adaptation, the mechanics of class-conditional generation for 100 or 200 classes are unexplained. This is load-bearing for the central claim of distilling a dataset into a generative model and for the efficiency comparison against pixel-space distillation; the paper must state how G was obtained for each dataset, whether it was trained or fine-tuned on the full target data, and report the corresponding cost.","section":"Section 3.1, Eq. (1); Section 4.1; Table 4"},{"comment":"The evaluation protocol uses a wall-clock latency budget rather than a fixed image count. In Section 4.1, the authors state that they measure the end-to-end latency of training a model on IPC images and then divide this time between generating images and training for D2M. Supplementary Section B confirms that the 'raw amount of images is determined by the fixed time ratio per configuration.' This means the effective number of training images seen by a D2M-trained model may differ from the IPC value reported in Tables 1–3 and 6. Since all baselines are evaluated by training on exactly IPC pre-computed images, the 'superior performance' claim is not yet established under a strictly matched data budget. The paper should report the actual number of generated images used for each IPC and either match the image count or explicitly justify why matching latency, rather than image count, is the appropriate fairness criterion.","section":"Section 4.1; Supplementary Section B"},{"comment":"The default value of the task-balance parameter λ is inconsistent across the manuscript. Section 4.1 states 'setting the task balance λ and the temperature T to 100 and 4, respectively,' while Section 4.3 states 'Our default value of λ = 10 yields the best results,' and Supplementary Table 11 lists the default for λ as 100. Since the reported results in Tables 1–3 depend on the actual λ used, this inconsistency undermines reproducibility of all reported numbers.","section":"Section 4.3 vs. Section 4.1 and Supplementary Table 11"},{"comment":"The model pool used for feature matching includes Depth-n ConvNets, which are exactly the architectures used for evaluation in Tables 1–3. Because D2M optimizes the generator to match features extracted by networks randomly drawn from this pool, part of the performance on the ConvNet evaluation may be attributed to matching the evaluation architecture rather than to generalizable distilled knowledge. The cross-architecture experiments in Table 5 partially address this concern, but the primary comparisons on the seen architecture remain partially circular; the paper should quantify this effect, for example by evaluating on a ConvNet variant that is excluded from the model pool.","section":"Section 4.1; Section 5"}],"minor_comments":[{"comment":"The heading 'Visulization' contains a typo, and 'ImegeNette' in Supplementary D.1 (Diffusion Model Analysis) should be 'ImageNette'.","section":"Section 4.4 and Supplementary D.1"},{"comment":"The dataset statistics table has mismatched rows: the row labeled 'ImageNet-1K (128 × 128)' lists resolution '64 × 64', and the row labeled 'ImageNet-1K Subsets (256×256)' lists resolution '128 × 128'. These are likely typographical errors and should be corrected to match the experimental setup described in Section 4.1.","section":"Supplementary Table 9"},{"comment":"Algorithm 1 lists 'Pre-trained generative model G' as an input but does not mention the discriminator D, even though Eq. (1) in Section 3.1 describes training with a discriminator. Clarify whether D is used in the distillation stage or only in the pretraining stage.","section":"Section 3, Algorithm 1"},{"comment":"The claim of state-of-the-art performance on 128×128 ImageNet-1K is supported only by a comparison to Random and Figure 3, which shows no other baseline results. Please provide numerical comparisons to any scalable baselines or explicitly state that no prior method reports results at this resolution.","section":"Section 4.2, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid empirical core and the proposed framework is interesting, but the underspecified pretrained-generator provenance is a serious correctness risk for the main efficiency and novelty claims. I would encourage the editor to request a revision that clarifies the generator's training data and cost, aligns the evaluation protocol with the baselines, and fixes the λ inconsistency. If the authors can resolve these points, the paper could become a strong contribution to the dataset distillation literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. D2M is a serious empirical paper with a genuinely new idea: instead of distilling a dataset into pixels or into a GAN's latent codes, it distills into the generator's parameters, using channel-attention embedding matching plus softened logit matching against a pool of ConvNets. That combination is novel relative to DiM and GLaD, and it delivers real practical advantages: one distillation serves multiple IPC settings, storage stays constant, and the 128x128 ImageNet-1K result is a first for this line. The cross-architecture table and NAS correlation of 0.80 are solid supporting evidence. This is not a restatement of prior work.\n\nThe soft spots are concentrated in the setup and evaluation. The stress-test about BigGAN lands hardest. The paper says 'we deploy the pre-trained BigGAN with default hyperparameters,' but BigGAN is a 1000-class ImageNet generator. How it becomes a 10-class CIFAR-10, 200-class TinyImageNet, or 7-class DermaMNIST generator is never explained. If the generator was trained or fine-tuned on the full target dataset before distillation, then the reported costs omit the dominant pretraining expense and the 'distillation' is partly fine-tuning an already-informed model. If it is an off-the-shelf ImageNet checkpoint, the class-embedding remapping needs to be described. This is not a minor omission; it bears directly on the efficiency claim.\n\nSecond, the evaluation protocol is latency-matched, not fixed-IPC. Appendix B is transparent about this, which I appreciate, but the paper never reports the actual number of images D2M uses at each nominally-IPC setting. Without that, Tables 1-3 are not directly comparable to the baselines. The protocol might be fair or even conservative, but as written the SOTA claim is not established.\n\nThird, there is a concrete reporting inconsistency: the implementation section says lambda = 100, while the ablation says the default lambda = 10 gives the best results. Table 9 also has resolution typos. These are minor but should be fixed.\n\nI agree with the conditional verdict. I weight the GAN provenance issue more heavily than the latency-protocol issue. Both are addressable. I would not desk-reject this. A serious referee should send it back for major revision with those questions. The audience is dataset distillation and synthetic-data-for-NAS researchers; after a clean revision, this is a citable contribution.","headline":"Genuinely new distillation target and impressive scaling, but the GAN provenance and latency-matched protocol must be clarified before the SOTA claim can be taken at face value.","tokens_in":26792,"tokens_out":4933,"would_cite":true,"duration_ms":49602,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes Data-to-Model Distillation (D2M), which distills a real dataset's knowledge into the parameters of a pre-trained generative model rather than into raw pixels, so one distilled generator can supply informative training…","keywords":["dataset distillation","generative model","data-efficient learning","synthetic data","image classification","neural architecture search","cross-architecture generalization","embedding matching"],"falsifier":"Distill with a generator that was pretrained without access to the target dataset's classes (for example, a BigGAN trained only on a disjoint subset of ImageNet, evaluated on CIFAR-10 or TinyImageNet classes), then measure IPC-10 accuracy; if it collapses toward random while the paper's numbers stand, the prior full-data pretraining of the generator is the source of the distilled knowledge.","tokens_in":1638,"feed_emoji":"🖼️","tokens_out":1531,"duration_ms":67867,"temperature":0.7,"pith_summary":"Dataset distillation normally squeezes a large training set into a handful of synthetic pixel images. This paper argues that the right container for the distilled knowledge is not pixels but the parameter space of a pre-trained generative model: after a short refinement, the generator itself becomes the distilled dataset and can emit informative training images at any desired number per class. The authors show on 15 datasets that this data-to-model approach outperforms pixel-space distillation, transfers to architectures it never saw during distillation, needs no re-distillation when the image budget changes, and scales to 128x128 ImageNet-1K and 256x256 medical images. If correct, it turns dataset distillation from a storage problem into a model problem, with a single fixed-size artifact replacing many synthetic-image sets.","feed_headline":"One generator, distilled once, serves any data budget","feed_subtitle":"D2M tunes a pre-trained GAN's weights so a single distillation outperforms pixel-wise methods on 15 datasets.","key_machinery":"The central object is a pre-trained generative model $G$ (BigGAN by default) serving as the distillation container; the identity is the two-term objective $G^* = \\arg\\min_G \\mathcal{L}_{EM} + \\lambda \\mathcal{L}_{PM}$. $\\mathcal{L}_{EM}$ aligns vectorized channel-wise attention maps between real and generated images across $L$ feature layers of a sampled network, while $\\mathcal{L}_{PM}$ minimizes KL divergence between softened class predictions for real and generated pairs with the same label. A model pool of randomly initialized ConvNets and ResNets supplies the networks $\\theta$, preventing overfitting to one architecture. This machinery moves the synthetic dataset out of pixel space into a fixed-size parameter vector, which is what makes re-distillation-free IPC changes and constant storage possible.","core_discovery":"The central claim is that a generator refined with two matching losses—embedding matching on channel-wise attention maps and prediction matching on softened logits—captures enough of a real dataset's training signal that random draws from it outperform the best pixel-optimized synthetic sets. The learned generator solves $G^* = \\arg\\min_G \\mathcal{L}_{EM} + \\lambda \\mathcal{L}_{PM}$, where the expectation over a pool of randomly initialized networks provides diverse supervision. Once distilled, one generator produces training images for any number of images per class, so re-distillation cost is zero and storage is constant; the paper reports state-of-the-art accuracy on CIFAR-10/100, TinyImageNet, ImageNet-1K and its subsets, and DermaMNIST, plus cross-architecture gains that reach 63.6% average on CIFAR-10 at IPC50.","pith_inferences":["If the generator was pretrained on the full target dataset, the reported distillation cost excludes the dominant pretraining cost; the fair comparison would amortize BigGAN training or use a generator with no target-data exposure.","The same parameter-space container could work with diffusion backbones: the paper's single SD-XL experiment on ImageNette (52.1% at IPC1) hints the framework is not GAN-specific, but the authors do not develop this direction.","The evaluation protocol matches total train-plus-generate time rather than exact image counts (Appendix B), so headline IPC numbers are not directly comparable to fixed-IPC baselines without re-basing.","Because the generator can emit unlimited images, D2M may enable privacy-focused or continual-learning settings where storing pixels is undesirable, but memorization of training images would need to be tested."],"forward_implications":["One distillation run serves all image-per-class settings; re-distilling from IPC1 to IPC50 costs 0 GPU hours, saving up to 42.5 GPU hours compared with pixel-space methods on CIFAR-10.","Storage and learnable-parameter count stay constant as IPC grows, up to roughly 35x fewer parameters than pixel-space distillation at IPC50.","The same distilled generator transfers to unseen architectures (AlexNet, VGG-11, ResNet-50, DenseNet-121, ViT), averaging 63.6% on CIFAR-10 with IPC50, 3.9 points above the best prior method.","D2M scales to 128x128 ImageNet-1K and 256x256 datasets such as ImageSquawk and DermaMNIST, where pixel-space distillation methods struggle.","Distilled images rank architectures for neural architecture search with a Spearman correlation of 0.80, approaching the whole-dataset ranking of 1.00."],"supporting_citations":[{"why":"Supplies the pre-trained generator whose parameters D2M refines; without it the distillation has no starting container.","marker":"[3]"},{"why":"Provides the attention-matching idea and model-pool training that D2M adapts, and is a strong pixel-space baseline in most comparisons.","marker":"[64]"},{"why":"Trajectory-matching baseline and source of ImageNet subsets used for high-resolution comparisons.","marker":"[5]"},{"why":"Generative-prior distillation baseline that stores latents and needs retraining per IPC, the contrast D2M emphasizes.","marker":"[6]"},{"why":"Feature-regression baseline that is D2M's closest high-resolution competitor on ImageNet subsets.","marker":"[97]"},{"why":"Concurrent method distilling a dataset into a generative model; D2M distinguishes its single-stage, any-dataset approach from DiM's two-stage 10-class pipeline.","marker":"[78]"},{"why":"Knowledge distillation with softened logits supplies the prediction-matching loss used in D2M.","marker":"[28]"},{"why":"Differentiable augmentation applied during both distillation and evaluation, and a baseline method.","marker":"[91]"}],"fun_headline_variants":["Distill a generator once, serve any data budget","One GAN distillation, zero re-distillation cost","Data-to-model: a GAN beats pixel-wise on 15 datasets","Store a GAN, not synthetic images: re-distill for free"],"cache_read_input_tokens":28800,"weakest_assumption_plain":"The method assumes a pre-trained class-conditional generator is already available for the target dataset's classes and resolution; if that generator was trained on the full dataset, the expensive pretraining has already encoded the data the method claims to distill.","fun_headline_variants_meta":{"raw":{"variants":["Distill a generator once, serve any data budget","One GAN distillation, zero re-distillation cost","Data-to-model: a GAN beats pixel-wise on 15 datasets","Store a GAN, not synthetic images: re-distill for free"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00035,"raw_usage":{"total_tokens":1895,"prompt_tokens":913,"completion_tokens":982,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":911}},"tokens_in":529,"tokens_out":982,"duration_ms":9773,"temperature":1.0,"reasoning_tokens":911,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:09:07.568107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Distill with a generator that was pretrained without access to the target dataset's classes (for example, a BigGAN trained only on a disjoint subset of ImageNet, evaluated on CIFAR-10 or TinyImageNet classes), then measure IPC-10 accuracy; if it collapses toward random while the paper's numbers stand, the prior full-data pretraining of the generator is the source of the distilled knowledge.","supporting_citations":[{"cited_title":"In: Advances in Neural Information Processing Systems (2022)","cited_arxiv_id":null,"evidence_quote":"Feature-regression baseline that is D2M's closest high-resolution competitor on ImageNet subsets."}],"review_version":1}