Pith. sign in

REVIEW 4 major objections 5 minor 48 references

DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a simple staggered optimization schedule—early synthetic images get 4,000 iterations, later ones as few as 500—increases intra-class diversity in dataset distillation, raising ImageNet-1K top-1 accuracy with IPC 50…

desk verdict The EarlyLate schedule is a simple and effective compute-saving trick, but the paper's headline claims of consistent SOTA gains and >5% diversity improvement are not supported by its own tables or by any reported metric. read the letter →

arxiv 2411.19946 v2 pith:EMXLVV3G submitted 2024-11-29 cs.CV cs.AIcs.LG

Zhiqiang Shen , Ammar Sherif , Zeyuan Yin , Shitong Shao This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords datasetdistillationsyntheticdatagenerationintra-classdiversitybatch-to-globalmatchingEarlyLatetrainingscheduleimageoptimizationNet-1K
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dataset distillation tries to compress a large training set into a few synthetic images per class that still train accurate models. In the batch-to-global matching variant, each synthetic image is optimized against global statistics of a frozen pretrained teacher, and a known failure mode is that images within the same class end up looking alike because they all receive the same supervision and the same number of optimization steps. This paper claims that a simple change—starting images at different times so that early images receive many more gradient steps (up to 4,000) than late ones (as few as 500)—breaks that uniformity. The resulting EarlyLate (DELT) images show lower intra-class cosine similarity, and across CIFAR-10, Tiny-ImageNet, ImageNet-100, ImageNette, and ImageNet-1K, the method reports average accuracy gains of 2 to 5 percentage points over prior methods while using roughly two-thirds of the optimization iterations. If the claim holds, dataset distillation becomes both more accurate and cheaper at large scale, with a method that requires no extra networks or matching signals beyond a frozen teacher and real image patches chosen at medium teacher confidence.

What carries the argument

The load-bearing object is the EarlyLate training schedule, defined by two numbers: maximum iteration MI=4,000 for the earliest batch and round iteration RI=500 as the gap between adjacent batches. Concretely, the procedure starts one gradient loop; after every RI iterations a new batch of synthetic images joins the optimization, so the first batch accumulates 4,000 steps, the second 3,500, and so on down to 500 for the last batch. All batches are concatenated in the objective, and a BatchNorm-distribution regularization term from the frozen pretrained teacher (the same term used in SRe2L) keeps the images realistic. The teacher also serves as a ranker to initialize each synthetic image from a real patch whose class probability sits near the class median, which the authors argue gives medium-difficulty starting points with room for the distillation gradients to add information. The schedule is what creates the claimed diversity: different images are optimized to different depths, so they settle into different local optima instead of converging to the same prototype.

What would settle it

A reader could settle the central claim in one experiment: with total compute matched, compare DELT against a baseline that gives every synthetic image the same MI=4,000 iterations but uses the same teacher-ranked medium-probability real patches as initialization, and measure both IPC-10 and IPC-50 top-1 accuracy on ImageNet-1K. If the equal-iteration baseline matches or beats DELT, the reported gains come from average iteration count or better initialization rather than from the stagger itself. A second check is the diversity metric: DELT predicts more than 5% lower intra-class cosine similarity to class centroids than SRe2L/CDA; on any dataset where that gap fails to appear, the paper's account predicts the accuracy gain should also disappear.

Watch

Extended reading notes

Core claim

The central discovery, on the paper's own terms, is that in batch-to-global dataset distillation—where each synthetic image is optimized against global statistics of a frozen pretrained teacher—the uniformity of synthetic images within a class is not fixed by the matching objective itself; it is a by-product of giving every image the same full-length optimization trajectory. DELT partitions the images-per-class budget into consecutive batches and runs them through one shared gradient loop, with the first batch optimized for the full MI=4,000 iterations and each later batch entering later so it receives fewer iterations, down to RI=500. Because the later batches are concatenated with the already-optimized earlier images in the synthesis objective, each batch is distilled against a slightly different surrounding distribution, and each image ends at a different phase of optimization—some heavily compressed, some close to the original patch. The paper reports that this increases intra-class diversity by more than 5% as measured by cosine similarity to class centroids, improves top-1 accuracy by 2 to 5 percentage points on average across five datasets and multiple IPC budgets, and yields 66.1% on ImageNet-1K with IPC 50 under ResNet-101, 4.9% above RDED, with up to 39.3% less synthesis time.

Load-bearing premise

The gain rests on the assumption that a single fixed stagger—4,000 steps for the first image down to 500 for the last, chosen by a small grid search on ImageNet-1K—creates complementary information on every dataset and IPC budget, rather than simply producing noisier late images, and that the frozen teacher's class probability reliably identifies which real patches make the best starting points.

Editorial extensions

If this is right

  • Because DELT is a schedule rather than a new matching loss, it can be layered onto SRe2L- or CDA-style batch-to-global synthesis; the paper's ablations show it adds roughly 1 to 2 points over real-patch initialization alone and 2.4 points even without initialization.
  • Distilled datasets made with DELT retain cross-architecture transfer: gains appear on ResNet-18, EfficientNet-B0, MobileNet-V2, MnasNet, and RegNet-Y-8GF at IPC 10, so the diversity is not memorizing the teacher backbone.
  • The computation saving is structural: total optimization iterations drop to roughly two-thirds of SRe2L/CDA's N times T, with measured wall-clock reductions up to 39.3% on ImageNet-1K, 32% on Tiny-ImageNet, and about 1% on CIFAR-10 where data handling dominates.
  • Downstream, the more diverse distilled set improves data-free pruning (for example, +4.7 points over RDED at IPC 10) and 100-step class-incremental learning on ImageNet-1K by about 10% on average over G-VBSM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's reported benchmarks, the proposed mechanism predicts that an EarlyLate schedule should also improve other batch-to-global objectives, such as Wasserstein-metric matching or class-wise soft-label alignment, because the diversity mechanism lives in the schedule rather than in the BatchNorm-statistic loss; testing DELT on those frameworks would show whether the gain generalizes.
  • The schedule was fixed by a small grid search on ImageNet-1K (MI 1K/2K/4K by RI 500/1K), so a natural testable extension is an adaptive schedule that sets each batch's iteration count from per-image difficulty or dataset size; the paper gives no theory for why 4,000/500 is optimal, and an adaptive policy could do better or reveal the choice as dataset-dependent.
  • The medium-probability initialization is a second load-bearing choice. If teacher confidence is a noisy difficulty signal, an ensemble of rankers or a confidence-calibrated teacher would change the initialization pool; the paper does not test ranker sensitivity, so a reader should treat the initialization benefit as provisional.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DELT, a simple diversity-driven EarlyLate training scheme for dataset distillation in the batch-to-global matching paradigm. The method initializes synthetic images from teacher-ranked real image patches and then optimizes different IPC subsets with progressively fewer iterations (MI = 4,000 for the earliest batch, RI = 500 for later batches), concatenating them into a single synthetic dataset. The authors claim this increases intra-class diversity by more than 5%, improves accuracy by 2 to 5 percentage points on average over prior state-of-the-art methods, and reduces synthesis time by up to 39.3%. Experiments are reported on CIFAR-10, Tiny-ImageNet, ImageNet-100, ImageNette, and ImageNet-1K across several backbones, with the headline result of 66.1% on ImageNet-1K IPC=50 with ResNet-101, outperforming RDED by 4.9%.

Significance. If the reported gains are robust, DELT is a useful contribution to large-scale dataset distillation: it is conceptually simple, orthogonal to existing diversity-enhancement methods, and it also cuts synthesis compute, which is a practical concern in the field. The paper's strengths include extensive comparisons with error bars across many datasets, IPC settings, and backbones; ablations that isolate the EarlyLate effect (Tables 4c, 4d, 9); a computational analysis with actual wall-clock times (Table 6); and a public code release. The main significance is the demonstration that a variable-iteration schedule can improve distilled-data diversity without additional models or matching signals. However, the significance is diminished by the paper's unsupported quantitative diversity claim and by clear counterexamples in its own Table 1 where DELT loses to RDED by large margins, so the central claim of consistent 2-5% gains is not established as stated.

major comments (4)
  1. [Abstract and Section 1, Fig. 2] The abstract's claim of 'increasing diversity per class by more than 5%' is not supported by any reported numerical diversity metric. The only evidence in Fig. 2 left is a cosine-similarity curve for SRe2L, CDA, and DELT, which omits the main competitor RDED and provides no numeric values, error bars, or protocol description. Please report the diversity metric (e.g., average pairwise intra-class cosine similarity or variance) with numbers and include RDED, or remove the quantitative '>5%' claim from the abstract.
  2. [Table 1, ImageNet-100 IPC=10 rows] The paper claims in Section 4.1 to establish state-of-the-art in 13 out of 15 configurations, and the abstract claims average gains of 2-5%, but Table 1 shows DELT trailing RDED by 7.8 points on ResNet-18 (28.2 vs 36.0), 11.5 points on ResNet-101 (22.4 vs 33.9), and 7.8 points on MobileNet-V2 (15.8 vs 23.6) for ImageNet-100 with IPC=10. These are not edge cases; they are three backbone evaluations on a benchmark explicitly covered by the paper, using the same MI/RI schedule. The paper needs to explain this failure or qualify the headline claims so that they accurately reflect the observed variability.
  3. [Section 4.3, Table 4c and Appendix Table 8b] The MI=4,000/RI=500 schedule is selected from a six-cell grid on ImageNet-1K (Table 4c) and then applied unchanged to all datasets and IPCs (Appendix Table 8b). The load-bearing assumption that this single schedule transfers is already contradicted by the ImageNet-100 IPC=10 results in Table 1. Please provide per-dataset sensitivity analysis for MI/RI, or explicitly treat these as dataset-specific hyperparameters and report how they were chosen for each benchmark.
  4. [Section 3, Selection Criteria and Table 4b] The 'medium probability' initialization hypothesis is supported by a difference of only 0.12 points over 'lowest probability' (57.67 vs 57.55) and 0.64 points over 'highest probability' (57.67 vs 57.03) in Table 4b. These gaps are within the typical run-to-run variation of distillation experiments, yet the text states this strategy 'achieves the best accuracy' and links it to the diversity mechanism. Please report error bars for these ablations and soften the claim unless the difference is statistically meaningful.
minor comments (5)
  1. [Title and Abstract] The title on page 1 reads 'Diversity-drivenEarlyLate' with a missing space, and the abstract's sentence 'Our approach is conceptually simple yet effective, it partitions...' would benefit from a semicolon or period instead of a comma.
  2. [Figure 2] The left subfigure's y-axis is unlabeled and the caption does not state which features were used for the cosine similarity computation; please clarify what is plotted and why lower values are better.
  3. [Table 1] For CIFAR-10 with MobileNet-V2, SRe2L results are omitted without explanation; please indicate whether this is due to scalability, missing values, or a formatting choice.
  4. [Appendix B, Ranking] The statement that for ImageNet-100 the medium images are extracted 'based on the rankings of the original ImageNet-1K' rather than a teacher trained on ImageNet-100 is surprising and should be justified, since the ranking criterion is supposed to reflect the teacher's difficulty assessment of the target dataset.
  5. [Section 4.4, Computational Analysis] The formula for total iterations 'N × T − j(j−1)/2 RI' is stated without derivation; please define j and show how it relates to M and RI, and verify the arithmetic for the reported 39.3% saving with j=8 (or whichever value is used).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central claims are empirical and benchmarked against external methods; self-citations are to independent peer-reviewed base frameworks.

full rationale

The paper makes no first-principles derivation that reduces to its own inputs. The central accuracy claims are evaluated against external baselines (RDED, MTT, TESLA, DATM, SRe2L, etc.) under the standard protocol inherited from prior work, so the reported gains are not constructed from the method's own definitions. The EarlyLate schedule is a heuristic; MI=4000 and RI=500 are chosen via the ablation in Table 4c on ImageNet-1K IPC 10 and then transferred to other settings. This is hyperparameter selection and a generalization risk, not a fitted parameter that is later renamed as a prediction: the same schedule empirically loses to RDED on several ImageNet-100 IPC 10 rows, showing the reported outcomes are not forced by construction. The self-citations to SRe2L, CDA, and G-VBSM supply base frameworks, initialization pipelines, and regularization terms; these are peer-reviewed, code-released, and externally benchmarked, and the paper does not invoke them as uniqueness theorems or as the sole support for its novelty. The abstract's 'diversity per class by more than 5%' is not numerically documented and Fig. 2 omits RDED, and the '13 out of 15' tally is strained by Table 1 losses on ImageNet-100 IPC 10; these are evidence and correctness concerns, not circular reductions. No step of the claimed derivation is equivalent to its input by definition.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claim depends on assumptions inherited from SRe2L/CDA (teacher BN statistics and soft labels) plus the paper's own design assumptions (medium-probability initialization; variable iteration counts create diversity). The main free parameters MI, RI, augmentation settings, and initial selection are chosen via ablations on the same benchmark family. No fundamentally new entity is introduced.

free parameters (7)
  • MI (maximum iterations for earliest batch) = 4000
    Selected via ablation on ImageNet-1K (Table 4c); controls how long the earliest synthetic images are optimized.
  • RI (round iterations) = 500
    Selected via ablation; controls the iteration gap between adjacent batches and the iteration count of the final batch.
  • IPC=1 recovery iterations = Dataset and backbone dependent, e.g., 2K/3K for CIFAR-10 ResNet-18/101, 3K for ImageNet-1K Conv4
    Hand-set per dataset and validation backbone (Table 8c); IPC=1 cannot use the EarlyLate schedule.
  • alpha_BN = 0.01
    Weight of the BatchNorm distribution regularization in Eq. 8; inherited from SRe2L and fixed.
  • RandAugment settings = m=4-6, n=2-4 per dataset
    Dataset-specific augmentation hyperparameters in Table 8c.
  • Initialization selection criterion = Medium probability patches, median-expanding order
    Chosen by ablations (Tables 4b, 10, 11); the teacher ranker's medium-probability crops initialize the synthetic images.
  • Mosaic pattern = 1x1
    Ablation in Table 4a shows 1x1 patch initialization outperforms 2x2 to 5x5.
assumptions (4)
  • domain assumption BatchNorm running statistics of a pretrained teacher model summarize the global data distribution well enough to supervise synthetic image synthesis.
    Used in Eq. 8 and throughout the synthesis objective; inherited from SRe2L and CDA.
  • ad hoc to paper The teacher model's class probability provides a valid difficulty ranking; medium-probability patches have more room for information enhancement via distillation gradients.
    Quoted in Section 3 (Initialization, Selection Criteria); not independently tested beyond the ablations in Tables 4b, 10, 11.
  • ad hoc to paper Varying the number of optimization iterations across synthetic images in a class creates complementary information rather than just varying image quality.
    The core EarlyLate hypothesis; supported only by empirical ablations (Tables 4d, 9), with no theoretical argument.
  • domain assumption Post-training with soft labels from a pretrained teacher and the standard evaluation protocol measures the quality of a distilled dataset.
    Evaluation setup follows SRe2L/RDED; accepted convention in the subfield.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation." pith.science (2026). https://pith.science/paper/EMXLVV3G

@misc{pith2026241119946,
  author       = {Pith},
  title        = {Pith review of: DELT: A Simple Diversity-driven EarlyLate Training for Dataset Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EMXLVV3G}},
  note         = {Machine review of arXiv:2411.19946}
}
abstract

Recent advances in dataset distillation have led to solutions in two main directions. The conventional batch-to-batch matching mechanism is ideal for small-scale datasets and includes bi-level optimization methods on models and syntheses, such as FRePo, RCIG, and RaT-BPTT, as well as other methods like distribution matching, gradient matching, and weight trajectory matching. Conversely, batch-to-global matching typifies decoupled methods, which are particularly advantageous for large-scale datasets. This approach has garnered substantial interest within the community, as seen in SRe$^2$L, G-VBSM, WMDD, and CDA. A primary challenge with the second approach is the lack of diversity among syntheses within each class since samples are optimized independently and the same global supervision signals are reused across different synthetic images. In this study, we propose a new Diversity-driven EarlyLate Training (DELT) scheme to enhance the diversity of images in batch-to-global matching with less computation. Our approach is conceptually simple yet effective, it partitions predefined IPC samples into smaller subtasks and employs local optimizations to distill each subset into distributions from distinct phases, reducing the uniformity induced by the unified optimization process. These distilled images from the subtasks demonstrate effective generalization when applied to the entire task. We conduct extensive experiments on CIFAR, Tiny-ImageNet, ImageNet-1K, and its sub-datasets. Our approach outperforms the previous state-of-the-art by 2$\sim$5% on average across different datasets and IPCs (images per class), increasing diversity per class by more than 5% while reducing synthesis time by up to 39.3% for enhancing the training efficiency. Code is available at: https://github.com/VILA-Lab/DELT.

Figures

Figures reproduced from arXiv: 2411.19946 by the authors.

Figure 1
Figure 1. Distilling datasets to IPCN requires N ×T iterations in traditional distillation processes (left) but fewer iterations in our EarlyLate strategy (right). IPC1:N represents a set of images from 1 to N. The red shaded area is our saved computational cost. cessible and affordable for the general public. Previous approaches [3, 4, 6, 14, 18, 21, 36, 42, 43, 47] primarily employ a batch-to-batch matching technique, where… view at source ↗
Figure 2
Figure 2. Left: Intra-class semantic cosine similarity after a pretrained ResNet-18 model on ImageNet-1K dataset, lower values are better. Right: Synthetic images from SRe2L, CDA and our DELT. running means and variances from the pre-trained model for supervising different intra-class image synthesis. This methodology results in a severely limited diversity within the same category of generated images. To address this issue, … view at source ↗
Figure 3
Figure 3. Batch–to-batch vs. batch-to-global matching in DD. θf indicates weights are pretrained and frozen in synthesis stage. tures [42], gradients [43], or the model weight trajecto￾ries [3, 6]. The primary optimization method used is bi-level optimization [20, 41], which involves optimizing model pa￾rameters and updating images simultaneously. For instance, using gradient matching, the process can be formulated as to mini… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: The proposed DELT learning procedure via a multi-round [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Selection criteria with a teach ranker. Diversity-driven IPC Concatenation Training. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Mosaic splicing patterns on ImageNet-1K using real [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: Distilled dataset visualization compared with other image optimization-based methods. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Continual learning results. periments on ImageNet-1K, comparing our results with the baselines G-VBSM and SRe2L. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Synthetic image visualizations on Tiny-ImageNet gen [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Synthetic image visualizations on ImageNette gener [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 12
Figure 12. Figure 12: Visualization of computation time consumption on our [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 11
Figure 11. Figure 11: Synthetic images on CIFAR-10 generated by our DELT. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 29 canonical work pages

  1. [1]

    Understand- ing and improving early stopping for learning with noisy la- bels

    Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understand- ing and improving early stopping for learning with noisy la- bels. Advances in Neural Information Processing Systems , 34:24392–24403, 2021. 4

    work page 2021

  2. [2]

    Yolov4: Optimal speed and accuracy of object detection

    Alexey Bochkovskiy, Chien-Yao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934, 2020. 5

    arXiv 2004

  3. [3]

    Dataset distillation by matching training trajectories

    George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4750–4759, 2022. 1, 2, 3, 5, 6, 7

    work page 2022

  4. [4]

    Data distillation can be like vodka: Distilling more times for better quality

    Xuxi Chen, Yu Yang, Zhangyang Wang, and Baharan Mirza- soleiman. Data distillation can be like vodka: Distilling more times for better quality. In The Twelfth International Confer- ence on Learning Representations, 2024. 1, 2, 3

    work page 2024

  5. [5]

    Randaugment: Practical automated data augmen- tation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmen- tation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703, 2020. 11

    2020

  6. [6]

    Scaling up dataset distillation to imagenet-1k with constant memory

    Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet-1k with constant memory. In International Conference on Machine Learning , pages 6565–6590. PMLR, 2023. 1, 2, 3, 5, 6

    work page 2023

  7. [7]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 5

    2009

  8. [8]

    Diversity-driven synthesis: Enhancing dataset distillation through directed weight adjustment

    Jiawei Du, Xin Zhang, Juncheng Hu, Wenxin Huang, and Joey Tianyi Zhou. Diversity-driven synthesis: Enhancing dataset distillation through directed weight adjustment. In Advances in neural information processing systems, 2024. 3

    work page 2024

Show all 48 references
  1. [9]

    Fastai/imagenette: A smaller subset of 10 easily clas- sified classes from imagenet, and a little more french

    Fastai. Fastai/imagenette: A smaller subset of 10 easily clas- sified classes from imagenet, and a little more french. 5

  2. [10]

    Dynamic few-shot visual learning without forgetting

    Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4367–4375, 2018. 5

    2018

  3. [11]

    Efficient dataset distillation via minimax diffusion

    Jianyang Gu, Saeed Vahidian, Vyacheslav Kungurtsev, Hao- nan Wang, Wei Jiang, Yang You, and Yiran Chen. Efficient dataset distillation via minimax diffusion. InCVPR, 2024. 6, 7

    2024

  4. [12]

    Towards lossless dataset distillation via difficulty-aligned trajectory matching

    Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. In The Twelfth In- ternational Conference on Learning Representations , 2024. 5, 6

    2024

  5. [13]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 5

    2016

  6. [14]

    Multisize dataset condensation

    Yang He, Lingao Xiao, Joey Tianyi Zhou, and Ivor Tsang. Multisize dataset condensation. ICLR, 2024. 1, 2, 3

    2024

  7. [15]

    Dataset condensation via efficient synthetic- data parameterization

    Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic- data parameterization. In Proceedings of the 39th Interna- tional Conference on Machine Learning, 2022. 7

    2022

  8. [16]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 5

    2009

  9. [17]

    Tiny imagenet visual recognition challenge

    Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015. 5

    2015

  10. [18]

    Dataset condensation with con- trastive signals

    Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with con- trastive signals. In International Conference on Machine Learning, pages 12352–12364. PMLR, 2022. 1, 2

    2022

  11. [19]

    Dataset distillation via the wasserstein metric

    Haoyang Liu, Tiancheng Xing, Luwei Li, Vibhu Dalal, Jin- grui He, and Haohan Wang. Dataset distillation via the wasserstein metric. arXiv preprint arXiv:2311.18531, 2023. 1, 3

    2023 arXiv

  12. [20]

    Investigating bi-level optimization for learn- ing and vision from a unified perspective: A survey and be- yond

    Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimization for learn- ing and vision from a unified perspective: A survey and be- yond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):10045–10067, 2021. 3

    2021

  13. [21]

    Dataset distillation via factorization

    Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xin- chao Wang. Dataset distillation via factorization. Advances in Neural Information Processing Systems , 35:1100–1113,

  14. [22]

    Learning efficient convolutional networks through network slimming

    Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Pro- ceedings of the IEEE international conference on computer vision, pages 2736–2744, 2017. 8

    2017

  15. [23]

    Pytorch: An im- perative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems ...

    2019

  16. [24]

    Early stopping-but when? In Neural Net- works: Tricks of the trade, pages 55–69

    Lutz Prechelt. Early stopping-but when? In Neural Net- works: Tricks of the trade, pages 55–69. Springer, 2002. 4

    2002

  17. [25]

    Distribu- tional dataset distillation with subtask decomposition

    Tian Qin, Zhiwei Deng, and David Alvarez-Melis. Distribu- tional dataset distillation with subtask decomposition. arXiv preprint arXiv:2403.00999, 2024. 3

    2024 arXiv

  18. [26]

    Designing network design spaces

    Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Doll ´ar. Designing network design spaces. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 10428–10436,

  19. [27]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 4510–4520, 2018. 5 9

    2018

  20. [28]

    Generalized large-scale data condensa- tion via various backbone and statistical matching

    Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensa- tion via various backbone and statistical matching. InCVPR,

  21. [29]

    A fast knowledge distillation framework for visual recognition

    Zhiqiang Shen and Eric Xing. A fast knowledge distillation framework for visual recognition. In European Conference on Computer Vision, pages 673–690. Springer, 2022. 11

    2022

  22. [30]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 8

    2014 arXiv

  23. [31]

    Data-free param- eter pruning for deep neural networks

    Suraj Srinivas and R Venkatesh Babu. Data-free param- eter pruning for deep neural networks. arXiv preprint arXiv:1507.06149, 2015. 7

    2015 arXiv

  24. [32]

    On the diversity and realism of distilled dataset: An efficient dataset distilla- tion paradigm

    Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distilla- tion paradigm. In CVPR, 2024. 2, 4, 5, 6, 7, 8

    2024

  25. [33]

    Efficientnet: Rethinking model scaling for convolutional neural networks

    Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105–6114. PMLR,

  26. [34]

    Mnas- net: Platform-aware neural architecture search for mobile

    Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnas- net: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF conference on computer vi- sion and pattern recognition, pages 2820–2828, 2019. 5

    2019

  27. [35]

    Con- trastive multiview coding

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Con- trastive multiview coding. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XI 16 , pages 776–794. Springer,

    2020

  28. [36]

    Dataset distillation

    Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018. 1, 2

    2018 arXiv

  29. [37]

    Mean squared error: Love it or leave it? a new look at signal fidelity measures

    Zhou Wang and Alan C Bovik. Mean squared error: Love it or leave it? a new look at signal fidelity measures. IEEE signal processing magazine, 26(1):98–117, 2009. 3

    2009

  30. [38]

    Are large-scale soft labels nec- essary for large-scale dataset distillation? In Advances in neural information processing systems, 2024

    Lingao Xiao and Yang He. Are large-scale soft labels nec- essary for large-scale dataset distillation? In Advances in neural information processing systems, 2024. 3

    2024

  31. [39]

    Dataset distillation via cur- riculum data synthesis in large data era

    Zeyuan Yin and Zhiqiang Shen. Dataset distillation via cur- riculum data synthesis in large data era. Transactions on Machine Learning Research. 1, 3, 5, 6, 7, 8, 11

  32. [40]

    Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective

    Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. In NeurIPS, 2023. 1, 3, 5, 6, 7, 8, 11

    2023

  33. [41]

    An introduc- tion to bi-level optimization: Foundations and applications in signal processing and machine learning

    Yihua Zhang, Prashant Khanduri, Ioannis Tsaknakis, Yuguang Yao, Mingyi Hong, and Sijia Liu. An introduc- tion to bi-level optimization: Foundations and applications in signal processing and machine learning. arXiv preprint arXiv:2308.00788, 2023. 3

    2023 arXiv

  34. [42]

    Dataset condensation with dis- tribution matching

    Bo Zhao and Hakan Bilen. Dataset condensation with dis- tribution matching. In IEEE/CVF Winter Conference on Ap- plications of Computer Vision, WACV 2023, Waikoloa, HI, USA, January 2-7, 2023, 2023. 1, 2, 3, 7

    2023

  35. [43]

    Dataset condensation with gradient matching

    Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. arXiv preprint arXiv:2006.05929, 2020. 1, 2, 3, 5

    2006 arXiv

  36. [44]

    Decoupled knowledge distillation

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953–11962, 2022. 6

    2022

  37. [45]

    Im- proved distribution matching for dataset condensation

    Ganlong Zhao, Guanbin Li, Yipeng Qin, and Yizhou Yu. Im- proved distribution matching for dataset condensation. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 7856–7865, 2023. 5, 6

    2023

  38. [46]

    Self-supervised dataset distillation: A good compression is all you need

    Muxin Zhou, Zeyuan Yin, Shitong Shao, and Zhiqiang Shen. Self-supervised dataset distillation: A good compression is all you need. arXiv preprint arXiv:2404.07976, 2024. 7

    2024 arXiv

  39. [47]

    Dataset distillation using neural feature regression.Advances in Neu- ral Information Processing Systems, 35:9813–9827, 2022

    Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression.Advances in Neu- ral Information Processing Systems, 35:9813–9827, 2022. 1, 2 10 Appendix A. Limitations Our method effectively avoids the issue of insufficient data diversity g...

    2022

  40. [48]

    Hyper-parameter settings

    In each figure, each column represents a different class, with images progressing from long optimization at the top 11 (a) Validation settings config value optimizer AdamW base learning rate 0.001 (all) 0.0025 (MobileNet-v2) weight decay 0.01 batch size 100 (IPC 50) 50 (IPC 10...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.