Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By moving residual skip connections from networks to the data itself, FADRM claims a dataset-distillation state of the art: 47.7% and 50.0% top-1 accuracy on ImageNet-1K at 0.8% compression, with compute roughly halved.

desk verdict The data-residual idea is worth taking seriously, but the paper's headline margin over EDC rests on best-of-four evaluation with no error bars, so the SOTA claim is not yet trustworthy. read the letter →

arxiv 2506.24125 v1 pith:RO6C24AJ submitted 2025-06-30 cs.CV cs.AI

Jiacheng Cui , Xinyue Bi , Yaxin Luo , Xiaohan Zhao , Jiacheng Liu , Zhiqiang Shen This is my paper
classification cs.CVcs.AI
keywords datasetdistillationdataresidualmatchingdata-levelskipconnectionsinformationvanishinguni-leveloptimizationmixedprecisiontrainingmulti-resolutionImageNet-1K
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dataset distillation compresses a huge training set into a tiny synthetic one, but the cheapest large-scale methods optimize the synthetic images without ever looking at the originals, so information about the real data steadily fades — the paper calls this 'information vanishing'. FADRM's central proposal is to carry residual connections over to the data itself: periodically blend the intermediate synthetic image with resized patches of the original images during optimization, so the distilled set keeps both newly learned detail and the original's local structure. Because the blending is cheap and the optimization can be run at reduced resolution with mixed-precision arithmetic, the method claims roughly a 50% cut in training time and peak GPU memory. If the results hold, the payoff is that large-scale, high-quality distilled datasets become substantially cheaper to produce, with a compact set of a few images per class training a model to 47.7-50.0% top-1 accuracy on ImageNet-1K.

What carries the argument

The central object is the Adjustable Residual Connection (ARC), a data-level skip connection that at each injection stage replaces the intermediate synthetic image with a convex blend of itself and the resampled original patch: $\tilde{x}_{ib} = \alpha\,\tilde{x}_{ib} + (1-\alpha)\,\mathrm{Resample}(P_s, D_{ib})$. It is the mechanism supposed to stop information vanishing and to deliver the tighter generalization bound, with $\alpha$ controlling how much original-detail is retained. Two efficiency components carry the cost claims: Multi-Resolution Optimization (MRO), which runs roughly half of the optimization steps at a downsampled resolution such as 200x200 instead of 224x224 and then rescales back, and Mixed-Precision Training (MPT), which computes logits and cross-entropy in FP16 while keeping the divergence-to-statistics loss and the gradients with respect to the image in FP32.

What would settle it

Run the paper's own ImageNet-1K IPC=10 recipe (ResNet-18 teacher, $\alpha = 0.5$, $k = 3$, $D_{ds} = 200$) and empirically estimate the two terms of condition (11): the Rademacher complexity of the hypothesis class on the original patches versus on the optimized-only synthetic images, and the average discrepancy $\Delta$ between them. If the left-hand side of the inequality is not negative enough, the theorem's conclusion does not follow from its stated assumptions. Independently, the information-vanishing mechanism is directly testable: feature-level entropy of the distilled images should fall during optimization and be restored at each residual injection, which a reader could reproduce with any pretrained feature extractor.

Watch

Extended reading notes

Core claim

FADRM claims that data-level residual matching solves the information-vanishing problem of uni-level dataset distillation, where synthesis is driven by matching a pretrained network's running BatchNorm statistics and prediction alignments without direct access to the original data. The paper shows (Theorem 1) that such pipelines are fundamentally information-starved: the mutual information between the synthetic set and the original set is bounded by $|C|\cdot H(f_\theta)$, the synthetic-set size times the maximum per-sample entropy of the pretrained network, so an over-confident model caps how much the distilled set can retain. Against this, FADRM injects the original data through an adjustable residual connection (ARC) that repeatedly fuses the optimized image with resampled original patches weighted by a merge ratio $\alpha \approx 0.5$, and the paper argues (Theorem 2) that this blending tightens a Rademacher-complexity generalization bound — a measure of how expressive the hypothesis class is — relative to optimizing alone. Empirically the package of ARC plus multi-resolution optimization and mixed-precision training reports state-of-the-art accuracy on CIFAR-100, Tiny-ImageNet, ImageNet subsets, and ImageNet-1K, with headline numbers of 47.7% (single-model) and 50.0% (multi-model) top-1 accuracy on ImageNet-1K at IPC=10 using ResNet-18, together with roughly 50% reductions in optimization time and peak GPU memory.

Load-bearing premise

The load-bearing premise is Theorem 2's condition (11), an inequality requiring that blending in the original patches shrink the model's effective hypothesis class by more than a threshold set by the Lipschitz constants, the loss bound, the merge ratio, and the discrepancy between optimized and real images. The paper never verifies this condition on any actual data or model, and the derivation leading to it contains a slip where '$L + h$' appears in place of the product $Lh$; until the condition is checked and that slip is corrected, the theory alone does not carry the claim that the residual connection improves generalization.

Editorial extensions

If this is right

  • If FADRM's claims hold, ImageNet-scale dataset distillation becomes feasible on a single consumer GPU: generating the 50-IPC ImageNet-1K set would take about 54 hours less than EDC and roughly 28 hours less than SRe2L++, with peak memory below 3 GB for the single-model variant.
  • A distilled set built from about 0.8% of ImageNet-1K (10 images per class) would train ResNet-18 to 47.7% top-1 accuracy, and an ensemble of four teachers (FADRM+) would push that to 50.0%; the same data transfers across eleven tested architectures, from ShuffleNetV2 at 30.2% to DenseNet201 at 59.7%.
  • Data-level residual connections become a reusable ingredient: any uni-level distillation objective, whether statistics matching, prediction matching, or their combination, can be run with periodic residual injection at negligible extra cost, since ARC is a per-element tensor fusion of complexity $O(H_t W_t C)$.
  • Information vanishing has a measurable signature — feature-level entropy of the distilled images falls as optimization proceeds and the residual connection restores it — so the mechanism is checkable independently of final accuracy.
  • The ablations put the optimal settings at $\alpha = 0.5$ and $k = 3$ residual injections, so the reported headline gains are tied to these specific hyperparameters rather than to the residual idea alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension of the paper's logic: replace the real patch in the residual blend with its pixel-shuffled or Fourier-shuffled version; if accuracy holds, the residual's benefit is statistical (color, energy, texture) rather than semantic, which would tell practitioners what ARC is actually preserving.
  • The Theorem 1 bound suggests a pragmatic diagnostic for any distillation run: measure the per-class output entropy of the pretrained teacher on the synthetic set; runs sitting near the bound $|C|\cdot H(f_\theta)$ are prime candidates for residual injection regardless of the distillation objective used.
  • Using the paper's own quantities, one can empirically test Theorem 2's condition (11): estimate the Rademacher-complexity difference between the original patch set and the optimized-only synthetic set on a real distillation run; if the inequality fails, the empirical gains stand but the theory, as written, does not yet explain them.
  • The paper itself flags that data-level residual signals may not transfer to non-visual modalities such as text or time series; applying ARC to those domains, or to iterative synthesis loops like generative replay in continual learning, would clarify how general the mechanism is.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes FADRM, a dataset distillation method that combines three components: mixed-precision training (MPT), multi-resolution optimization (MRO), and adjustable residual connections (ARC). Synthetic images are initialized from real patches, optimized against a pretrained model at alternating resolutions, and periodically fused with resampled original patches via a merge ratio α. The authors report state-of-the-art results on CIFAR-100, Tiny-ImageNet, ImageNet-1K and its subsets, including 47.7% (single-model) and 50.0% (multi-model) top-1 accuracy on ImageNet-1K at IPC=10 with ResNet-18, alongside roughly 50% reductions in generation time and peak GPU memory. Two theoretical results are stated: Theorem 1 bounds the mutual information between uni-level synthetic data and the original dataset, and Theorem 2 gives a generalization-bound argument for ARC. The appendix provides proofs, extensive hyperparameter tables, ablations, cross-architecture generalization results, efficiency comparisons, and a continual learning application.

Significance. If the empirical claims hold, FADRM is a strong practical contribution: it is simple, efficient, and shows consistent gains across several datasets and architectures, with concrete reductions in time and memory. The paper also documents its hyperparameters carefully and makes code available, which are positive reproducibility features. However, the central SOTA claim is currently supported by an evaluation protocol that selects the best of four post-evaluation schedules and reports no error bars, and the theoretical justification of ARC depends on an unverified condition with a proof error. These issues are load-bearing for the paper's two headline claims (state-of-the-art accuracy and theoretically grounded residual connections), so the present evidence is suggestive rather than conclusive.

major comments (3)
  1. [§4.2, Table 1, Appendix F] The reported FADRM/FADRM+ numbers in Table 1 are the best of four post-evaluation settings S1–S4 described in Appendix F, and the mapping from each result to a specific setting is not given. No repeated runs or error bars are reported. For ImageNet-1K IPC=10 with ResNet-18, the abstract and Table 2 report FADRM+ at 50.0%, while Table 1 reports 50.9%, without explanation of which configuration produced which number. Since the claimed margin over EDC is only +1.4 to +2.3 percentage points, and schedule-to-schedule or seed-to-seed variation in large-scale distillation can plausibly account for that difference, the current evidence does not establish a SOTA margin over EDC. The authors should prespecify a single evaluation protocol, report means and variances over multiple seeds, and reconcile the inconsistent 50.0/50.9 numbers.
  2. [§3.4, Eq. (11), Appendix A.3] Theorem 2's conclusion is conditioned on inequality (11), Rn(H∘O) − Rn(H∘C̃res) < −LhΔ(Ll+2Bα)/(2B), but the paper never verifies this condition for any dataset, hypothesis class, or trained hypothesis, and gives no argument that it should hold in practice. Without that verification, the theorem does not establish that ARC improves generalization. In addition, the proof contains an algebraic error: Eq. (49) writes 'L + hα(1−α)·Δ' where dimensional consistency requires Lh, and the step from Eqs. (44) and (48) to Eq. (49) is not derived. Because the paper explicitly advertises the method as 'theoretically grounded,' this theorem needs to be corrected and its condition either verified or replaced by an unconditional analysis.
  3. [§3.2, §4.2, Tables 2 and 4] The efficiency comparisons mix two different method variants: FADRM uses a single pretrained model (ResNet-18), while FADRM+ uses an ensemble of four models (Table 8), yet Table 1 and the cross-architecture table compare FADRM+ against single-model baselines such as RDED. The claimed 50% reduction in time and peak memory is presented for both variants, but Table 4 reports FADRM+ peak memory as 12 GB while Table 2 reports 11.0 GB for the same variant, and the per-image time of 1.09 s for FADRM+ presumably includes all four teachers while FADRM's 0.47 s uses one. The comparison should report which variant, which teacher set, and which compute budget underlies each efficiency claim so that the gains are attributable to the method and not to an asymmetric comparison.
minor comments (6)
  1. [§3.4] The text contains a typo: 'see Theroem 2' should read 'see Theorem 2'.
  2. [§3 (Theorem 1) and Appendix A.2] Theorem 1's statement uses H(fθ) both as the maximum per-sample entropy of the network output and, in the proof, as the entropy of the model fθ; these are different quantities and should be disambiguated.
  3. [Table 2 (left) and Table 4] The baseline 'SRe2L++' is cited as reference [6], but reference [6] is the CV-DD paper; SRe2L++ is reference [43]. This citation error appears in both tables.
  4. [Eq. (8)] The saved-computation formula in Eq. (8) uses '⌊k/2⌋' without derivation; it is not immediately clear which stages of Algorithm 1 run at downsampled resolution for odd versus even k, so a short derivation or a precise counting of stages would improve clarity.
  5. [Appendix F] The post-evaluation hyperparameter tables list which settings S1–S4 are used for each dataset and model, but Table 1's cells do not indicate which setting produced each reported accuracy; a mapping table from Table 1 entries to S1–S4 would make the evaluation reproducible.
  6. [Figure 2] The diagram is dense and does not label the resolution at every stage; adding the current spatial resolution and the merge step at each data residual block would make the multi-resolution schedule easier to follow.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FADRM's derivation chain is self-contained; the theoretical bounds are conditional and the empirical claims are external-benchmark evaluations, not outputs re-derived from their own premises.

full rationale

The paper's derivation chain is not circular. Algorithm 1 and Eqs. (5)-(10) define the synthetic-image generation; Theorem 1 bounds mutual information for uni-level methods and is not used to define FADRM's output. Theorem 2 (Appendix A.3) is conditional: it states that if condition (11) on Rademacher-complexity differences holds, then the FADRM generalization bound is tighter. The condition is neither derived nor verified, and Eq. (49) contains a notational slip ('L + h' for Lh), but the conclusion is not assumed in the premise; those are correctness gaps, not circularity. The empirical SOTA comparisons are against external benchmarks with baselines re-run under a common protocol, and Table 5 shows ARC adds accuracy beyond the original-patch initialization. Two caveats are worth stating but are not circularity: (i) alpha, k, Dds, and post-evaluation schedules S1-S4 are selected on the target benchmark's post-evaluation accuracy, so reported numbers are best-of-grid statistics rather than unbiased predictions; this weakens the +1.4% margin over EDC, but the reported accuracy is not mathematically identical to the selected hyperparameters. (ii) Appendix D itself concedes the residual-signal assumption may not generalize across domains. Neither caveat makes the derivation equivalent to its inputs.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central method relies on several design choices tuned on the target datasets, and on unproven theoretical conditions. No newly invented physical entities are introduced.

free parameters (4)
  • Merge ratio α = 0.5
    Chosen from ablation on ImageNet-1K IPC=10 (Table 5); controls the trade-off between optimized image and original patch.
  • Number of ARC stages k = 3
    Chosen from ablation (Table 6 Left) on ImageNet-1K and CIFAR-100; determines frequency of residual injections.
  • Downsampled resolution Dds = 200
    Chosen from ablation (Table 6 Right) on ImageNet-1K; applies only to large-scale datasets.
  • Optimization budget B = 2000
    Set as a fixed budget following prior uni-level frameworks; not tuned but central to the recipe.
assumptions (3)
  • domain assumption The pretrained model fθ optimized on the original dataset O provides a sufficient statistic for the information needed to distill C (Markov chain O → fθ → x̃).
    Used in Theorem 1 (Appendix A.2) to bound the mutual information between C and O. This is a heuristic from uni-level frameworks, not proven.
  • ad hoc to paper The condition in Eq. (11), Rn(H∘O) − Rn(H∘C̃res) < −LhΔ(Ll+2Bα)/(2B), holds for the trained hypotheses.
    Theorem 2's conclusion that ARC improves the generalization bound depends entirely on this unverified inequality. No experiments check it.
  • standard math The loss ℓ is bounded in [0, B] and Lipschitz continuous, with the KL divergence as a specific case.
    Used in the Rademacher complexity bounds (Theorem 3, Lemma 2), with assumptions on output bounds and Lipschitz constants.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation." pith.science (2026). https://pith.science/paper/RO6C24AJ

@misc{pith2026250624125,
  author       = {Pith},
  title        = {Pith review of: FADRM: Fast and Accurate Data Residual Matching for Dataset Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RO6C24AJ}},
  note         = {Machine review of arXiv:2506.24125}
}
read the original abstract

Residual connection has been extensively studied and widely applied at the model architecture level. However, its potential in the more challenging data-centric approaches remains unexplored. In this work, we introduce the concept of Data Residual Matching for the first time, leveraging data-level skip connections to facilitate data generation and mitigate data information vanishing. This approach maintains a balance between newly acquired knowledge through pixel space optimization and existing core local information identification within raw data modalities, specifically for the dataset distillation task. Furthermore, by incorporating optimization-level refinements, our method significantly improves computational efficiency, achieving superior performance while reducing training time and peak GPU memory usage by 50%. Consequently, the proposed method Fast and Accurate Data Residual Matching for Dataset Distillation (FADRM) establishes a new state-of-the-art, demonstrating substantial improvements over existing methods across multiple dataset benchmarks in both efficiency and effectiveness. For instance, with ResNet-18 as the student model and a 0.8% compression ratio on ImageNet-1K, the method achieves 47.7% test accuracy in single-model dataset distillation and 50.0% in multi-model dataset distillation, surpassing RDED by +5.7% and outperforming state-of-the-art multi-model approaches, EDC and CV-DD, by +1.4% and +4.0%. Code is available at: https://github.com/Jiacheng8/FADRM.

Figures

Figures reproduced from arXiv: 2506.24125 by the authors.

Figure 1
Figure 1. Total training hours on a single RTX￾4090 vs. test set accuracy, comparing prior state￾of-the-art methods with our proposed framework (+ denotes multi-model distillation). In recent years, the computer vision and natural language processing communities have predom￾inantly focused on model-centric research, driv￾ing an unprecedented expansion in the scale of neural networks. Landmark developments such as LLMs and MLL… view at source ↗
Figure 2
Figure 2. Overview of FADRM. It starts by downsampling the real data patches (both 1×1 and 2×2 [32] can be used as initialization and perform well in our experiments, meanwhile imposing downsampling to reduce cost). These downsampled images are subsequently processed through a series of proposed Data Residual Blocks. Each block utilizes a pretrained model to optimize the images within a predefined optimization budget, resampl… view at source ↗
Figure 3
Figure 3. The above figures illustrate the phenomenon of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Visualization of the distilled images with [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Visualization of dataset distilled by FADRM and SRe2L++ on Tiny-ImageNet (top two rows) and ImageNet-1k (bottom two rows). Impact of Downsampled Input Size in MRO. To determine the optimal downsampled input size (Dds) for MRO, we conduct an ablation study, as presented…
Figure 6
Figure 6. Figure 6: Five-step and Ten-step class-incremental learning on Tiny-ImageNet with IPC=50. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Visualization of synthetic data on CIFAR-100 generated by [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]
Figure 8
Figure 8. Figure 8: Visualization of synthetic data on CIFAR-100 generated by [PITH_FULL_IMAGE:figures/full_fig_p026_8.png]
Figure 9
Figure 9. Figure 9: Visualization of synthetic data on Tiny-ImageNet generated by [PITH_FULL_IMAGE:figures/full_fig_p027_9.png]
Figure 10
Figure 10. Figure 10: Visualization of synthetic data on Tiny-ImageNet generated by [PITH_FULL_IMAGE:figures/full_fig_p028_10.png]
Figure 11
Figure 11. Figure 11: Visualization of synthetic data on ImageNette generated by [PITH_FULL_IMAGE:figures/full_fig_p029_11.png]
Figure 12
Figure 12. Figure 12: Visualization of synthetic data on ImageNette generated by [PITH_FULL_IMAGE:figures/full_fig_p030_12.png]
Figure 13
Figure 13. Figure 13: Visualization of synthetic data on ImageWoof generated by [PITH_FULL_IMAGE:figures/full_fig_p031_13.png]
Figure 14
Figure 14. Figure 14: Visualization of synthetic data on ImageWoof generated by [PITH_FULL_IMAGE:figures/full_fig_p032_14.png]
Figure 15
Figure 15. Figure 15: Visualization of synthetic data on ImageNet-1k generated by [PITH_FULL_IMAGE:figures/full_fig_p033_15.png]
Figure 16
Figure 16. Figure 16: Visualization of synthetic data on ImageNet-1k generated by [PITH_FULL_IMAGE:figures/full_fig_p034_16.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hard Labels In! Rethinking the Role of Hard Labels in Mitigating Local Semantic Drift

    cs.CV 2025-12 conditional novelty 6.0 of 10

    A soft-hard-soft training schedule uses hard labels as an intermediate anchor to correct local semantic drift and improves accuracy under 100x-reduced soft-label storage.

Reference graph

Works this paper leans on

51 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Rademacher and gaussian complexities: Risk bounds and structural results

    Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463–482, 2002

    work page 2002

  3. [3]

    An intuitive proof of the data processing inequality

    Normand J Beaudry and Renato Renner. An intuitive proof of the data processing inequality. arXiv preprint arXiv:1107.0740, 2011

    arXiv 2011

  4. [4]

    Dataset distillation by matching training trajectories

    George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 4750–4759, 2022. 10

    work page 2022

  5. [5]

    Dataset Distillation via Adversarial Prediction Matching

    Mingyang Chen, Bo Huang, Junda Lu, Bing Li, Yi Wang, Minhao Cheng, and Wei Wang. Dataset distillation via adversarial prediction matching. arXiv preprint arXiv:2312.08912, 2023

    work page Pith review arXiv 2023

  6. [6]

    Dataset distillation via committee voting

    Jiacheng Cui, Zhaoyi Li, Xiaochen Ma, Xinyue Bi, Yaxin Luo, and Zhiqiang Shen. Dataset distillation via committee voting. arXiv preprint arXiv:2501.07575, 2025

    arXiv 2025

  7. [7]

    Scaling up dataset distillation to imagenet- 1k with constant memory

    Justin Cui, Ruochen Wang, Si Si, and Cho-Jui Hsieh. Scaling up dataset distillation to imagenet- 1k with constant memory. In International Conference on Machine Learning, pages 6565–6590. PMLR, 2023

    work page 2023

  8. [8]

    Imagenet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

Show all 51 references
  1. [9]

    Remember the past: Distilling datasets into addressable memories for neural networks

    Zhiwei Deng and Olga Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. arXiv preprint arXiv:2206.02916, 2022

    2022 arXiv

  2. [10]

    To- wards lossless dataset distillation via difficulty-aligned trajectory matching

    Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. To- wards lossless dataset distillation via difficulty-aligned trajectory matching. In The Twelfth International Conference on Learning Representations, 2024

    2024

  3. [11]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  4. [12]

    Multisize dataset condensation

    Yang He, Lingao Xiao, Joey Tianyi Zhou, and Ivor Tsang. Multisize dataset condensation. ICLR, 2024

    2024

  5. [13]

    Densely connected convolutional networks

    Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017

    2017

  6. [14]

    Dataset condensation via efficient synthetic-data param- eterization

    Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data param- eterization. In International Conference on Machine Learning, pages 11102–11118. PMLR, 2022

    2022

  7. [15]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, ON, Canada, 2009

    2009

  8. [16]

    Dataset condensation with latent space knowledge factorization and sharing

    Hae Beom Lee, Dong Bok Lee, and Sung Ju Hwang. Dataset condensation with latent space knowledge factorization and sharing. arXiv preprint arXiv:2208.10494, 2022

    2022 arXiv

  9. [17]

    Dataset condensation with contrastive signals

    Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In International Conference on Machine Learning, pages 12352–12364. PMLR, 2022

    2022

  10. [18]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  11. [19]

    The evolution of dataset distillation: Toward scalable and generalizable solutions

    Ping Liu and Jiawei Du. The evolution of dataset distillation: Toward scalable and generalizable solutions. arXiv preprint arXiv:2502.05673, 2025

    2025 arXiv

  12. [20]

    Dataset distillation via factorization

    Songhua Liu, Kai Wang, Xingyi Yang, Jingwen Ye, and Xinchao Wang. Dataset distillation via factorization. Advances in Neural Information Processing Systems, 35:1100–1113, 2022

    2022

  13. [21]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021

    2021

  14. [22]

    Efficient dataset distillation using random feature approximation

    Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. arXiv preprint arXiv:2210.12067, 2022. 11

    2022 arXiv

  15. [23]

    Mixed precision training

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017

    arXiv

  16. [24]

    Dataset distillation with infinitely wide convolutional networks

    Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. Advances in Neural Information Processing Systems, 34:5186–5198, 2021

    2021

  17. [25]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. OpenAI Blog, 2018

    2018

  18. [26]

    Liu, Yuri A

    Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z. Liu, Yuri A. Lawryshyn, and Kon- stantinos N. Plataniotis. Datadam: Efficient dataset distillation with attention matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 17097–17107, ...

    2023

  19. [27]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018

    2018

  20. [28]

    Dataset distillation in the era of large-scale data: Methods, analysis, and future directions

    Xinyi Shang, Peng Sun, Zhiqiang Shen, Tao Lin, and Jing-Hao Xue. Dataset distillation in the era of large-scale data: Methods, analysis, and future directions. 2025

    2025

  21. [29]

    Generalized large-scale data condensation via various backbone and statistical matching

    Shitong Shao, Zeyuan Yin, Muxin Zhou, Xindong Zhang, and Zhiqiang Shen. Generalized large-scale data condensation via various backbone and statistical matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16709–16718, 2024

    2024

  22. [30]

    Elucidating the design space of dataset condensation

    Shitong Shao, Zikai Zhou, Huanran Chen, and Zhiqiang Shen. Elucidating the design space of dataset condensation. arXiv preprint arXiv:2404.13733, 2024

    2024 arXiv

  23. [31]

    Frequency domain-based dataset distilla- tion

    Donghyeok Shin, Seungjae Shin, and Il-Chul Moon. Frequency domain-based dataset distilla- tion. Advances in Neural Information Processing Systems, 36, 2024

    2024

  24. [32]

    On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm

    Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9390–9399, 2024

    2024

  25. [33]

    Inception-v4, inception-resnet and the impact of residual connections on learning

    Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017

    2017

  26. [34]

    Efficientnetv2: Smaller models and faster training

    Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. InInternational conference on machine learning, pages 10096–10106. PMLR, 2021

    2021

  27. [35]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  28. [36]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  29. [37]

    Cafe: Learning to condense dataset by aligning features

    Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  30. [38]

    Dataset distillation

    Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018

    2018 arXiv

  31. [39]

    Are large-scale soft labels necessary for large-scale dataset distilla- tion? arXiv preprint arXiv:2410.15919, 2024

    Lingao Xiao and Yang He. Are large-scale soft labels necessary for large-scale dataset distilla- tion? arXiv preprint arXiv:2410.15919, 2024. 12

    2024 arXiv

  32. [40]

    Towards adversarially robust dataset distillation by curvature regularization

    Eric Xue, Yijiang Li, Haoyang Liu, Yifan Shen, and Haohan Wang. Towards adversarially robust dataset distillation by curvature regularization. arXiv preprint arXiv:2403.10045, 2024

    2024 arXiv

  33. [41]

    Image classification using deep convolutional neural net- works

    Lian Yao, Yin Li, and Li Fei-Fei. Image classification using deep convolutional neural net- works. https://cs231n.stanford.edu/reports/2015/pdfs/yle_project. pdf, 2015. CS231n: Convolutional Neural Networks for Visual Recognition, Stanford Univer- sity, Course Project Report

    2015

  34. [42]

    Dataset distillation via curriculum data synthesis in large data era

    Zeyuan Yin and Zhiqiang Shen. Dataset distillation via curriculum data synthesis in large data era. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: //openreview.net/forum?id=PlaZD2nGCl

    2024

  35. [43]

    Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective

    Zeyuan Yin, Eric Xing, and Zhiqiang Shen. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. Advances in Neural Information Processing Systems, 36, 2024

    2024

  36. [44]

    Shufflenet: An extremely efficient convolutional neural network for mobile devices

    Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848–6856, 2018

    2018

  37. [45]

    Dataset condensation with differentiable siamese augmentation

    Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pages 12674–12685. PMLR, 2021

    2021

  38. [46]

    Dataset condensation with distribution matching

    Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2023, Waikoloa, HI, USA, January 2-7, 2023, 2023

    2023

  39. [47]

    Dataset condensation with gradient matching

    Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021

    2021

  40. [48]

    Improve cross-architecture generalization on dataset distillation

    Binglin Zhou, Linhao Zhong, and Wentao Chen. Improve cross-architecture generalization on dataset distillation. arXiv preprint arXiv:2402.13007, 2024

    2024 arXiv

  41. [49]

    Dataset distillation using neural feature regression

    Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. Advances in Neural Information Processing Systems, 35:9813–9827, 2022. 13 Appendix of FADRM Contents A Theoretical Derivation 15 B Optimization Details 18 C Resampling via Bili...

    2022

  42. [50]

    ( Bounded range) For every admissible pair (p, z), 0 ≤ ℓ p, q(T )(z) ≤ B, B := log k + 2C T . (16)

  43. [51]

    theℓ∞ norm with L = 1 T

    (ℓ∞-Lipschitz continuity in logits) The map z 7→ ℓ p, q(T )(z) is L-Lipschitz w.r.t. theℓ∞ norm with L = 1 T . Consequently, it is √ k/T -Lipschitz w.r.t. the Euclidean norm. proof of Theorem 3. (i) Boundedness. Write KL p ∥ q(T ) = kX i=1 pi log pi − kX i=1 pi log q(T ) i . (...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.