Pith. sign in

REVIEW 4 major objections 6 minor 92 references

UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read UniDFKD replaces architecture-specific batch-norm statistics with explicit, architecture-agnostic semantic priors and reports state-of-the-art data-free distillation across CNNs and ViTs, with average absolute gains over 20% over prior…

desk verdict UniDFKD is a genuinely useful step for DFKD with BN-free teachers; the CDSA linchpin concern is real but not fatal, and the missing error bars and code are the bigger practical disappointments. read the letter →

arxiv 2608.09287 v1 pith:2GK7CUXL submitted 2026-08-10 cs.CV cs.AI

Xuewan He , Tong Chu , Zihan Cheng , Yuchen Su , Qianxin Xia , Guoming Lu , Jielei Wang , Wen Li This is my paper · ORCID
classification cs.CVcs.AI
keywords data-freeknowledgedistillationsemanticpriorbatchnormalizationvisiontransformerspatialattributionsyntheticdatagenerationarchitecture-agnostic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Data-free knowledge distillation (DFKD) has relied on a teacher's batch-norm running statistics as the prior that shapes synthesized training images, so DFKD collapses when the teacher is a batch-norm-free architecture such as a Vision Transformer or Swin. The paper argues that the real information in the batch-norm prior lives in the deepest, class-discriminative layers, not in low-level statistics. It therefore replaces the statistical prior with three explicit semantic priors: language-conditioned categorical semantics for what to synthesize, a center-biased Gaussian anchoring of class-discriminative spatial attribution for where evidence sits, and spatial semantic distillation for how the student inherits that evidence. On CIFAR-10/100 and ImageNet-100 across ResNet, ConvNeXt, ViT, and Swin teachers and students, UniDFKD reports an average absolute improvement over 20% over the previous best method in both homogeneous and heterogeneous settings, approaching real-data student accuracy on CIFAR and recovering 71% top-1 on ImageNet-100 where prior baselines fall to 30-40%.

What carries the argument

The load-bearing object is the class-discriminative spatial attribution (CDSA) map, defined as $M_f(\hat{x}) = R_f(\hat{x}) W_{C,f} \in \mathbb{R}^{H_p \times W_p \times C}$, where $R_f$ is the deepest spatial response before global pooling and $W_{C,f}$ the classification-head weights; for CLS-token ViTs, $R_f$ is assembled from attention-weighted patch values. SSA anchors the batch-averaged CDSA to a center-biased Gaussian prior $\pi[i,j] = B + A \exp\big(-\tfrac12((i-\mu_x)^2/\sigma_x^2 + (j-\mu_y)^2/\sigma_y^2)\big)$, and SSD aligns teacher and student CDSA maps per sample. Categorical Semantic Conditioning uses spherical k-means filtering of LLM captions encoded by a frozen text encoder, injected through Conditional Batch Normalization $\gamma_j(c_y)\odot \mathrm{Norm}(\tilde{h}_j) + \beta_j(c_y)$ in every generator block. The CDSA interface is what makes the whole pipeline architecture-agnostic: the same $C$-dimensional category space receives spatial evidence from ResNet, ConvNeXt, Swin, and ViT alike.

What would settle it

Run UniDFKD on a teacher whose last-layer responses are known to be poorly localized (or on a non-object-centric dataset such as pathology or satellite imagery), keeping SSA/SSD active; the paper's own ablations show a corner-biased prior costs 2–4 accuracy points, so a setting where the center-biased prior is systematically wrong should show a measurable drop. Independently, benchmark CDSA maps against an external attribution-localization test: if the maps do not localize the target class on ViT, Swin, or ConvNeXt, the architecture-agnostic claim fails at its interface.

Watch

Extended reading notes

Core claim

UniDFKD establishes that the effectiveness of the batch-norm prior in DFKD comes almost entirely from its deep semantic representations: a layer-wise diagnosis on ResNet and ViT teachers shows that matching the deepest layer alone nearly matches full-network matching, while shallow-layer constraints add little. From this it concludes that DFKD should be reframed as the construction of explicit, architecture-agnostic semantic priors. The framework instantiates those priors along three axes: Categorical Semantic Conditioning (CSC) persistently injects sampled language-derived embeddings into every generator block so synthetic images keep inter-class relations and intra-class diversity; Spatial Semantic Anchoring (SSA) projects the teacher's deepest spatial responses through the classification head into a class-discriminative spatial attribution map and aligns the batch-averaged map with a center-biased Gaussian; Spatial Semantic Distillation (SSD) aligns per-sample attribution maps between teacher and student alongside logit matching. The result, per the paper, is that data-free distillation no longer depends on normalization type and transfers spatial evidence across heterogeneous architectures.

Load-bearing premise

The spatial attribution map is assumed to be a faithful, class-localized signal for every teacher architecture; if the map responds to background or texture instead of the target object, the Gaussian anchor and the spatial distillation would both enforce a faulty prior, and the reported gains would not transfer.

Editorial extensions

If this is right

  • Data-free distillation extends to batch-norm-free teachers: ViT, Swin, and ConvNeXt teachers can be distilled without real data, something BN-statistics methods cannot do.
  • In both homogeneous and heterogeneous teacher–student pairs, reported accuracy approaches students trained on real data for CIFAR-10/100 and exceeds 71% on ImageNet-100, versus 30–40% for previous approaches.
  • Because the semantic priors are orthogonal to BN statistics, they can be combined with BN matching when available, preserving state-of-the-art results on classic CNN settings rather than sacrificing them.
  • The layer-wise diagnosis implies that future DFKD synthesis can concentrate on deep semantic constraints, potentially reducing the cost of synthesis.
  • Spatial semantic distillation transfers where the teacher looks, not just what it predicts, so students may inherit localization behavior along with accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The what/where/how decomposition is not tied to classification: the same CDSA interface and semantic priors could be tested for data-free quantization calibration or for detecting distribution shift, where BN statistics are also unavailable.
  • The paper's own limitation note suggests the center-biased Gaussian will need replacement in non-object-centric domains; a natural extension is to estimate the prior per task from a small set of teacher attributions instead of fixing it analytically.
  • The reported margins are largest precisely where BN statistics vanish (LN-based and heterogeneous settings); on saturated BN-CNN benchmarks the improvements are marginal, so the practical value is concentrated in modern low-data/high-resolution regimes.
  • CDSA as a shared attribution space could serve as a currency for cross-architecture transfer beyond DFKD, e.g., comparing where different backbones look at the same image and distilling those alignment maps directly.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes UniDFKD, a data-free knowledge distillation framework that replaces Batch Normalization (BN) statistic priors with explicit, architecture-agnostic semantic priors. The method is organized around three components: Categorical Semantic Conditioning (CSC), which injects LLM-derived text embeddings into every generator block to preserve class identity and intra-class diversity; Spatial Semantic Anchoring (SSA), which aligns a teacher's class-discriminative spatial attribution (CDSA) maps with a center-biased Gaussian prior; and Spatial Semantic Distillation (SSD), which aligns teacher and student CDSA maps during distillation. Experiments cover four architecture families (ResNet, ConvNeXt, ViT, Swin) on CIFAR-10, CIFAR-100, and ImageNet-100 in 24 homogeneous and heterogeneous teacher–student settings, reporting large gains over prior DFKD methods, especially for LayerNorm-based teachers. The paper also presents a layer-wise diagnosis of BN statistics, extensive ablations, synthetic distribution quality metrics, and supplementary experiments on ImageNet-1K under limited synthesis budgets.

Significance. If the empirical results hold, the paper addresses a real and timely limitation of current DFKD methods: their dependence on BN statistics, which are absent in modern architectures. The what/where/how decomposition of the semantic prior is conceptually clear, and the breadth of the experimental sweep across four architecture families is an asset. The paper includes useful controls, including a corner-biased Gaussian ablation (Table S5), a vision-agnostic text-encoder robustness check (Table S4), and an explicit limitations discussion. The headline claim of over 20% average absolute improvement is driven primarily by settings in which baselines collapse, which is exactly the intended use case. However, the central CDSA interface and the fairness of the baseline comparisons need additional support before the architecture-agnostic claim can be fully credited.

major comments (4)
  1. [Main Results / Table 1] Table 1 and the accompanying text report averages over three seeds but no standard deviations. In several configurations the gap between UniDFKD and the strongest baseline is small or reversed (e.g., CIFAR-100 homogeneous RN34→RN18: UniDFKD 77.37% vs. NAYER 77.54%). Without error bars, the claim of state-of-the-art performance 'across almost all 24 settings' is not statistically substantiated. Please report mean ± std at least for all configurations in Table 1 and for the reported averages.
  2. [Spatial Semantic Anchoring / Eq. (6) and Supplementary Eqs. (S1)–(S3)] CDSA is the load-bearing interface for both SSA and SSD, but its validity is supported only by in-paper visualizations and internal ablations. For ViT, Eq. (S3) assembles the spatial response from attention-weighted patch values merged through the output projection, which omits the final LayerNorm, residual connection, and MLP contributions to the class logit; the resulting map is therefore an approximation of a partial logit decomposition, not an exact one. The authors should validate CDSA quantitatively on real data (e.g., pointing-game accuracy against ground-truth masks, or perturbation-based faithfulness) and demonstrate that the maps are meaningfully class-discriminative and comparable across ResNet, ConvNeXt, Swin, and ViT, since both SSA and SSD enforce this map as a prior.
  3. [Spatial Semantic Anchoring / Eq. (8)] Eq. (8) averages a cosine loss over all C classes, but the manuscript does not specify the batch class composition. If a batch contains only a subset of classes, the aggregated CDSA maps for absent classes are likely near-zero or noise-dominated, making the cosine term ill-defined or uninformative. If all classes are present, the objective still weights non-target classes equally with target classes. The manuscript should clarify how batches are assembled (single-class vs. mixed), describe how Eq. (8) treats classes not represented in the batch, and provide an analysis or a weighting/sampling scheme for the low-magnitude regime.
  4. [Experiments] The baselines are described as 'optimized for BN-free architectures' and 'comprehensively swept', but the supplementary material reports only UniDFKD's own hyperparameters. To support the 20%+ margin claim, the authors must report the search ranges and final hyperparameters for each baseline (learning rate, optimizer, augmentation, synthesis iterations) in each setting. Without this information, the observed improvement could be partially attributable to under-tuned baselines, and the comparison is not fully reproducible.
minor comments (6)
  1. [Abstract and Introduction] The phrase 'catastrophic performance degradation' is overly strong for settings such as ViT-S→ViT-T on CIFAR-10, where NAYER reaches 85.86%; consider using 'substantial degradation' instead.
  2. [Eq. (8)] The notation M̄_T^c is used without defining the per-class slice of the aggregated tensor M̄; please add an explicit definition.
  3. [Spatial Semantic Anchoring / Eq. (7)] The Gaussian prior parameters ρ, β_min, β_max, A, and B are fixed across all architectures, but no sensitivity analysis for these values is reported; a supplementary figure analogous to Fig. 6 would strengthen the claim that the prior is broadly stable.
  4. [Fig. 3] For ViT, the layer-wise diagnosis uses real-data moments as the statistical target; the caption should explicitly state that this is an oracle setting and not part of the proposed data-free procedure.
  5. [Supplementary, Table S3] The 'Data Ratio' of 1% and 5% is described as the number of synthesized images relative to the full ImageNet-1K scale; please clarify whether this refers to image count or storage size and specify the absolute number of synthesized images.
  6. [Extended Implementation Details] The text mentions DeepSeekV4 and Qwen3 as LLM prompt sources but does not report the sampling temperature or decoding parameters; adding these details would improve reproducibility of the semantic bank.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the spatial and categorical priors are explicit losses with fixed hyperparameters, and the only self-citation is a non-load-bearing related-work pointer.

full rationale

The paper's derivation chain is self-contained and does not reduce any reported prediction to its own inputs by construction. The CDSA interface (Eq. 6) is a linear projection of the deepest spatial responses through the classification-head weights, and for ViT (Eqs. S2-S3) it is a per-patch decomposition of the class evidence; SSA (Eq. 8) and SSD (Eq. 9) use these maps as explicit loss terms against an analytic Gaussian prior and teacher-student alignment, respectively. These losses are not fitted to the reported accuracies: the Gaussian prior parameters are fixed a priori to encode center bias, the loss weights lambda_SSA=5 and lambda_SSD=3 are set on validation and held constant across all evaluated settings, and the semantic-bank hyperparameters (K=7, r=15) are ablated on a validation basis. The layer-wise diagnosis motivating deep semantic priors is an empirical observation, not an input definition. The only self-citation (He et al. 2025, PRISM) appears in the discussion of diffusion-based synthesis and is not load-bearing; no uniqueness theorem or prior result by the same authors is invoked to forbid alternatives. The skeptic's concern that CDSA maps may not be class-discriminative for all architectures is an external robustness and falsifiability risk, not a circular equivalence: nothing in Eqs. 6-9 reduces a reported accuracy to a fitted parameter or to the paper's own assumptions by construction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central claim rests on empirical design choices: text embeddings as categorical priors, a hand-specified Gaussian spatial prior, and a new attribution interface (CDSA). Several hyperparameters (K, r, lambda_SSA, lambda_SSD, prior geometry) are tuned on validation sets and fixed across experiments. There are no formal proofs; the evidence is experimental.

free parameters (6)
  • K = 7
    Number of spherical k-means clusters for the semantic bank, tuned on CIFAR-100 (Fig. 6a) and fixed across all settings.
  • r = 15
    Top-r embeddings retained per cluster, tuned on CIFAR-100 (Fig. 6a) and fixed across all settings.
  • lambda_SSA = 5
    Weight of the Spatial Semantic Anchoring loss, tuned on CIFAR-100 (Fig. 6b) and fixed across all settings.
  • lambda_SSD = 3
    Weight of the Spatial Semantic Distillation loss, tuned on CIFAR-100 (Fig. 6b) and fixed across all settings.
  • Gaussian prior parameters (rho, beta_min, beta_max, A, B) = rho=1/7, beta_min=1/7, beta_max=3/14, A=1, B~U[0,0.3]
    Hand-set center jitter, width ranges, peak amplitude, and background offset for the SSA prior; the authors report robustness but these are chosen constants.
  • Synthesis iterations per batch = 40/60/120 (CIFAR-10/100/ImageNet-100)
    Number of generator updates per batch, set per dataset; affects synthesis quality and the reported runtimes.
assumptions (5)
  • domain assumption Frozen text embeddings from CLIP or Qwen3 encode inter-class and intra-class semantic geometry that is useful for steering image synthesis.
    CSC relies on this to build the semantic bank; Section 'Categorical Semantic Conditioning' and Table S4 test this empirically.
  • domain assumption Class-discriminative evidence in natural images is center-biased and can be approximated by a jittered Gaussian prior.
    SSA aligns CDSA maps to Eq. 7; the authors cite Schauerte and Stiefelhagen 2015 and ablate the prior in Table S5, but the prior is a modeling assumption.
  • ad hoc to paper The teacher's deep spatial responses projected through the classification head (Eq. 6) yield valid class-discriminative attribution maps for all considered architectures.
    CDSA is introduced by this paper; its validity is supported by visualizations and ablations in Fig. 5 and Table 2, but it is not an established external benchmark.
  • domain assumption BN running statistics in a pretrained teacher summarize real-data statistics and their deep-layer alignment is what drives DFKD success.
    The layer-wise analysis in Fig. 3 underpins the paradigm shift; the ViT portion uses real data to compute reference moments, so this conclusion is partly empirical.
  • domain assumption Access to the teacher in eval mode with intermediate representations is permitted in DFKD.
    Standard DFKD assumption used throughout the method; stated in the 'Preliminary Analysis' section.
invented entities (1)
  • Class-Discriminative Spatial Attribution (CDSA) map
    purpose: A unified architecture-agnostic interface to extract class-discriminative spatial evidence from any teacher, used by both SSA and SSD.
    CDSA is a new construct defined in Eq. 6 with per-architecture recipes in the supplementary. The paper validates it through ablations and visualizations, but there is no external benchmark confirming it is the correct attribution for all architectures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation." pith.science (2026). https://pith.science/paper/2GK7CUXL

@misc{pith2026260809287,
  author       = {Pith},
  title        = {Pith review of: UniDFKD: A Unified Semantic Prior Framework for Architecture-Agnostic Data-Free Knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2GK7CUXL}},
  note         = {Machine review of arXiv:2608.09287}
}
read the original abstract

Data-Free Knowledge Distillation (DFKD) transfers knowledge from a pretrained teacher model to a compact student model by synthesizing semantically informative data, eliminating the need for access to the original training dataset. Existing DFKD methods rely heavily on architecture-specific statistical priors (e.g., Batch Normalization statistics) to guide data synthesis, however, such architecture-dependent priors are often absent in modern architectures such as Vision Transformers (ViTs), resulting in degraded semantic quality of the synthesized data and consequently catastrophic performance degradation. In this paper, we propose \emph{UniDFKD}, a unified data-free knowledge distillation framework that replaces architecture-specific statistics with explicit, architecture-agnostic semantic priors. \emph{UniDFKD} governs the entire synthesis-distillation pipeline along three dimensions: (1) Categorical Semantic Conditioning (CSC) defines \emph{what} to synthesize by persistently modulating the generator with language-derived embeddings to capture semantic diversity; (2) Spatial Semantic Anchoring (SSA) dictates \emph{where} evidence belongs by anchoring the teacher's spatial attributions to a Gaussian prior; and (3) Spatial Semantic Distillation (SSD) controls \emph{how} knowledge is transferred by explicitly aligning teacher-student spatial evidence alongside predictions. Extensive experiments across CNNs and ViTs demonstrate that UniDFKD establishes a new state-of-the-art, outperforming existing methods by an average absolute margin of over 20\% in both homogeneous and heterogeneous settings.

Figures

Figures reproduced from arXiv: 2608.09287 by the authors.

Figure 1
Figure 1. Comparison of student accuracy (%) achieved [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of DFKD paradigms. (a) Existing methods adopt BN statistics as an implicit semantic prior, confining [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Layer-wise feature-statistic matching on CIFAR [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Overview of UniDFKD. Synthesis (left). CSC builds a semantic bank Ey from LLM-generated class descriptions and modulates every generator block with a sampled embedding cy via CBN, while SSA aligns the teacher’s batch-averaged attribution M¯ T with a Gaussian prior; ite…
Figure 5
Figure 5. Figure 5: Visual comparison of our class-discriminative spa [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Hyperparameter analysis on CIFAR-100 (CNX [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

92 extracted references · 53 canonical work pages

  1. [1]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , year =

    Data-Free Network Quantization With Adversarial Knowledge Distillation , author =. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , year =

  2. [2]

    Advances in Neural Information Processing Systems , volume =

    Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation , author =. Advances in Neural Information Processing Systems , volume =

  3. [3]

    arXiv preprint arXiv:2606.19348 , year =

    DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , author =. arXiv preprint arXiv:2606.19348 , year =

    arXiv

  4. [4]

    arXiv preprint arXiv:2505.09388 , year =

    Qwen3 Technical Report , author =. arXiv preprint arXiv:2505.09388 , year =

    arXiv

  5. [5]

    European Conference on Computer Vision , pages =

    Yonglong Tian and Dilip Krishnan and Phillip Isola , title =. European Conference on Computer Vision , pages =

  6. [6]

    Friedman, Dan and Dieng, Adji Bousso , journal =. The

  7. [7]

    Heusel, Martin and Ramsauer, Hubert and Unterthiner, Thomas and Nessler, Bernhard and Hochreiter, Sepp , booktitle =

  8. [8]

    Transactions on Machine Learning Research , year =

    A Comprehensive Survey on Knowledge Distillation , author =. Transactions on Machine Learning Research , year =

Show all 92 references
  1. [9]

    arXiv preprint arXiv:1503.02531 , year=

    Distilling the knowledge in a neural network , author=. arXiv preprint arXiv:1503.02531 , year=

    arXiv

  2. [10]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Understanding deep image representations by inverting them , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

  3. [11]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Deep neural networks are easily fooled: High confidence predictions for unrecognizable images , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

  4. [12]

    International Conference on Machine Learning , pages=

    Batch normalization: Accelerating deep network training by reducing internal covariate shift , author=. International Conference on Machine Learning , pages=. 2015 , organization=

    2015

  5. [13]

    arXiv preprint arXiv:1607.06450 , year=

    Layer normalization , author=. arXiv preprint arXiv:1607.06450 , year=

    arXiv

  6. [14]

    Machine learning , volume=

    A theory of learning from different domains , author=. Machine learning , volume=. 2010 , publisher=

    2010

  7. [15]

    Deep compression: Compressing deep neural networks with pruning, trained quantization and

    Han, Song and Mao, Huizi and Dally, William J , journal=. Deep compression: Compressing deep neural networks with pruning, trained quantization and

  8. [16]

    arXiv preprint arXiv:1710.07535 , year=

    Data-free knowledge distillation for deep neural networks , author=. arXiv preprint arXiv:1710.07535 , year=

    arXiv

  9. [17]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Few sample knowledge distillation for efficient network compression , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  10. [18]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Progressive network grafting for few-shot knowledge distillation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  11. [19]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Data-free learning of student networks , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  12. [20]

    International Conference on Learning Representations , volume=

    Dataset condensation with gradient matching , author=. International Conference on Learning Representations , volume=

  13. [21]

    Romero, Adriana and Ballas, Nicolas and Kahou, Samira Ebrahimi and Chassang, Antoine and Gatta, Carlo and Bengio, Yoshua , journal=

  14. [22]

    arXiv preprint arXiv:1612.03928 , year=

    Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer , author=. arXiv preprint arXiv:1612.03928 , year=

    arXiv

  15. [23]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Few shot network compression via cross distillation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  16. [24]

    International Conference on Machine Learning , pages=

    Zero-shot knowledge distillation in deep networks , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  17. [25]

    Advances in Neural Information Processing Systems , volume=

    Zero-shot knowledge transfer via adversarial belief matching , author=. Advances in Neural Information Processing Systems , volume=

  18. [26]

    Dreaming to distill: Data-free knowledge transfer via

    Yin, Hongxu and Molchanov, Pavlo and Alvarez, Jose M and Li, Zhizhong and Mallya, Arun and Hoiem, Derek and Jha, Niraj K and Kautz, Jan , booktitle=. Dreaming to distill: Data-free knowledge transfer via

  19. [27]

    Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence , pages=

    Contrastive model inversion for data-free knowledge distillation , author=. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence , pages=. 2021 , doi=

    2021

  20. [28]

    Proceedings of the 41st International Conference on Machine Learning , pages=

    Sparse model inversion: Efficient inversion of vision transformers for data-free applications , author=. Proceedings of the 41st International Conference on Machine Learning , pages=. 2024 , volume=

    2024

  21. [29]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Up to 100x faster data-free knowledge distillation , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  22. [30]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Robust and resource-efficient data-free knowledge distillation by generative pseudo replay , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  23. [31]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Coupling the generator with teacher for effective data-free knowledge distillation , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

  24. [32]

    Proceedings of the 42nd International Conference on Machine Learning , pages=

    When data-free knowledge distillation meets non-transferable teacher: Escaping out-of-distribution trap is all you need , author=. Proceedings of the 42nd International Conference on Machine Learning , pages=. 2025 , volume=

    2025

  25. [33]

    arXiv preprint arXiv:1411.1784 , year=

    Conditional generative adversarial nets , author=. arXiv preprint arXiv:1411.1784 , year=

    arXiv

  26. [34]

    Technical report, University of Toronto , year=

    Learning multiple layers of features from tiny images , author=. Technical report, University of Toronto , year=

  27. [35]

    2009 , organization=

    Deng, Jia and Dong, Wei and Socher, Richard and Li, Li-Jia and Li, Kai and Fei-Fei, Li , booktitle=. 2009 , organization=

    2009

  28. [36]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Adversarial feature hallucination networks for few-shot learning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  29. [37]

    Visualizing data using

    van der Maaten, Laurens and Hinton, Geoffrey , journal=. Visualizing data using

  30. [38]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Deep residual learning for image recognition , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

  31. [39]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Image style transfer using convolutional neural networks , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

  32. [40]

    Howard, Andrew G and Zhu, Menglong and Chen, Bo and Kalenichenko, Dmitry and Wang, Weijun and Weyand, Tobias and Andreetto, Marco and Adam, Hartwig , journal=

  33. [41]

    arXiv preprint arXiv:1409.1556 , year=

    Very deep convolutional networks for large-scale image recognition , author=. arXiv preprint arXiv:1409.1556 , year=

    arXiv

  34. [42]

    Differentiable augmentation for data-efficient

    Zhao, Shengyu and Liu, Zhijian and Lin, Ji and Zhu, Jun-Yan and Han, Song , journal=. Differentiable augmentation for data-efficient

  35. [43]

    arXiv preprint arXiv:1605.07146 , year=

    Wide residual networks , author=. arXiv preprint arXiv:1605.07146 , year=

    arXiv

  36. [44]

    International Conference on Machine Learning , pages=

    Dataset condensation with differentiable siamese augmentation , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  37. [45]

    Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

    Dataset condensation with distribution matching , author=. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

  38. [46]

    Proceedings of the 39th International Conference on Machine Learning , pages =

    Dataset condensation via efficient synthetic-data parameterization , author =. Proceedings of the 39th International Conference on Machine Learning , pages =. 2022 , editor =

    2022

  39. [47]

    Proceedings of the International Conference on Machine Learning , pages=

    Privacy for free: How does dataset condensation help privacy? , author=. Proceedings of the International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  40. [48]

    Advances in Neural Information Processing Systems , year=

    Dataset distillation using neural feature regression , author=. Advances in Neural Information Processing Systems , year=

  41. [49]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Dataset distillation by matching training trajectories , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  42. [50]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Data-free knowledge distillation with soft targeted transfer set synthesis , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  43. [51]

    The Twelfth International Conference on Learning Representations , year=

    Embarrassingly simple dataset distillation , author=. The Twelfth International Conference on Learning Representations , year=

  44. [52]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Small-scale data-free knowledge distillation , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  45. [53]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Data-free knowledge distillation via feature exchange and activation region constraint , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  46. [54]

    The Twelfth International Conference on Learning Representations , year=

    Multisize dataset condensation , author=. The Twelfth International Conference on Learning Representations , year=

  47. [55]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , year=

    Dataset distillation: A comprehensive review , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , year=

  48. [56]

    arXiv preprint arXiv:2112.15278 , year=

    Data-free knowledge transfer: A survey , author=. arXiv preprint arXiv:2112.15278 , year=

    arXiv

  49. [57]

    Advances in Neural Information Processing Systems , volume=

    Bias and generalization in deep generative models: An empirical study , author=. Advances in Neural Information Processing Systems , volume=

  50. [58]

    Proceedings of the Royal Society A , volume=

    Inductive biases for deep learning of higher-level cognition , author=. Proceedings of the Royal Society A , volume=. 2022 , publisher=

    2022

  51. [59]

    ACM Computing Surveys , volume=

    A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities , author=. ACM Computing Surveys , volume=. 2023 , publisher=

    2023

  52. [60]

    Proceedings of the 4th Workshop on NLP for Conversational AI , pages=

    Knowledge distillation meets few-shot learning: An approach for few-shot intent classification within and across domains , author=. Proceedings of the 4th Workshop on NLP for Conversational AI , pages=

  53. [61]

    Mind the gap between synthetic and real: Utilizing transfer learning to probe the boundaries of

    Hennicke, Leonhard and Adriano, Christian Medeiros and Giese, Holger and Koehler, Jan Mathias and Schott, Lukas , journal=. Mind the gap between synthetic and real: Utilizing transfer learning to probe the boundaries of

  54. [62]

    Journal on Autonomous Transportation Systems , volume=

    Bridging the domain gap between synthetic and real-world data for autonomous driving , author=. Journal on Autonomous Transportation Systems , volume=. 2024 , publisher=

    2024

  55. [63]

    International Conference on Learning Representations , year=

    An image is worth 16x16 words: Transformers for image recognition at scale , author=. International Conference on Learning Representations , year=

  56. [64]

    Liu, Zhuang and Mao, Hanzi and Wu, Chao-Yuan and Feichtenhofer, Christoph and Darrell, Trevor and Xie, Saining , booktitle=. A

  57. [65]

    Liu, Ze and Lin, Yutong and Cao, Yue and Hu, Han and Wei, Yixuan and Zhang, Zheng and Lin, Stephen and Guo, Baining , booktitle=

  58. [66]

    2023 , month=

    Woo, Sanghyun and Debnath, Shoubhik and Hu, Ronghang and Chen, Xinlei and Liu, Zhuang and Kweon, In So and Xie, Saining , booktitle=. 2023 , month=

    2023

  59. [67]

    Oquab, Maxime and Darcet, Timothée and Moutakanni, Théo and Vo, Huy V. and Szafraniec, Marc and Khalidov, Vasil and Fernandez, Pierre and Haziza, Daniel and Massa, Francisco and El-Nouby, Alaaeldin and Assran, Mido and Ballas, Nicolas and Galuba, Wojciech and Howes, Russell an...

    2024

  60. [68]

    Liu, Xinyu and Peng, Houwen and Zheng, Ningxin and Yang, Yuqing and Hu, Han and Yuan, Yixuan , booktitle=

  61. [69]

    arXiv preprint arXiv:2405.00314 , year=

    Model quantization and hardware acceleration for vision transformers: A comprehensive survey , author=. arXiv preprint arXiv:2405.00314 , year=

    arXiv

  62. [70]

    Tran, Minh-Tuan and Le, Trung and Le, Xuan-May and Harandi, Mehrtash and Tran, Quan Hung and Phung, Dinh , booktitle=

  63. [71]

    Large-scale data-free knowledge distillation for

    Tran, Minh-Tuan and Le, Trung and Le, Xuan-May and Cai, Jianfei and Harandi, Mehrtash and Phung, Dinh , journal=. Large-scale data-free knowledge distillation for

  64. [72]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Semantic alignment and reinforcement for data-free quantization of vision transformers , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=. 2025 , month=

    2025

  65. [73]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    A style-based generator architecture for generative adversarial networks , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  66. [74]

    Proceedings of the 38th International Conference on Machine Learning , pages=

    Learning transferable visual models from natural language supervision , author=. Proceedings of the 38th International Conference on Machine Learning , pages=. 2021 , volume=

    2021

  67. [75]

    Advances in Neural Information Processing Systems , volume=

    Modulating early visual processing by language , author=. Advances in Neural Information Processing Systems , volume=

  68. [76]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    What does a platypus look like? Generating customized prompts for zero-shot image classification , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=. 2023 , month=

    2023

  69. [77]

    Machine Learning , volume=

    Concept decompositions for large sparse text data using clustering , author=. Machine Learning , volume=. 2001 , publisher=

    2001

  70. [78]

    Normalization techniques in training

    Huang, Lei and Qin, Jie and Zhou, Yi and Zhu, Fan and Liu, Li and Shao, Ling , journal=. Normalization techniques in training. 2023 , doi=

    2023

  71. [79]

    Proceedings of the IEEE International Conference on Computer Vision , pages=

    Learning to predict where humans look , author=. Proceedings of the IEEE International Conference on Computer Vision , pages=

  72. [80]

    European Conference on Computer Vision , pages=

    Patch similarity aware data-free quantization for vision transformers , author=. European Conference on Computer Vision , pages=

  73. [81]

    Ramachandran, Akshat and Kundu, Souvik and Krishna, Tushar , booktitle=

  74. [82]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Reproducible scaling laws for contrastive language-image learning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  75. [83]

    and Cogswell, Michael and Das, Abhishek and Vedantam, Ramakrishna and Parikh, Devi and Batra, Dhruv , booktitle=

    Selvaraju, Ramprasaath R. and Cogswell, Michael and Das, Abhishek and Vedantam, Ramakrishna and Parikh, Devi and Batra, Dhruv , booktitle=

  76. [84]

    PLOS ONE , volume=

    On the distribution of salient objects in web images and its influence on salient object detection , author=. PLOS ONE , volume=. 2015 , doi=

    2015

  77. [85]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Learning deep features for discriminative localization , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

  78. [86]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Transformer interpretability beyond attention visualization , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  79. [87]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Squeeze-and-excitation networks , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=. 2018 , doi=

    2018

  80. [88]

    International Conference on Machine Learning , pages=

    Reliable fidelity and diversity metrics for generative models , author=. International Conference on Machine Learning , pages=. 2020 , organization=

    2020

  81. [89]

    Li, Yuhang and Kim, Youngeun and Lee, Donghyun and Kundu, Souvik and Panda, Priyadarshini , booktitle=

  82. [90]

    Proceedings of the 32nd ACM International Conference on Multimedia , pages=

    Towards effective data-free knowledge distillation via diverse diffusion augmentation , author=. Proceedings of the 32nd ACM International Conference on Multimedia , pages=. 2024 , doi=

    2024

  83. [91]

    IEEE International Conference on Multimedia and Expo , pages=

    Data-free knowledge distillation with diffusion models , author=. IEEE International Conference on Multimedia and Expo , pages=

  84. [92]

    He, Xuewan and Wang, Jielei and Cheng, Zihan and Su, Yuchen and Huang, Shiyue and Lu, Guoming , journal=

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.