Pith. sign in

REVIEW 5 major objections 5 minor 37 references

DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A smaller HuBERT trained on the teacher's own k-means cluster targets with the original masked-prediction objective outperforms all feature-distillation baselines on SUPERB.

desk verdict Simple, sensible distillation recipe with strong results, but the headline improvement numbers are internally inconsistent and overstated; fix before publish. read the letter →

arxiv 2507.02911 v1 pith:IJN6AP72 submitted 2025-06-25 cs.LG cs.AIcs.SDeess.AS

Hyung Gun Chi , Zakaria Aldeneh , Tatiana Likhomanenko , Oggi Rudovic , Takuya Higuchi , Li-Wei Chen , Shinji Watanabe , Ahmed Hussen Abdelaziz This is my paper
classification cs.LGcs.AIcs.SDeess.AS
keywords knowledgedistillationself-supervisedlearningspeechrepresentationHuBERTmaskedpredictionk-meansclusteringSUPERBbenchmarkmodelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Knowledge distillation for speech foundation models has mostly meant aligning teacher and student hidden features layer by layer. This paper claims that a simpler recipe works better: take the teacher's k-means cluster assignments from a single hidden layer and train the student with HuBERT's original masked token prediction objective on those targets. The authors show that a 26M-parameter student trained this way outperforms several published feature-distillation methods on four SUPERB tasks — phoneme recognition, ASR, speaker verification, and speaker identification — with over 21% relative improvement in phoneme error rate and more than 14% in word error rate. Because the recipe only needs target labels, the student architecture is unrestricted, making the approach easier to adopt. If the paper is right, self-supervised objectives are a stronger and more natural distillation signal for HuBERT than feature matching.

What carries the argument

The load-bearing machinery is HuBERT's masked token prediction objective, $\mathcal{L}_{SSL} = -\sum_{t \in M} \log p(z_t \mid \tilde{X}, t)$, where $M$ is the set of masked timestamps, $\tilde{X}$ is the masked input, and $z_t$ is a target class at timestamp $t$. The targets are generated by $k$-means clustering of a selected teacher layer's features, in hard form (argmin one-hot) or soft form (softmax over $L_2$ distances to all centroids with temperature $\tau$). This is exactly the objective HuBERT uses in its iterative self-distillation, so DiceHuBERT reduces distillation to choosing a smaller student architecture between iteration $N$ and $N+1$. Two student families are studied: HuBERT-shallow (fewer layers) and HuBERT-narrow (smaller feature and feed-forward dimensions); the narrow student at $D_S = D_{base}/2$ is the default.

What would settle it

Compare DiceHuBERT against a feature-distillation baseline on a SUPERB task outside the four evaluated, such as emotion recognition or intent classification, using the same 26M-parameter student; if single-layer cluster targets drop the information that task needs, the advantage should shrink or reverse. A more direct check within the paper's own setup is to generate targets by averaging features across multiple teacher layers (as suggested by the paper's cited future work [30]) and measure whether ASV and SID improve relative to the single-layer targets used here.

Watch

Extended reading notes

Core claim

In the paper's own framing, DiceHuBERT distills HuBERT by replacing the teacher with a smaller student inside HuBERT's iterative self-distillation loop, training the student with the same SSL loss — cross-entropy over masked timestamps predicting cluster targets — while keeping the architecture change as the only difference. With hard one-hot labels from a single teacher layer (layer 6 for iteration 1, layer 9 for iteration 2 of HuBERT-base, layer 18 for HuBERT-large) and a student that halves the feature dimension while keeping all 12 layers, DiceHuBERT at 26M parameters beats all prior HuBERT distillation methods on SUPERB PR, ASR, ASV, and SID. Ablations show that the SSL loss alone outperforms feature distillation alone at a fixed architecture, that keeping depth matters more than width, and that soft labels only help phoneme recognition at moderate temperature while hurting other tasks. The paper also reports a lower bound absent from prior work: a from-scratch HuBERT-narrow trained with two iterations already surpasses several feature-distillation methods.

Load-bearing premise

The method assumes that one teacher layer's k-means cluster assignments contain all of the knowledge worth transferring; if those labels discard speaker, prosodic, or other information, a student trained only on them could fall behind feature distillation on tasks that need that information.

Editorial extensions

If this is right

  • A student trained with the teacher's own SSL objective on cluster targets outperforms layer-wise feature-distillation baselines at the same parameter budget on SUPERB content and speaker tasks.
  • The student architecture is unrestricted, so any model that can take the same input and predict cluster targets — including convolutional-only designs — can be distilled without extra alignment modules.
  • Keeping the number of layers matters more than keeping feature dimension: HuBERT-narrow $D_{base}/2$ beats HuBERT-shallow $L_{base}/4$ on most tasks despite comparable compression.
  • A stronger teacher improves the student, but the gain is uneven: HuBERT-large as teacher improves ASR while not improving speaker tasks, pointing to a single-layer target bottleneck.
  • Adding feature distillation on top of the SSL objective helps only marginally on speaker identification and hurts other tasks, so the SSL-only recipe is preferable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This suggests that the knowledge worth transferring from HuBERT lives in the cluster-assignment structure of its hidden units rather than in raw feature geometry, which would explain why cross-entropy on cluster targets beats MSE feature matching.
  • Because soft labels at temperature 5 improved phoneme recognition while hurting speaker tasks, a task-family-dependent temperature or label-type schedule might combine the benefits — an extension the paper does not test.
  • The reported from-scratch lower bound implies that some previously published distillation gains may come from longer training or better hyperparameters rather than the distillation signal itself, so future work should always report this baseline.
  • The single-layer target limitation points to a concrete next step: aggregating features across teacher layers (for example, averaging representations as in the paper's cited future work [30]) could recover the speaker and prosodic information lost by choosing one layer, potentially closing the SID/ASV gap for larger teachers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes DiceHuBERT, a knowledge-distillation framework for compressing HuBERT in which a smaller student model is trained with the same masked-prediction cross-entropy objective as the teacher, using k-means cluster assignments (hard labels) or distance-based soft labels computed from a selected teacher layer as targets. The authors argue this 'SSL-based distillation' is simpler and more architecture-agnostic than existing layer-wise feature-matching methods, and they report SUPERB results on phoneme recognition (PR), ASR, speaker identification (SID), and speaker verification (ASV) claiming consistent improvements over prior HuBERT distillation methods. The paper also includes ablations over student width/depth, SSL vs. feature-distillation losses, hard vs. soft labels, and different teachers, plus a from-scratch lower-bound baseline that prior work omitted.

Significance. The central idea is attractive: reusing HuBERT's own target-generation mechanism for distillation avoids the architectural couplings and extra projection modules of feature-distillation methods, and the paper gives a clean demonstration that this recipe can beat published feature-distillation baselines on several SUPERB tasks. The inclusion of a from-scratch student lower bound is a genuine contribution, because it isolates the value of distillation per se, and the ablations (loss combination, soft-label temperature, teacher quality) are useful for practitioners. However, the paper's headline quantitative claims are currently unreliable: the abstract, Section 4.3, and the Conclusion state inconsistent relative-improvement figures, and the abstract's 'over 21%' PR improvement is not supported once the actual best prior PR baseline is used. Because the magnitude of improvement is the main advertised contribution, this issue is load-bearing and must be fixed before the claimed advantage can be accepted.

major comments (5)
  1. [Abstract, §4.3, §5] The relative-improvement figures are mutually inconsistent and at least one is inflated. The abstract says 'over 21%' improvement in PR and 'more than 14%' in ASR; Section 4.3 says 'a 21% relative improvement in ASR and a 14% relative improvement in PR'; the Conclusion repeats 'over 21% improvement in ASR and 14% in PR.' From Table 1, the best prior PR result among distillation methods is MaskHuBERT at 7.30 PER, so DiceHuBERT's 6.23 is a 14.7% relative improvement, not 'over 21%.' The 21% figure is only obtained by comparing PR to StarHuBERT-L (7.97), which is not the per-task state of the art. The ASR figure of 14.3% against StarHuBERT-L (8.91 vs. 7.64) is approximately correct, but the text swaps the task labels. Please state the exact baseline used for each percentage and make the abstract and body consistent; as written, the paper's central quantitative claim is not supported.
  2. [§4.2, Table 1] The evaluation is limited to four SUPERB tasks that were explicitly selected because they show 'significant performance gaps between the teacher and student models' (Section 4.2). This makes the general claim that 'SSL-based distillation consistently outperforms feature distillation across various SUPERB downstream tasks' (Introduction) vulnerable to selection bias. Either report results on the full SUPERB benchmark or temper the claim to the tasks evaluated and explain why the selected tasks are representative enough to support the broader conclusion.
  3. [§4.4, Table 5] The comparison across teachers is confounded by training data and model scale, as the paper itself acknowledges: HuBERT-base was trained on LibriSpeech 960h while HuBERT-large was trained on Libri-Light 60k hours. The observation that HuBERT-large improves ASR but not speaker tasks is then attributed to single-layer target generation, but this attribution is not defensible given the data mismatch. Please either control for training data, or explicitly reframe this subsection as a pilot observation rather than evidence about target-layer information loss.
  4. [Tables 1–5] All metrics are reported as single runs with no variance estimates, number of seeds, or statistical significance tests. Several claimed wins are small (e.g., ASV 5.25 vs. 5.38 in Table 1, SID 79.23 vs. 78.66, and the ASV/SID differences in Tables 3–4), so it is currently impossible to tell whether the 'consistently outperforms' claim holds beyond seed noise. At minimum, report mean and standard deviation over multiple runs for the main comparison and the ablation tables, or state the standard evaluation protocol used by prior works if single-run reporting is conventional for these benchmarks.
  5. [§4.4, Table 2] The lower-bound result is striking: a from-scratch HuBERT-narrow model beats most prior distillation methods on ASR and ASV. This is framed as a missing baseline in prior work, but it also raises a question about the paper's own comparison: if the same student trained from scratch already achieves close to the distilled performance on several tasks, the advantage of DiceHuBERT over feature distillation should be quantified relative to this lower bound, not only relative to published methods. Please discuss whether the main Table 1 improvements are attributable to the SSL objective or to the student architecture itself, and add a direct comparison of DiceHuBERT against the from-scratch lower bound in the same table.
minor comments (5)
  1. [§2.2] The sentence 'The each following iteration can be viewed as a self-distillation process' is grammatically broken ('The each') and should be revised.
  2. [§3.2] Equation (3) defines soft labels as a softmax over negative L2 distances to k-means centroids, but the text then says 'hard labels are computed via argmin of these distances'; this is consistent, but the definition of hard labels should explicitly state that they are the argmin cluster index, not one-hot distances, to avoid confusion with the soft-label formula.
  3. [End of manuscript] The GitHub link appears to point to the generic facebookresearch/fairseq repository rather than to a DiceHuBERT-specific release; if no dedicated code is provided, the link should be removed or replaced with a clear statement about code availability.
  4. [Table 1] LightHuBERT is excluded with a one-line justification about computational cost, but it is a published distillation method and its exclusion should be explained in more detail, including what 'extensive computational cost' means in terms of GPU hours or hardware requirements.
  5. [§4.1] The phrase 'following exactly the same setup of HuBERT second iteration pre-training' is not fully precise because the student architecture differs; please specify which hyperparameters (mask span, masking rate, batch size, learning rate, iterations) are unchanged and which, if any, are re-tuned for the smaller models.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DiceHuBERT is an empirical distillation recipe evaluated on an external benchmark, not a derivation that reduces to its inputs.

full rationale

DiceHuBERT's central claim is that training a smaller HuBERT with the original masked-prediction cross-entropy loss on k-means cluster targets from a teacher outperforms feature-distillation baselines on SUPERB. This is an empirical comparison on an independent benchmark, not a quantity derived from a fitted constant or from a self-citation. The student is trained with the same SSL objective as HuBERT, but the targets come from teacher cluster centroids, and the evaluation is external (SUPERB tasks with standard recipes), so the outcome is not forced by construction. The choice of teacher layer (6 for N=1, 9 for N=2, 18 for HuBERT-large) follows the original HuBERT paper, and no parameter is fitted to the reported downstream numbers. Self-citations appear only in a forward-looking suggestion about averaging layers (ref [30]) and are not load-bearing. The inconsistent relative-improvement percentages between the abstract (PR >21%, ASR >14%) and Section 4.3/Conclusion (ASR 21%, PR 14%) are a reporting/correctness issue, not circularity, and do not change the assessment that the derivation chain is self-contained.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

DiceHuBERT introduces no new model components, losses, or physical entities. It reuses HuBERT's SSL objective and k-means target generation, so the ledger is dominated by inherited hyperparameters and domain assumptions rather than invented structure.

free parameters (3)
  • k-means cluster count K = Not reported in this paper; standard HuBERT value (typically 500) used
    Target labels are generated by k-means; cluster count controls target granularity and affects distillation quality, but the paper provides no ablation or justification for K in the DiceHuBERT setting.
  • Teacher target layer index = 6 (HuBERT-base N=1), 9 (HuBERT-base N=2), 18 (HuBERT-large)
    A single hidden layer is selected for k-means target generation following the HuBERT paper; this choice determines what knowledge the student can learn and is not swept in this paper.
  • Soft-label temperature tau = 1, 5, 10 in ablation; default final model uses hard labels
    Equation 3 introduces tau as a hand-set parameter; the paper sweeps it and finds tau=10 hurts performance, so it is not used in the headline result, but it is an ad hoc choice affecting soft-label variants.
assumptions (5)
  • domain assumption k-means cluster assignments of teacher hidden features provide useful training targets for masked prediction.
    Used throughout Sections 3.2 and 4; no independent evidence in this paper that cluster labels from a compressed teacher retain sufficient information, though results suggest it works on selected tasks.
  • domain assumption HuBERT's iterative self-distillation is a valid mechanism for transferring knowledge to a smaller model.
    The core replacement of teacher with student in the next iteration assumes that the SSL masked-prediction objective alone, without feature alignment, transfers the teacher's knowledge; asserted in Section 3.
  • domain assumption LibriSpeech 960h is sufficient data for training compressed speech SSL models.
    Training and evaluation use LibriSpeech 960h; the paper does not study data scaling.
  • domain assumption Four selected SUPERB tasks (PR, ASR, SID, ASV) are representative of overall compression quality.
    Section 4.2 explicitly chooses tasks with large teacher-student gaps, so conclusions may not generalize to the other 16 SUPERB tasks.
  • domain assumption Standard optimization machinery and hyperparameters from HuBERT apply unchanged to student architectures.
    Implementation uses the same setup as HuBERT second iteration, assuming no architecture-specific retuning is needed beyond the reported SID learning rate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective." pith.science (2026). https://pith.science/paper/IJN6AP72

@misc{pith2026250702911,
  author       = {Pith},
  title        = {Pith review of: DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IJN6AP72}},
  note         = {Machine review of arXiv:2507.02911}
}
read the original abstract

We introduce DiceHuBERT, a knowledge distillation framework for compressing HuBERT, a widely used self-supervised learning (SSL)-based speech foundation model. Unlike existing distillation methods that rely on layer-wise and feature-wise mapping between teacher and student models, DiceHuBERT leverages HuBERT's iterative self-distillation mechanism by directly replacing the original model with a student model. This replacement allows the student to be trained using the same SSL objective used when pre-training HuBERT, eliminating the need for additional modules or architectural constraints. Experimental results on SUPERB show that DiceHuBERT consistently outperforms existing distillation methods, improving phoneme recognition performance by over 21% and ASR performance by more than 14%. Furthermore, DiceHuBERT demonstrates competitive performance across multiple tasks, highlighting its clear advantage.

Figures

Figures reproduced from arXiv: 2507.02911 by the authors.

Figure 1
Figure 1. DiceHuBERT differs from feature distillation-based methods in that it distills knowledge from target labels generated by a teacher model using k-means clustering. This process allows DiceHuBERT to be trained using an SSL objective that closely resembles the training objective of the teacher model. W(l) ∈ R DS×DT is employed: Lfeat = X l∈{1,...,L} α(l)MSE(H T (l) , HS (l)W(l)), (1) where H(l) is a feature from l-th l… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 27 canonical work pages

  1. [1]

    teacher” model to a smaller “student

    Introduction Self-supervised learning (SSL)-based speech foundation mod- els [1–8] have received significant attention due to their ability to generate general-purpose speech representations for a wide range of speech tasks. However, the substantial size of the SSL-based foundation models makes them unsuitable for de- ployment in mobile applications [9, 1...

    work page

  2. [2]

    Preliminaries 2.1. Prior HuBERT Distillation Most prior works on HuBERT distillation [18, 21, 22, 24] fo- cus on transferring knowledge from the pre-trained second- iteration HuBERT model (used as the teacher) to the student model through layer-wise feature distillation, as shown in Fig- ure 1 (b). The student model usually has the same number of layers a...

    work page Pith review arXiv 2025

  3. [3]

    DiceHuBERT We propose DiceHuBERT, a HuBERT distillation framework that offers a simpler and natural distillation approach within HuBERT itself compared to previous methods. DiceHuBERT leverages HuBERT’s iterative self-distillation process by reduc- ing the model size when transitioning from iterationN(teacher) to iterationN+1(student), all while maintaini...

    work page

  4. [4]

    Experiments 4.1. Implementation Details Since there are no publicly available pre-train weights for the first iteration of HuBERT, we train this model from scratch us- ing the hyper-parameters reported in the HuBERT paper [2]. For our distillation process, we follow exactly the same setup of HuBERT second iteration pre-training, while changing only the mo...

    work page

  5. [5]

    Conclusion In this paper, we introduce DiceHuBERT, a model compression framework that leverages HuBERT’s self-distillation mecha- nism by directly replacing the original model with a student model. DiceHuBERT outperforms feature distillation methods across various SUPERB downstream tasks, achieving over 21% improvement in ASR and 14% in PR compared to pre...

    work page

  6. [6]

    wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech repre- sentations,”Advances in neural information processing systems, vol. 33, pp. 12 449–12 460, 2020

    2020

  7. [7]

    Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM transactions on audio, speech, and language processing, vol. 29, pp. 3451–3460, 2021

    2021

  8. [8]

    Wavlm: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “Wavlm: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

Show all 37 references
  1. [9]

    Xls-r: Self- supervised cross-lingual speech representation learning at scale,

    A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. V on Platen, Y . Saraf, J. Pinoet al., “Xls-r: Self- supervised cross-lingual speech representation learning at scale,” arXiv preprint arXiv:2111.09296, 2021

    2021 arXiv

  2. [10]

    Speech-xlnet: Unsupervised acoustic model pretraining for self- attention networks,

    X. Song, G. Wang, Y . Huang, Z. Wu, D. Su, and H. Meng, “Speech-xlnet: Unsupervised acoustic model pretraining for self- attention networks,” 2020

    2020

  3. [11]

    Investigating self-supervised learning for speech enhancement and separation,

    Z. Huang, S. Watanabe, S.-w. Yang, P. Garc´ıa, and S. Khudanpur, “Investigating self-supervised learning for speech enhancement and separation,” inICASSP 2022-2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6837–6841

    2022

  4. [12]

    An explo- ration of self-supervised pretrained representations for end-to-end speech recognition,

    X. Chang, T. Maekaku, P. Guo, J. Shi, Y .-J. Lu, A. S. Subrama- nian, T. Wang, S.-w. Yang, Y . Tsao, H.-y. Leeet al., “An explo- ration of self-supervised pretrained representations for end-to-end speech recognition,” in2021 IEEE Automatic Speech Recognition and Understanding ...

    2021

  5. [13]

    W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,

    Y .-A. Chung, Y . Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y . Wu, “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,” in 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, p...

    2021

  6. [14]

    Convolutional neural networks for small-footprint keyword spotting

    T. N. Sainath and C. Parada, “Convolutional neural networks for small-footprint keyword spotting.” inInterspeech, 2015, pp. 1478–1482

    2015

  7. [15]

    Monophone-based background modeling for two-stage on-device wake word de- tection,

    M. Wu, S. Panchapagesan, M. Sun, J. Gu, R. Thomas, S. N. P. Vi- taladevuni, B. Hoffmeister, and A. Mandal, “Monophone-based background modeling for two-stage on-device wake word de- tection,” in2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICAS...

    2018

  8. [16]

    Distilling the knowledge in a neural network,

    G. Hinton, “Distilling the knowledge in a neural network,”arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  9. [17]

    Knowledge distilla- tion: A survey,

    J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distilla- tion: A survey,”International Journal of Computer Vision, vol. 129, no. 6, pp. 1789–1819, 2021

    2021

  10. [18]

    Do deep nets really need to be deep?

    J. Ba and R. Caruana, “Do deep nets really need to be deep?” Advances in neural information processing systems, vol. 27, 2014

    2014

  11. [19]

    Learn- ing efficient object detection models with knowledge distillation,

    G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker, “Learn- ing efficient object detection models with knowledge distillation,” Advances in neural information processing systems, vol. 30, 2017

    2017

  12. [20]

    Knowledge distillation across ensembles of multilingual mod- els for low-resource languages,

    J. Cui, B. Kingsbury, B. Ramabhadran, G. Saon, T. Sercu, K. Audhkhasi, A. Sethy, M. Nussbaum-Thom, and A. Rosenberg, “Knowledge distillation across ensembles of multilingual mod- els for low-resource languages,” in2017 IEEE International Con- ference on Acoustics, Speech and S...

    2017

  13. [21]

    Distilling knowl- edge from graph convolutional networks,

    Y . Yang, J. Qiu, M. Song, D. Tao, and X. Wang, “Distilling knowl- edge from graph convolutional networks,” inProceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, 2020, pp. 7074–7083

    2020

  14. [22]

    Domain adaptation of dnn acoustic models using knowledge distillation,

    T. Asami, R. Masumura, Y . Yamaguchi, H. Masataki, and Y . Aono, “Domain adaptation of dnn acoustic models using knowledge distillation,” in2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 5185–5189

    2017

  15. [23]

    Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling,

    S. Gandhi, P. von Platen, and A. M. Rush, “Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling,”arXiv preprint arXiv:2311.00430, 2023

    2023 arXiv

  16. [24]

    Knowledge distillation with the reused teacher classifier,

    D. Chen, J.-P. Mei, H. Zhang, C. Wang, Y . Feng, and C. Chen, “Knowledge distillation with the reused teacher classifier,” inPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 11 933–11 942

    2022

  17. [25]

    Dphubert: Joint distillation and pruning of self-supervised speech models,

    Y . Peng, Y . Sudo, S. Muhammad, and S. Watanabe, “Dphubert: Joint distillation and pruning of self-supervised speech models,” inProceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, vol. 2023, 2023, pp. 62–66

    2023

  18. [26]

    Fithubert: Going thinner and deeper for knowledge distillation of speech self-supervised learning,

    Y . Lee, K. JANG, J. Goo, Y . Jung, and H.-R. Kim, “Fithubert: Going thinner and deeper for knowledge distillation of speech self-supervised learning,” in23rd Annual Conference of the In- ternational Speech Communication Association, INTERSPEECH

  19. [27]

    Su- perb: Speech processing universal performance benchmark,

    S. W. Yang, P. H. Chi, Y . S. Chuang, C. I. J. Lai, K. Lakhotia, Y . Y . Lin, A. T. Liu, J. Shi, X. Chang, G. T. Linet al., “Su- perb: Speech processing universal performance benchmark,” in 22nd Annual Conference of the International Speech Communi- cation Association, INTERSP...

    2021

  20. [28]

    Recycle-and-distill: Universal compression strategy for transformer-based speech ssl models with attention map reusing and masking distillation,

    K. Jang, S. Kim, S. Yun, and H.-R. Kim, “Recycle-and-distill: Universal compression strategy for transformer-based speech ssl models with attention map reusing and masking distillation,” in24th International Speech Communication Association, Inter- speech 2023. International S...

    2023

  21. [29]

    Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert,

    H.-J. Chang, S.-w. Yang, and H.-y. Lee, “Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert,” inICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 7087–7091

    2022

  22. [30]

    Star: Distilling speech temporal relation for lightweight speech self-supervised learning models,

    K. Jang, S. Kim, and H. Kim, “Star: Distilling speech temporal relation for lightweight speech self-supervised learning models,” inICASSP 2024-2024 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10 721–10 725

    2024

  23. [31]

    Lighthubert: Lightweight and configurable speech representation learning with once-for-all hidden-unit bert,

    R. Wang, Q. Bai, J. Ao, L. Zhou, Z. Xiong, Z. Wei, Y . Zhang, T. Ko, and H. Li, “Lighthubert: Lightweight and configurable speech representation learning with once-for-all hidden-unit bert,” Proc. Interspeech 2022, pp. 1686–1690, 2022

    2022

  24. [32]

    k-means++: The advantages of careful seeding,

    D. Arthur and S. Vassilvitskii, “k-means++: The advantages of careful seeding,” Stanford, Tech. Rep., 2006

    2006

  25. [34]

    Why distillation helps: a statistical perspective,

    A. K. Menon, A. S. Rawat, S. J. Reddi, S. Kim, and S. Kumar, “Why distillation helps: a statistical perspective,”arXiv preprint arXiv:2005.10419, 2020

    2005 arXiv

  26. [35]

    Lib- rispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: an asr corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  27. [36]

    Exploring prediction targets in masked pre-training for speech foundation models,

    L.-W. Chen, T. Higuchi, H. Bai, A. Hussen Abdelaziz, A. Rud- nicky, S. Watanabe, T. Likhomanenko, B.-J. Theobald, and Z. Aldeneh, “Exploring prediction targets in masked pre-training for speech foundation models,”arXiv e-prints, pp. arXiv–2409, 2024

    2024

  28. [37]

    Libri-light: A benchmark for asr with limited or no supervision,

    J. Kahn, M. Rivi `ere, W. Zheng, E. Kharitonov, Q. Xu, P. E. Mazar´e, J. Karadayi, V . Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux, “Libri-light: A benchmark for asr with limited or no supervision,” inICASSP 2020 - 2...

    2020

  29. [2022]

    3588–3592

    ISCA, 2022, pp. 3588–3592

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.