Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

USAD: Universal Speech and Audio Representation via Distillation

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single audio encoder, trained by distilling a speech and an audio specialist, approaches each expert on its own benchmark.

desk verdict USAD is a solid, compute-efficient demonstration that one encoder can handle speech and audio via two-teacher distillation, but the 'near state-of-the-art' claim overreaches and the layer-selection robustness is untested. read the letter →

arxiv 2506.18843 v2 pith:LLT7XBLY submitted 2025-06-23 cs.SD cs.CLeess.AS

Heng-Jui Chang , Saurabhchand Bhati , James Glass , Alexander H. Liu This is my paper · ORCID
classification cs.SDcs.CLeess.AS
keywords speechandaudiorepresentationlearningself-supervisedknowledgedistillationlayer-to-layeruniversalencoderSUPERBbenchmarkHEAR
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that one general-purpose audio encoder can replace the current practice of using separate self-supervised models for speech, environmental sound, and music. USAD reaches this goal by distilling knowledge from two pre-trained specialists, a speech model and an audio model, into a single student transformer using a small number of matched layers and a simple frame-level L1-cosine loss. Training on a balanced mixture of speech, sound, and music data lets the student absorb both domains at once. The paper reports that the resulting encoder is competitive with both teachers and approaches state of the art on the SUPERB and HEAR benchmarks, a step toward one audio representation for many downstream systems.

What carries the argument

The central mechanism is sparse layer-to-layer distillation with two teachers. At $K=4$ matched student layers, the student's hidden states pass through two MLP prediction heads that try to reproduce the corresponding teacher layers' feed-forward-network features; the loss is per-frame $\frac{1}{D}\|\tilde{z}-z\|_1 - \log\sigma(\cos(\tilde{z},z))$, summed over both teachers, layers, and frames. Sparsity (every third layer) cuts the distillation loss compute by about 75% compared with dense layer matching, and the L1-cosine objective replaces the negative-sample contrastive loss of earlier layer-to-layer methods.

What would settle it

Take the released USAD Base checkpoint and evaluate it on two held-out tasks that are not in SUPERB or HEAR, such as LibriSpeech test-other (word error rate) and FSD50K (audio tagging mAP); if it cannot match the speech teacher within 2 points on the first and the audio teacher within 2 points on the second, the claim that one encoder remains competitive across both domains is not supported.

Watch

Extended reading notes

Core claim

The central claim is that domain-specificity is not an inherent property of audio representations but a consequence of training data and teacher choice. In USAD, a student transformer is trained to reproduce the feed-forward network features of a speech teacher (WavLM Base+) at layers {3,6,9,12} and of an audio teacher (ATST Frame) at the corresponding layers, through two separate MLP heads, with a frame-wise objective that minimizes L1 distance and maximizes cosine similarity. The paper's central discovery is that this sparse, non-contrastive, frame-aligned distillation over a balanced multi-domain dataset yields a single encoder whose SUPERB and HEAR scores approach or exceed those of the individual teachers, and whose average HEAR performance surpasses the concatenated features of both teachers. A second supporting finding is that frame-based audio teachers, rather than patch-based ones, are the right partner for a speech teacher, because their temporal alignment prevents conflicting learning targets.

Load-bearing premise

The load-bearing assumption is that distilling only K=4 selected layers of the student, with a frame-wise L1-cosine loss, can transfer enough knowledge from both teachers simultaneously without one domain's target overwhelming the other.

Editorial extensions

If this is right

  • A single frozen encoder can serve speech, sound, and music downstream tasks, removing the need to run multiple domain-specific SSL models for mixed audio.
  • Distilling from frame-based audio teachers preserves speech phonetic resolution while still transferring audio-event knowledge; patch-based teachers degrade phoneme recognition.
  • Sparse distillation at K=4 cuts the distillation compute by about 75% relative to dense layer matching, making multi-teacher distillation practical.
  • Scaling USAD from Small to Large narrows the gap to both teachers, with instance-level speech tasks surpassing the speech teacher and audio tasks approaching or beating the audio teacher.
  • Training-data balance matters: a 50/50 speech-to-non-speech ratio gives the best overall trade-off, with phoneme recognition more robust to non-speech oversampling than audio tagging is.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the teacher-choice and layer-choice results generalize, a practical recipe for other modalities is to pick a frame-aligned teacher per domain and distill only a few matched layers, which should lower the cost of building multimodal encoders.
  • The fixed K=4 layer schedule suggests that teacher layers are not equally informative; a learned or input-dependent layer selection might outperform the paper's uniform choice.
  • The strong correlation between speech-to-non-speech ratio and downstream performance implies that the ideal training mixture depends on the target benchmark, so USAD could be tuned per deployment by re-weighting the same corpus.
  • USAD's representation quality for audio language models is not directly evaluated; a testable extension is to plug the USAD encoder into an audio language model and compare against domain-specific encoders on instruction-following benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes USAD, a universal speech/audio encoder trained by sparse layer-to-layer distillation from two domain-specific self-supervised teachers: WavLM Base+ for speech and ATST Frame for audio. The student is trained on a balanced mixed dataset (speech, sound, music) using a per-frame L1-cosine loss applied at a sparse set of student layers (K=4: layers 3, 6, 9, 12). The authors evaluate on SUPERB, HEAR, AS-20K, and ESC-50, and report that USAD is competitive with, and in some speech tasks close to, domain-specific specialists, while using substantially less training compute than training a mixed-domain SSL model from scratch. The paper also presents ablations on teacher-model choice, data distribution, distillation objective, number of distilled layers, and model scale.

Significance. If the claims hold, USAD is a useful contribution: it shows that a single encoder trained by distilling two domain-specific teachers can cover speech, sound, and music with reasonable compute, and it provides a practical recipe (frame-based teachers, sparse layer matching, L1-cosine loss) for multi-teacher audio distillation. The paper's strengths include extensive ablations on teacher pairs, data ratios, distillation objectives, and layer counts; released model checkpoints; and a clear efficiency comparison in terms of training FLOPS. The main risk is that the headline claim of 'near state-of-the-art' is not uniformly supported by the reported numbers, and the central sparse-layer mechanism is not fully validated because the layer-selection rule is not ablated.

major comments (3)
  1. [§IV-C, Table III] The abstract's claim of 'near state-of-the-art results' on audio benchmarks is not supported by the reported numbers. USAD Large obtains 92.7 on ESC-50, while EAT and SSLAM both obtain 95.9, and on AS-20K it obtains 37.4 mAP versus 40.9 for SSLAM, 40.2 for EAT, and 38.3 for BEATs. These are sizable gaps on the primary audio tasks, not near-SOTA. The claim should be either softened or explicitly qualified as 'near state-of-the-art for a single general-purpose encoder with the studied teacher pair.'
  2. [§III-B, Table VI] The robustness of the sparse layer selection is not established. The paper fixes K=4 at equidistant layers {3, 6, 9, 12}, citing high similarity between consecutive transformer layers [36], but it never varies which four layers are distilled. Table VI shows that K=6 and K=12 degrade both PR and AS-20K relative to K=4, which indicates that some teacher layers provide unhelpful targets; without ablating alternative 4-layer subsets (e.g., {2, 5, 8, 11} or {1, 4, 7, 10}), the method's success could depend on a particular hyperparameter choice rather than on a robust distillation principle. Since the 'universal' claim rests on the student absorbing both domains through these four layers, this is a load-bearing missing ablation.
  3. [§IV-E, Table V] The default teacher pair (WavLM Base+ and ATST Frame) is not explicitly justified. Table V shows a trade-off: data2vec 2.0 Speech + ATST Frame gives better PR (7.4 vs. 8.7) but worse AS-20K (29.2 vs. 30.6) than WavLM Base+ + ATST Frame. The text says the default setting is underlined, but the underline is not visible in the typeset table, and the selection criterion (e.g., maximizing AS-20K, or balancing the two metrics) is never stated. This makes the main result partly dependent on an unexplained modeling choice.
minor comments (4)
  1. [Table III] The four numeric columns following ESC-50 are labeled only as 'Speech Audio Avg' in the header; the individual subcolumn names (e.g., Speech Frame, Speech Instance, Audio Frame, Audio Instance) are not defined in the caption or text. Please add an explicit legend.
  2. [§IV-D] The 'WavLM + ATST' topline is described as concatenating the representations of the two teachers, but the implementation details (how the concatenated representation is fed to each downstream head, and how dimensional mismatch is handled) are not given. One sentence would suffice.
  3. [Abstract and Conclusion] The abstract says 'near state-of-the-art results' while the conclusion says 'competitive performance'; these claims should be aligned and quantified, especially given the ESC-50 and AS-20K gaps noted in the major comments.
  4. [§IV-A, Table II] The fine-tuning protocol for AS-20K and ESC-50 follows EAT, but the number of epochs, learning rates, and number of random seeds are not reported. If these are standard from the cited EAT repository, a brief statement to that effect would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: USAD is a distillation method evaluated on external benchmarks, with no claim that reduces to its own fitted inputs.

full rationale

USAD's training objective is to predict teacher representations via layer-to-layer distillation, and all central claims are evaluated on external downstream benchmarks (SUPERB, HEAR, AS-20K, ESC-50) that are not used for fitting teacher choice, layer count K, or data mix. The method's loss is defined in Section III-B as L1-cosine similarity between student MLP outputs and teacher FFN features; this is exactly what is optimized, and the paper does not rename this optimization as a prediction. The self-citations to CoLLD [18] and DistilHuBERT [30] are used only as methodological precedents for layer-to-layer distillation and the L1-cosine objective, not as unverified support for the paper's empirical conclusions. Ablations in Tables V and VI vary teachers, distillation objectives, and K on held-out benchmark performance, which is a legitimate empirical search rather than a circular fit to the test set. The observed degradation at larger K is reported as a finding, not hidden. No equation in the paper defines a downstream result in terms of the fitted parameters, and no "uniqueness theorem" or author-imported constraint forces the design choice. The paper is self-contained against external benchmarks, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim relies on a few design choices (K, data ratio, teacher pair) and empirical assumptions about the distillation loss and feature types. No new theoretical entities are invented. The assumptions are load-bearing because changing them could affect the student's ability to learn both domains.

free parameters (3)
  • K = 4
    Number of distillation layers, chosen after ablations showing K=3 and K=4 behave similarly; not derived from theory.
  • Speech-to-non-speech upsampling ratio = 2X
    Upsampling factor for sound and music data to balance the training set, chosen to improve AS-20K while keeping PR reasonable.
  • Teacher pair = WavLM Base+ and ATST Frame
    Selected after teacher ablation experiments; other combinations yield different trade-offs.
assumptions (3)
  • domain assumption Consecutive transformer layers have high similarity, justifying sparse layer-to-layer distillation
    Invoked in Section III-B to reduce the number of distilled layers from 12 to 4, relying on reference [36].
  • domain assumption L1-cosine similarity loss is a sufficient objective for transferring teacher knowledge
    The loss in Section III-B omits negative sampling and masking; the paper provides empirical support via ablations but no theoretical justification.
  • domain assumption Frame-based feature extraction is preferable to patch-based for joint speech and audio representation learning
    Argued in Section III-C and supported by the teacher ablation in Table V, but only tested on a limited set of tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of USAD: Universal Speech and Audio Representation via Distillation." pith.science (2026). https://pith.science/paper/LLT7XBLY

@misc{pith2026250618843,
  author       = {Pith},
  title        = {Pith review of: USAD: Universal Speech and Audio Representation via Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LLT7XBLY}},
  note         = {Machine review of arXiv:2506.18843}
}
read the original abstract

Self-supervised learning (SSL) has revolutionized audio representations, yet models often remain domain-specific, focusing on either speech or non-speech tasks. In this work, we present Universal Speech and Audio Distillation (USAD), a unified approach to audio representation learning that integrates diverse audio types - speech, sound, and music - into a single model. USAD employs efficient layer-to-layer distillation from domain-specific SSL models to train a student on a comprehensive audio dataset. USAD offers competitive performance across various benchmarks and datasets, including frame and instance-level speech processing tasks, audio tagging, and sound classification, achieving near state-of-the-art results with a single encoder on SUPERB and HEAR benchmarks.

Figures

Figures reproduced from arXiv: 2506.18843 by the authors.

Figure 1
Figure 1. The proposed USAD. In stage (I), a student model distills knowledge from two domain-specific teachers through sparse [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Training dataset speech-to-non-speech ratio vs. down [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations

    eess.AS 2025-10 conditional novelty 6.0 of 10

    SPEAR unifies speech and audio self-supervised learning by masked prediction of multi-codebook tokens distilled from two domain teachers, beating WavLM Large on 12 of 15 SUPERB tasks and scoring competitively on HEAR.

Reference graph

Works this paper leans on

59 extracted references · 53 canonical work pages · cited by 1 Pith paper

  1. [34]

    Distilling a speech and music encoder with task arithmetic,

    F. Ritter-Gutierrez, Y .-C. Lin, J.-C. Wei, J. H. Wong, E. S. Chng, N. F. Chen, and H.-y. Lee, “Distilling a speech and music encoder with task arithmetic,” in Interspeech, 2025

    work page 2025

  2. [36]

    Layer-wise analysis of a self- supervised speech representation model,

    A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self- supervised speech representation model,” in ASRU, 2021

    work page 2021

  3. [1]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in NeurIPS, 2020

    2020

  4. [2]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” TASLP, vol. 29, 2021

    work page 2021

  5. [3]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen et al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE JSTSP, vol. 16, 2022

    work page 2022

  6. [4]

    Ssast: Self-supervised audio spectrogram transformer,

    Y . Gong, C.-I. Lai, Y .-A. Chung, and J. Glass, “Ssast: Self-supervised audio spectrogram transformer,” in AAAI, 2022

    work page 2022

  7. [5]

    Beats: Audio pre-training with acoustic tokenizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei, “Beats: Audio pre-training with acoustic tokenizers,” in ICML, 2022

    work page 2022

  8. [6]

    Mert: Acoustic music understanding model with large-scale self-supervised training,

    Y . Li, R. Yuan, G. Zhang, Y . Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos et al., “Mert: Acoustic music understanding model with large-scale self-supervised training,” ICLR, 2024

    work page 2024

Show all 59 references
  1. [7]

    Listen, think, and understand,

    Y . Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass, “Listen, think, and understand,” in ICLR, 2024

    2024

  2. [8]

    SALMONN: Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang, “SALMONN: Towards generic hearing abilities for large language models,” in ICLR, 2024

    2024

  3. [9]

    Qwen2-audio technical report,

    Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Lin et al., “Qwen2-audio technical report,” arXiv, 2024

    2024

  4. [10]

    Gama: A large audio- language model with advanced audio understanding and complex rea- soning abilities,

    S. Ghosh, S. Kumar, A. Seth, C. K. R. Evuru, U. Tyagi, S. Sakshi, O. Nieto, R. Duraiswami, and D. Manocha, “Gama: A large audio- language model with advanced audio understanding and complex rea- soning abilities,” in EMNLP, 2024

    2024

  5. [11]

    Google usm: Scaling automatic speech recognition beyond 100 languages,

    Y . Zhang et al. , “Google usm: Scaling automatic speech recognition beyond 100 languages,” arXiv, 2023

    2023

  6. [12]

    Speechtokenizer: Unified speech tokenizer for speech language models,

    X. Zhang, D. Zhang, S. Li, Y . Zhou, and X. Qiu, “Speechtokenizer: Unified speech tokenizer for speech language models,” in ICLR, 2024

    2024

  7. [13]

    Soundstorm: Efficient parallel audio generation,

    Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, and M. Tagliasacchi, “Soundstorm: Efficient parallel audio generation,” arXiv, 2023

    2023

  8. [14]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D ´efossez, L. Mazar ´e, M. Orsini, A. Royer, P. P ´erez, H. J ´egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,” arXiv, 2024

    2024

  9. [15]

    Dc-spin: A speaker-invariant speech tokenizer for spoken language models,

    H.-J. Chang, H. Gong, C. Wang, J. Glass, and Y .-A. Chung, “Dc-spin: A speaker-invariant speech tokenizer for spoken language models,” in Interspeech, 2025

    2025

  10. [16]

    Joint audio and speech understanding,

    Y . Gong, A. H. Liu, H. Luo, L. Karlinsky, and J. Glass, “Joint audio and speech understanding,” in ASRU, 2023

    2023

  11. [17]

    U-sam: An audio language model for unified speech, audio, and music understanding,

    Z. Wang, X. Xia, X. Zhu, and L. Xie, “U-sam: An audio language model for unified speech, audio, and music understanding,” in Interspeech, 2025

    2025

  12. [18]

    CoLLD: Contrastive layer-to-layer distillation for compressing multi- lingual pre-trained speech encoders,

    H.-J. Chang, N. Dong, R. Mavlyutov, S. Popuri, and Y .-A. Chung, “CoLLD: Contrastive layer-to-layer distillation for compressing multi- lingual pre-trained speech encoders,” in ICASSP, 2024

    2024

  13. [19]

    Mae-ast: Masked autoencoding audio spectrogram transformer,

    A. Baade, P. Peng, and D. Harwath, “Mae-ast: Masked autoencoding audio spectrogram transformer,” in Interspeech, 2022

    2022

  14. [20]

    Masked spectrogram modeling using masked autoencoders for learning general-purpose audio representation,

    D. Niizumi, D. Takeuchi, Y . Ohishi, N. Harada, and K. Kashino, “Masked spectrogram modeling using masked autoencoders for learning general-purpose audio representation,” in HEAR: Holistic Evaluation of Audio Representations (NeurIPS 2021 Competition) , 2022

    2021

  15. [21]

    Masked autoencoders that listen,

    P.-Y . Huang, H. Xu, J. Li, A. Baevski, M. Auli, W. Galuba, F. Metze, and C. Feichtenhofer, “Masked autoencoders that listen,” NeurIPS, 2022

    2022

  16. [22]

    data2vec: A general framework for self-supervised learning in speech, vision and language,

    A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “data2vec: A general framework for self-supervised learning in speech, vision and language,” in ICML, 2022

    2022

  17. [23]

    Efficient self-supervised learning with contextualized target representations for vision, speech and language,

    A. Baevski, A. Babu, W.-N. Hsu, and M. Auli, “Efficient self-supervised learning with contextualized target representations for vision, speech and language,” in ICML, 2023

    2023

  18. [24]

    Dinosr: Self-distillation and online clustering for self-supervised speech repre- sentation learning,

    A. H. Liu, H.-J. Chang, M. Auli, W.-N. Hsu, and J. R. Glass, “Dinosr: Self-distillation and online clustering for self-supervised speech repre- sentation learning,” in NeurIPS, 2023

    2023

  19. [25]

    Eat: Self-supervised pre-training with efficient audio transformer,

    W. Chen, Y . Liang, Z. Ma, Z. Zheng, and X. Chen, “Eat: Self-supervised pre-training with efficient audio transformer,” in IJCAI, 2024

    2024

  20. [26]

    Sslam: Enhancing self-supervised models with audio mixtures for polyphonic soundscapes,

    T. Alex, S. Atito, A. Mustafa, M. Awais, and P. J. Jackson, “Sslam: Enhancing self-supervised models with audio mixtures for polyphonic soundscapes,” in ICLR, 2025

    2025

  21. [27]

    Byol for audio: Self-supervised learning for general-purpose audio representation,

    D. Niizumi, D. Takeuchi, Y . Ohishi, N. Harada, and K. Kashino, “Byol for audio: Self-supervised learning for general-purpose audio representation,” in IJCNN, 2021

    2021

  22. [28]

    Self-supervised audio teacher-student transformer for both clip-level and frame-level tasks,

    X. Li, N. Shao, and X. Li, “Self-supervised audio teacher-student transformer for both clip-level and frame-level tasks,” TASLP, 2024

    2024

  23. [29]

    Masked modeling duo: Learning representations by encouraging both networks to model the input,

    D. Niizumi, D. Takeuchi, Y . Ohishi, N. Harada, and K. Kashino, “Masked modeling duo: Learning representations by encouraging both networks to model the input,” in ICASSP, 2023

    2023

  24. [30]

    DistilHuBERT: Speech rep- resentation learning by layer-wise distillation of hidden-unit bert,

    H.-J. Chang, S.-w. Yang, and H.-y. Lee, “DistilHuBERT: Speech rep- resentation learning by layer-wise distillation of hidden-unit bert,” in ICASSP, 2022

    2022

  25. [31]

    Dphubert: Joint dis- tillation and pruning of self-supervised speech models,

    Y . Peng, Y . Sudo, S. Muhammad, and S. Watanabe, “Dphubert: Joint dis- tillation and pruning of self-supervised speech models,” in Interspeech, 2023

    2023

  26. [32]

    Dass: Distilled audio state space models are stronger and more duration- scalable learners,

    S. Bhati, Y . Gong, L. Karlinsky, H. Kuehne, R. Feris, and J. Glass, “Dass: Distilled audio state space models are stronger and more duration- scalable learners,” in SLT, 2024

    2024

  27. [33]

    Ensemble knowledge distillation of self- supervised speech models,

    K.-P. Huang, T.-h. Feng, Y .-K. Fu, T.-Y . Hsu, P.-C. Yen, W.-C. Tseng, K.-W. Chang, and H.-y. Lee, “Ensemble knowledge distillation of self- supervised speech models,” in ICASSP, 2023

    2023

  28. [35]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in NIPS, 2017

    2017

  29. [37]

    Robust speech recognition via large-scale weak super- vision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak super- vision,” in ICML, 2023

    2023

  30. [38]

    Librispeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in ICASSP, 2015

    2015

  31. [39]

    Libri-light: A benchmark for asr with limited or no supervision,

    J. Kahn, M. Riviere, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazar ´e, J. Karadayi, V . Liptchinsky, R. Collobert, C. Fuegen et al., “Libri-light: A benchmark for asr with limited or no supervision,” in ICASSP, 2020

    2020

  32. [40]

    Mls: A large-scale multilingual dataset for speech research,

    V . Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “Mls: A large-scale multilingual dataset for speech research,” in Interspeech, 2020

    2020

  33. [41]

    V oxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,

    C. Wang, M. Riviere, A. Lee, A. Wu, C. Talnikar, D. Haziza, M. Williamson, J. Pino, and E. Dupoux, “V oxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation,” in ACL, 2021

    2021

  34. [42]

    Gigaspeech: An evolving, multi- domain asr corpus with 10,000 hours of transcribed audio,

    G. Chen, S. Chai, G. Wang, J. Du, W.-Q. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhang et al. , “Gigaspeech: An evolving, multi- domain asr corpus with 10,000 hours of transcribed audio,” in Inter- speech, 2021

    2021

  35. [43]

    Common voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in LREC, 2020

    2020

  36. [44]

    The fisher corpus: A resource for the next generations of speech-to-text

    C. Cieri, D. Miller, and K. Walker, “The fisher corpus: A resource for the next generations of speech-to-text.” in LREC, 2004

    2004

  37. [45]

    V oxlingua107: a dataset for spoken language recognition,

    J. Valk and T. Alum ¨ae, “V oxlingua107: a dataset for spoken language recognition,” in SLT, 2021

    2021

  38. [46]

    Audio set: An ontology and human-labeled dataset for audio events,

    J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in ICASSP, 2017

    2017

  39. [47]

    Soundnet: Learning sound representations from unlabeled video,

    Y . Aytar, C. V ondrick, and A. Torralba, “Soundnet: Learning sound representations from unlabeled video,” in NIPS, 2016

    2016

  40. [48]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in ICASSP, 2023

    2023

  41. [49]

    Music4all: A new music database and its applications,

    I. A. P. Santana, F. Pinhelli, J. Donini, L. Catharin, R. B. Mangolin, V . D. Feltrim, M. A. Domingues et al., “Music4all: A new music database and its applications,” in IWSSIP, 2020

    2020

  42. [50]

    fairseq: A fast, extensible toolkit for sequence modeling,

    M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng, D. Grangier, and M. Auli, “fairseq: A fast, extensible toolkit for sequence modeling,” in NAACL-HLT, 2019

    2019

  43. [51]

    Layer normalization,

    J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv, 2016

    2016

  44. [52]

    Self-attention with relative position representations,

    P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention with relative position representations,” in NAACL, 2018

    2018

  45. [53]

    Speech commands: A dataset for limited-vocabulary speech recognition,

    P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” arXiv, 2018

    2018

  46. [54]

    SUPERB: Speech processing universal performance benchmark,

    S.-w. Yang et al., “SUPERB: Speech processing universal performance benchmark,” in Interspeech, 2021

    2021

  47. [55]

    SUPERB-SG: Enhanced speech processing universal PERformance benchmark for semantic and generative capabilities,

    H.-S. Tsai et al., “SUPERB-SG: Enhanced speech processing universal PERformance benchmark for semantic and generative capabilities,” in ACL, 2022

    2022

  48. [56]

    A large-scale evaluation of speech foundation models,

    S.-w. Yang, H.-J. Chang, Z. Huang, A. T. Liu, C.-I. Lai, H. Wu, J. Shi, X. Chang, H.-S. Tsai, W.-C. Huang et al., “A large-scale evaluation of speech foundation models,” TASLP, 2024

    2024

  49. [57]

    Hear: Holistic evaluation of audio representations,

    J. Turian, J. Shier, H. R. Khan, B. Raj, B. W. Schuller, C. J. Steinmetz, C. Malloy, G. Tzanetakis, G. Velarde, K. McNally et al., “Hear: Holistic evaluation of audio representations,” in NeurIPS 2021 Competitions and Demonstrations Track, 2022

    2021

  50. [58]

    ESC: Dataset for Environmental Sound Classification,

    K. J. Piczak, “ESC: Dataset for Environmental Sound Classification,” in ACM MM, 2015

    2015

  51. [59]

    Superb@ slt 2022: Challenge on generalization and efficiency of self-supervised speech representation learning,

    T.-h. Feng et al. , “Superb@ slt 2022: Challenge on generalization and efficiency of self-supervised speech representation learning,” in SLT, 2022

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.