Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

X-ARES: A Comprehensive Framework for Assessing Audio Encoder Performance

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An open benchmark ranks 13 audio encoders on 22 tasks spanning speech, environmental sound, and music, under both a trained linear readout and a parameter-free k-NN classifier.

desk verdict A useful, well-scoped benchmark extension whose headline ranking is readier than the evidence supports; worth reviewing, but the aggregate score needs sensitivity analysis before it becomes a standard reference. read the letter →

arxiv 2505.16369 v2 pith:CHVGOW5L submitted 2025-05-22 cs.SD eess.AS

Junbo Zhang , Heinrich Dinkel , Yadong Niu , Chenyu Liu , Si Cheng , Anbei Zhao , Jian Luan This is my paper
classification cs.SDeess.AS
keywords audioencoderevaluationgeneralbenchmarkself-supervisedrepresentationlinearprobingk-nearestneighborsspeechrecognitionsoundeventdetectionmusicinformationretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces X-ARES, an open-source benchmark for comparing pretrained audio encoders. The suite assembles 22 evaluation tasks spanning speech, environmental sound, and music, and scores each encoder under two protocols: a trained linear readout and a parameter-free k-nearest-neighbor classifier. The central contention is that judging encoders on a single domain or a single readout hides how representation quality transfers. The reported results show that speech-specialized encoders dominate speech tasks but drop sharply on environmental and music tasks, sound-event encoders score zero on speech recognition, and the ranking differs between the two protocols. If the benchmark is sound, it gives the field a shared yardstick for comparing audio representations and a template for adding new tasks and encoders.

What carries the argument

The load-bearing object is the metric-normalization and aggregation scheme. Each task's raw score is rescaled to a 0-1 scale by Equation (1), using the theoretical worst and best possible values, and the 22 normalized scores are then combined into a single number by Equation (2), a weighted average whose weights are the sizes of each task's test set. This one-number score per encoder is what produces the headline orderings in Table 2. Two evaluation heads carry the comparison: a linear MLP readout, which measures how easily a task can be solved from frozen embeddings, and a k-NN classifier, which measures raw feature-space geometry without training. The framework also enforces one input convention, a single frame-level embedding sequence with a stated temporal resolution, and uses streaming tar archives plus frozen task-specific decoders to keep the pipeline uniform across tasks.

What would settle it

Recompute the aggregate score S in Equation (2) under equal task weights and under rank-based normalization; if the top MLP or k-NN encoder changes, the reported ordering is an artifact of the test-set-size weighting rather than a stable property of the encoders.

Watch

Extended reading notes

Core claim

On the paper's own terms, X-ARES is the claim that a single framework with 22 tasks across speech, environmental sound, and music, scored by both a trained linear readout and a parameter-free k-NN classifier, can serve as a standardized platform for audio-encoder evaluation. Every user encoder is reduced to one frame-level embedding sequence, evaluated by a frozen linear MLP head for each task and by k-NN, with specialized tasks such as speech recognition and audio captioning adding frozen pretrained decoders. Across thirteen state-of-the-art encoders, the weighted-average ranking places a large-scale masked-audio encoder (Dasheng) first under MLP and a knowledge-distilled sound-event encoder (CED) first under k-NN; speech encoders such as Whisper lead speech tasks but drop sharply on sound and music tasks, while sound-event encoders score zero on automatic speech recognition. These per-task and aggregate scores are the paper's evidence that no tested encoder is uniformly general and that a benchmark spanning domains is needed to reveal specialization.

Load-bearing premise

The whole ranking rests on treating scores from very different tasks as commensurable after rescaling each to a 0-1 range and weighting them by test-set size; if that rescaling is not a fair common currency, the headline orderings could change.

Editorial extensions

If this is right

  • If the benchmark is adopted, a new audio encoder can be positioned against existing ones by running one pipeline over 22 tasks, instead of re-implementing per-dataset evaluation scripts.
  • The large performance gaps between domains imply that a model trained on one audio domain cannot be assumed to transfer to others; claims of general audio understanding need evidence across speech, environmental sound, and music.
  • The disagreement between the MLP and k-NN rankings implies that reported representation quality should always state the readout protocol, since an encoder can be strong under linear probing but weak under raw nearest-neighbor geometry.
  • Because the aggregate score weights by test-set size, adding a task with a large but easy test set will shift rankings toward encoders that happen to do well on it, so the aggregate is only meaningful when the task mix and weights are reported alongside it.
  • The framework's uniform embedding interface and packaged datasets make it possible to compare new encoders without modifying task code, which the paper presents as a reproducibility and integration benefit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension the paper does not run: recompute both rankings under equal task weights, since Equation (2)'s test-set-size weighting is a policy choice, not a mathematical necessity, and the headline winners may shift.
  • The same pipeline could quantify information loss in discrete audio tokens by feeding tokenized reconstructions through the 22 tasks and comparing their scores with continuous embeddings, directly complementing discrete-token benchmarks.
  • Because the k-NN protocol operates on utterance-level embeddings, it may systematically handicap frame-level speech encoders; a frame-aligned or time-pooled k-NN variant would test whether the observed speech-encoder weakness is an artifact of pooling.
  • The benchmark could serve as a monitoring signal during pretraining: if a new training objective raises the aggregate score while lowering a specific domain's score, X-ARES would expose that trade-off in a way single-dataset evaluation cannot.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces X-ARES, an open-source benchmark for evaluating audio encoders across 22 tasks in speech, environmental sound, and music domains. It proposes two evaluation protocols: linear fine-tuning with an MLP head and unparameterized k-NN classification. The authors evaluate 13 publicly available audio encoders and report per-task scores (Figures 2 and 3) and a weighted average score per encoder (Table 2). The central claim is that X-ARES provides a comprehensive and standardized platform for comparing audio encoders, and that the evaluation reveals significant performance differences across tasks and domains. The paper also highlights practical contributions including WebDataset-based data loading, pre-packaged datasets on Zenodo, and a flexible TaskConfig system.

Significance. If the framework is sound, X-ARES fills a real gap: existing benchmarks such as HEAR, SUPERB, and DASB are either narrower in domain coverage or focused on discrete tokens. The open-source release, unified embedding interface, and dual evaluation paradigm are potentially useful to the community. The paper reports a broad comparative evaluation of 13 encoders, and the raw per-task results (Figures 2 and 3) are descriptively informative. However, the aggregate ranking in Table 2 rests on an unvalidated weighting scheme, and the absence of any uncertainty quantification weakens the claim of 'significant performance differences.' The authors also evaluate two of their own encoders without explicit disclosure, which is a self-evaluation risk. These issues are load-bearing for the quantitative conclusions, but they are addressable within the manuscript's scope.

major comments (4)
  1. [Section 5.2, Eq. (2)] The aggregate score S is defined as a test-set-size-weighted mean of normalized task metrics. Since every metric used in the paper (Acc, mAP, Segment-F1, iWER, Recall@1) already lies in [0,1] and the normalization in Eq. (1) uses the worst/best possible values, Eq. (1) is effectively the identity for these metrics. The only substantive aggregation decision is therefore the choice of weights n_i. Test-set sizes in Table 1 differ by orders of magnitude (e.g., FSD50K has on the order of 10,000 evaluation clips, while GTZAN and ESC-50 have only a few hundred), so a model that performs well on a single large test set can dominate the weighted mean regardless of its performance on dozens of smaller tasks. The paper provides no argument that test-set size encodes task importance, difficulty, or discriminative power. Because the ranks in Table 2 and the conclusion about 'significant performance differences' (Section 7) depend on this weighting, the authors should either justify the weights with a principled argument, or provide sensitivity analyses with unweighted averages, per-domain averages, or other aggregation schemes.
  2. [Section 6, Figures 2-3, Table 2] All results are reported as single point estimates with no variance, no number of random seeds, and no statistical significance tests. For example, in Table 2, ATST-Frame (0.622) and Whisper (0.646) differ by 0.024 on the MLP average, and many other adjacent scores are similarly close. Without repeated runs or significance testing, the claim of 'significant performance differences' among state-of-the-art encoders is unsupported. The paper should report results as mean plus/minus standard deviation over multiple seeds, and perform pairwise significance tests (e.g., paired bootstrap or Wilcoxon) on key comparisons.
  3. [Section 6, Table 2] Two of the evaluated encoders, Dasheng [1] and CED [40], are from the authors' own prior work, and Dasheng ranks first on the MLP average. This is not inherently problematic, but the paper never discloses this connection in the experiment section or discussion. The authors should explicitly state which models are their own, and discuss potential sources of bias such as checkpoint selection, hyperparameter tuning, or inadvertent advantages in the evaluation pipeline. Without this disclosure, readers may question the objectivity of the ranking.
  4. [Sections 3 and 6] The reproducibility of the benchmark depends critically on experimental details that are not given. The paper does not specify the MLP training hyperparameters (learning rate, number of epochs, hidden layer size, batch size, optimizer), the k-NN parameters (value of k, distance metric), or the exact train/validation/test split protocol for each of the 22 tasks, particularly for datasets like Clotho and FSD50K that have multiple official splits. Since one of the stated contributions is 'promoting reproducibility' (Section 1), this omission is load-bearing. The authors should provide a full setup description, ideally in an appendix or in the released code documentation.
minor comments (4)
  1. [Abstract and Section 7] Grammar errors: 'We introduces' in the Abstract and 'We presents' in Section 7 should be 'We introduce' and 'We present.'
  2. [Table 1 and Figure 3] There is an inconsistency between the set of tasks indicated as supporting k-NN in Table 1 and the tasks shown in Figure 3. For example, DESED and MAESTRO are marked as supporting k-NN in Table 1, but the k-NN heatmap in Figure 3 does not appear to include them, while it does include tasks such as Clotho that are not checked in Table 1. The authors should reconcile the table and figure.
  3. [Section 4 and Table 1] Several typos and formatting inconsistencies appear: 'V oxCeleb1' and 'V oxLingua33' have an extra space, 'identificaton' is misspelled, and 'classifation' appears in Section 4. The table's column formatting for 'Frame-level' and 'k-NN' is ambiguous in the rendered text.
  4. [Figures 2 and 3] The heatmaps show numeric values but no colorbar or legend explaining the color scale. Since the values are normalized scores, a colorbar would help interpret the magnitude. The figures are also hard to read in reproduced form due to overlapping task labels.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: X-ARES evaluates external encoder checkpoints on independently curated task datasets; the aggregation rule is an explicit design choice, not a quantity derived from the encoders' own outputs.

full rationale

The paper derives no predictive result from the encoders themselves. Equation (1) is a min-max normalization applied to externally defined metrics, and Equation (2) is an explicitly stated weighted average using test-set sizes; neither equation defines the metric in terms of any encoder's parameters or predictions, so the aggregate scores are not forced by construction. The benchmark tasks and datasets (LibriSpeech, ESC-50, GTZAN, etc.) are independent of the evaluated encoders, and the per-task scores in Figures 2 and 3 are computed by training a linear MLP or running k-NN on held-out test splits. The authors do evaluate their own models, Dasheng [1] and CED [40], and Dasheng ranks first on the MLP average; this is a self-evaluation risk that a reviewer might flag for fairness, but it is not circular reasoning because the ranking is produced by external task data and standard protocols rather than by importing the Dasheng paper's own claims. The only potentially debatable aspect is the choice of test-set-size weights in Equation (2), which could make the aggregate ranking sensitive to a few large test sets; however, that is a methodological validity or robustness concern, not a case of a prediction reducing to its input by definition. No load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation were found. The paper is therefore self-contained as an evaluation study, and the correct circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark's central claim depends on choices about what to evaluate and how to combine scores, rather than on free parameters or invented entities. The key premises are listed as domain assumptions.

assumptions (4)
  • domain assumption A linear-probe MLP and a k-NN classifier are sufficient probes of encoder representation quality.
    The entire benchmark rests on the premise that these two probes reveal generalizable encoder performance; Section 3.1 introduces them without validation against full fine-tuning.
  • domain assumption Weighted averaging across tasks by test-set size yields a meaningful single score.
    Equation (2) weights tasks by ni but does not account for task difficulty, metric scale, or class imbalance; this aggregation drives the headline rankings in Table 2.
  • domain assumption The selected 22 public datasets represent essential aspects and real-world scenarios of audio processing.
    The task selection in Table 1 is presented as comprehensive but is not justified by a formal coverage criterion.
  • domain assumption Frozen task-specific decoders (BERT and Qwen2.5-0.5B) are strong enough to expose encoder quality.
    Section 3.3 fixes these decoders; if a decoder dominates the loss or fails on some encoder outputs, comparisons are confounded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of X-ARES: A Comprehensive Framework for Assessing Audio Encoder Performance." pith.science (2026). https://pith.science/paper/CHVGOW5L

@misc{pith2026250516369,
  author       = {Pith},
  title        = {Pith review of: X-ARES: A Comprehensive Framework for Assessing Audio Encoder Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CHVGOW5L}},
  note         = {Machine review of arXiv:2505.16369}
}
read the original abstract

We introduces X-ARES (eXtensive Audio Representation and Evaluation Suite), a novel open-source benchmark designed to systematically assess audio encoder performance across diverse domains. By encompassing tasks spanning speech, environmental sounds, and music, X-ARES provides two evaluation approaches for evaluating audio representations: linear fine-tuning and unparameterized evaluation. The framework includes 22 distinct tasks that cover essential aspects of audio processing, from speech recognition and emotion detection to sound event classification and music genre identification. Our extensive evaluation of state-of-the-art audio encoders reveals significant performance variations across different tasks and domains, highlighting the complexity of general audio representation learning.

Figures

Figures reproduced from arXiv: 2505.16369 by the authors.

Figure 1
Figure 1. The proposed X-ARES framework. Users provide a single pretrained audioencoder, which outputs frame-level embeddings. Embeddings are evaluated using a fine-tuned MLP layer for clip- and frame-level tasks. Further a non-parameterized kNN algorithm is used to evaluate the quality of embeddings. For specialized tasks, pre-trained decoders are incorporated as task-specific components. 2.2. SUPERB: Speech processing Unive… view at source ↗
Figure 2
Figure 2. MLP evaluation results for each model and task, where higher is better. ATST-Frame [41], BYOL-A [42], Dasheng-base [1] and MSM￾MAE [42]. The averaged results are presented in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Probing Spatial Structure in Pretrained Audio Representations

    cs.SD 2026-06 unverdicted novelty 7.0 of 10

    Introduces SARL benchmark showing pretrained audio encoders encode source-level spatial factors more readily than room-level factors, with patterns shaped by input configuration and training paradigm.

Reference graph

Works this paper leans on

53 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    Introduction The field of audio representation learning has witnessed remark- able progress in recent years [1, 2, 3], driven by the increas- ing availability of audio data and advancements in deep learn- ing methodologies. Effective audio encoders, capable of trans- forming raw audio waveforms into meaningful representations, are crucial for a wide range...

    work page

  2. [40]

    Fma: A dataset for music analysis,

    M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “Fma: A dataset for music analysis,” arXiv preprint arXiv:1612.01840 , 2016

    arXiv 2016

  3. [2]

    We present X-ARES, a comprehensive benchmark suite that evaluates audio encoders across a diverse set of tasks span- ning speech, environmental sounds, and music domains

    work page

  4. [3]

    We introduce two complementary evaluation methodolo- gies: parameterized multilayer perceptron (MLP) and unpa- rameterized k-nearest neighbors (k-NN), providing a more nuanced assessment of encoder performance

    work page

  5. [4]

    We provide an extensive evaluation of state-of-the-art audio encoders using X-ARES, showing their relative strengths and weaknesses

    work page

  6. [5]

    We release X-ARES as an open-source toolkit, facilitating easy integration of new encoders and tasks, and promoting reproducibility in audio representation research

    work page

  7. [6]

    Related Work 2.1. HEAR: Holistic Evaluation of Audio Representations X-ARES is strongly inspired by the HEAR benchmark [7], which assesses audio representations across environmental sound and music tasks. While HEAR provides an excellent foundation, X- ARES introduces several enhancements: Unified performance evaluation Performance for frame- and clip-lev...

    arXiv 2025

  8. [7]

    Framework Design X-ARES 1 is designed as a flexible and extensible framework for evaluating audio encoders across a diverse range of tasks. 3.1. Overall Architecture The X-ARES framework, illustrated in Figure 1, offers an auto- mated pipeline to comprehensively evaluate pretrained audio en- coders. X-ARES employs two distinct evaluation methodologies: ML...

    work page

Show all 53 references
  1. [8]

    Speech tasks in X-ARES assess both linguistic content (e.g., speech content, word spotting) and paralinguistic features (e.g., emotion, speaker identity, accent)

    Task Categories An overview of X-ARES tasks across three fundamental audio domains: speech, environmental sounds, and music can be seen in Table 1. Speech tasks in X-ARES assess both linguistic content (e.g., speech content, word spotting) and paralinguistic features (e.g., em...

  2. [9]

    Task-Specific Metrics A summary of all metrics used in X-ARES is provided in Ta- ble 1

    Evaluation Metrics 5.1. Task-Specific Metrics A summary of all metrics used in X-ARES is provided in Ta- ble 1. Accuracy (Acc) is used for multi-class classification tasks across all domains, while Mean Average Precision (mAP) is used for multi-class multi-label classification...

  3. [10]

    First, speech encoders such as data2vec [3], HuBERT [35], wav2vec2- large [2], WavLM [36] and Whisper-base [37]

    Experiments and Results We run X-ARES over a plethora of publicly available audio- encoders, which can be categorized into three domains. First, speech encoders such as data2vec [3], HuBERT [35], wav2vec2- large [2], WavLM [36] and Whisper-base [37]. Second, sound encoders suc...

  4. [11]

    Conclusion We presents X-ARES, a comprehensive framework for evaluat- ing audio encoder performance that addresses critical limitations in existing benchmarks. By designing a diverse task set across speech, environmental sound, and music domains, and imple- menting both MLP an...

  5. [12]

    Scal- ing up masked audio encoder learning for general audio classifica- tion,

    H. Dinkel, Z. Yan, Y . Wang, J. Zhang, Y . Wang, and B. Wang, “Scal- ing up masked audio encoder learning for general audio classifica- tion,” in Interspeech 2024, 2024

    2024

  6. [13]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Advances in neural information processing systems , vol. 33, pp. 12 449–12 460, 2020

    2020

  7. [14]

    Data2vec: A general framework for self-supervised learning in speech, vision and language,

    A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “Data2vec: A general framework for self-supervised learning in speech, vision and language,” in International Conference on Ma- chine Learning. PMLR, 2022, pp. 1298–1312

    2022

  8. [15]

    Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,” arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  9. [16]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D ´efossez, L. Mazar ´e, M. Orsini, A. Royer, P. P ´erez, H. J ´egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,” Tech. Rep., 2024

    2024

  10. [17]

    A comparative study of discrete speech tokens for semantic-related tasks with large language models,

    D. Wang, M. Cui, D. Yang, X. Chen, and H. Meng, “A comparative study of discrete speech tokens for semantic-related tasks with large language models,” arXiv preprint arXiv:2411.08742, 2024

    2024 arXiv

  11. [18]

    HEAR: Holistic evaluation of audio representations,

    J. Turian, J. Shier, H. R. Khan, B. Raj, B. W. Schuller, C. J. Stein- metz, C. Malloy, G. Tzanetakis, G. Velarde, K. McNally et al. , “HEAR: Holistic evaluation of audio representations,” in NeurIPS 2021 Competitions and Demonstrations Track. PMLR, 2022, pp. 125–145

    2021

  12. [19]

    SUPERB: Speech processing universal performance benchmark,

    S.-w. Yang, P.-H. Chi, Y .-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y . Y . Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Linet al., “SUPERB: Speech processing universal performance benchmark,” Interspeech 2021, 2021

    2021

  13. [20]

    DASB–discrete audio and speech benchmark,

    P. Mousavi, L. Della Libera, J. Duret, A. Ploujnikov, C. Subakan, and M. Ravanelli, “DASB–discrete audio and speech benchmark,” arXiv preprint arXiv:2406.14294, 2024

    2024 arXiv

  14. [21]

    Webdataset: A library for efficient loading of large-scale datasets,

    A. Perlmutter, “Webdataset: A library for efficient loading of large-scale datasets,” https://github.com/webdataset/webdataset, 2021, accessed: [current date]. [Online]. Available: https: //github.com/webdataset/webdataset

    2021

  15. [22]

    Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation,

    W. Yu, S. Wang, X. Yang, X. Chen, X. Tian, J. Zhang, G. Sun, L. Lu, Y . Wang, and C. Zhang, “Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation,” arXiv preprint arXiv:2411.18138, 2024

    2024 arXiv

  16. [23]

    Auto- matic speaker verification spoofing and countermeasures challenge (asvspoof 2015) database,

    T. Kinnunen, Z. Wu, E. Nicholas Evans, and J. Yamagishi, “Auto- matic speaker verification spoofing and countermeasures challenge (asvspoof 2015) database,” 2018

    2015

  17. [24]

    Crema-d: Crowd-sourced emotional multimodal actors dataset,

    H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, “Crema-d: Crowd-sourced emotional multimodal actors dataset,” IEEE transactions on affective computing, 2014

    2014

  18. [25]

    Speech model pre-training for end-to-end spoken language under- standing,

    L. Lugosch, M. Ravanelli, P. Ignoto, V . S. Tomar, and Y . Bengio, “Speech model pre-training for end-to-end spoken language under- standing,” arXiv preprint arXiv:1904.03670, 2019

    1904 arXiv

  19. [26]

    Libricount, a dataset for speaker count estimation,

    F.-R. St ¨oter, S. Chakrabarty, E. Habets, and B. Edler, “Libricount, a dataset for speaker count estimation,” 2018

    2018

  20. [27]

    Librispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal process- ing, 2015

    2015

  21. [28]

    The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, mul- timodal set of facial and vocal expressions in north american en- glish,

    S. R. Livingstone and F. A. Russo, “The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, mul- timodal set of facial and vocal expressions in north american en- glish,” PloS one, vol. 13, no. 5, p. e0196391, 2018

    2018

  22. [29]

    V ocalsound: A dataset for improving human vocal sounds recognition,

    Y . Gong, J. Yu, and J. Glass, “V ocalsound: A dataset for improving human vocal sounds recognition,” in IEEE International Confer- ence on Acoustics, Speech and Signal Processing, 2022

    2022

  23. [30]

    Speech commands: A dataset for limited-vocabulary speech recognition,

    P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” arXiv preprint arXiv:1804.03209, 2018

    2018 arXiv

  24. [31]

    V oxceleb: Large-scale speaker verification in the wild,

    A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “V oxceleb: Large-scale speaker verification in the wild,” Computer Speech & Language, 2020

    2020

  25. [32]

    V oxlingua107: a dataset for spoken lan- guage recognition,

    J. Valk and T. Alum ¨ae, “V oxlingua107: a dataset for spoken lan- guage recognition,” in 2021 IEEE Spoken Language Technology Workshop (SLT), 2021

    2021

  26. [33]

    Clotho: An audio cap- tioning dataset,

    K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio cap- tioning dataset,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2020, pp. 736–740

    2020

  27. [34]

    Sound event detection in domestic environments with weakly labeled data and soundscape synthesis,

    N. Turpault, R. Serizel, A. P. Shah, and J. Salamon, “Sound event detection in domestic environments with weakly labeled data and soundscape synthesis,” inWorkshop on Detection and Classification of Acoustic Scenes and Events, 2019

    2019

  28. [35]

    Esc: Dataset for environmental sound classification,

    K. J. Piczak, “Esc: Dataset for environmental sound classification,” in Proceedings of the 23rd ACM international conference on Multi- media, 2015

    2015

  29. [36]

    General-purpose tagging of freesound audio with audioset labels: Task description, dataset, and baseline,

    E. Fonseca, M. Plakal, F. Font, D. P. Ellis, X. Favory, J. Pons, and X. Serra, “General-purpose tagging of freesound audio with audioset labels: Task description, dataset, and baseline,” arXiv preprint arXiv:1807.09902, 2018

    2018 arXiv

  30. [37]

    Fsd50k: an open dataset of human-labeled sound events,

    E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “Fsd50k: an open dataset of human-labeled sound events,” IEEE/ACM Transac- tions on Audio, Speech, and Language Processing, 2021

    2021

  31. [38]

    A dataset and taxonomy for urban sound research,

    J. Salamon, C. Jacoby, and J. P. Bello, “A dataset and taxonomy for urban sound research,” in Proceedings of the 22nd ACM interna- tional conference on Multimedia, 2014

    2014

  32. [39]

    V ocal imitation set: a dataset of vocally imitated sound events using the audioset ontol- ogy

    B. Kim, M. Ghei, B. Pardo, and Z. Duan, “V ocal imitation set: a dataset of vocally imitated sound events using the audioset ontol- ogy.” inDCASE, 2018, pp. 148–152

    2018

  33. [41]

    The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use,

    B. L. Sturm, “The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use,” arXiv preprint arXiv:1306.1461, 2013

    2013 arXiv

  34. [42]

    Enabling factorized piano music modeling and generation with the MAESTRO dataset,

    C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C.-Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” in International Conference on Learning Representations, 2019

    2019

  35. [43]

    Neural audio synthesis of musical notes with wavenet autoencoders,

    J. Engel, C. Resnick, A. Roberts, S. Dieleman, D. Eck, K. Si- monyan, and M. Norouzi, “Neural audio synthesis of musical notes with wavenet autoencoders,” 2017

    2017

  36. [44]

    Well-read stu- dents learn better: On the importance of pre-training compact mod- els,

    I. Turc, M.-W. Chang, K. Lee, and K. Toutanova, “Well-read stu- dents learn better: On the importance of pre-training compact mod- els,” arXiv preprint arXiv:1908.08962v2, 2019

    1908 arXiv

  37. [45]

    Qwen2. 5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Weiet al., “Qwen2. 5 technical report,”arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  38. [46]

    Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing , vol. 29, pp. 3451–3460, 2021

    2021

  39. [47]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Se- lected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  40. [48]

    Robust speech recognition via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

    2023

  41. [49]

    Beats: Audio pre-training with acoustic tokenizers,

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, and F. Wei, “Beats: Audio pre-training with acoustic tokenizers,”arXiv preprint arXiv:2212.09058, 2022

    2022 arXiv

  42. [50]

    Byol-s: Learning self-supervised speech representations by bootstrapping,

    G. Elbanna, N. Scheidwasser-Clow, M. Kegler, P. Beckmann, K. El Hajal, and M. Cernak, “Byol-s: Learning self-supervised speech representations by bootstrapping,” in HEAR: Holistic Eval- uation of Audio Representations. PMLR, 2022

    2022

  43. [51]

    Ced: Consis- tent ensemble distillation for audio tagging,

    H. Dinkel, Y . Wang, Z. Yan, J. Zhang, and Y . Wang, “Ced: Consis- tent ensemble distillation for audio tagging,” in IEEE International Conference on Acoustics, Speech and Signal Processing . IEEE, 2024, pp. 291–295

    2024

  44. [52]

    Self-supervised audio teacher-student transformer for both clip-level and frame-level tasks,

    X. Li, N. Shao, and X. Li, “Self-supervised audio teacher-student transformer for both clip-level and frame-level tasks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

    2024

  45. [53]

    Masked modeling duo: Learning representations by encouraging both networks to model the input,

    D. Niizumi, D. Takeuchi, Y . Ohishi, N. Harada, and K. Kashino, “Masked modeling duo: Learning representations by encouraging both networks to model the input,” in IEEE International Confer- ence on Acoustics, Speech and Signal Processing. IEEE, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.