Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

TouchASP: Elastic Automatic Speech Perception that Everyone Can Touch

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One trained speech model, pruned at inference, matches per-size models.

desk verdict Genuinely new eMoE idea and a solid data pipeline, but the 'train once, deploy elastically' claim is riding on a single run and an unreported sampling schedule. read the letter →

arxiv 2412.15622 v1 pith:TXILJREG submitted 2024-12-20 eess.AS cs.CLeess.SP

Xingchen Song , Chengdong Liang , Binbin Zhang , Pengshen Zhang , ZiYu Wang , Youcheng Ma , Menglong Xu , Lin Wang
show 4 more authors
Di Wu Fuping Pan Dinghao Zhou Zhendong Peng
This is my paper · ORCID
classification eess.AScs.CLeess.SP
keywords elasticmixtureofexpertsautomaticspeechrecognitionperceptionweaksupervisiondatapipelinemultilingualASRsoundeventdetectionedgedeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's central claim is that a speech perception model can be trained once and then served at many different parameter sizes without retraining. It introduces eMoE, a mixture-of-experts architecture where experts are arranged in nested groups and a group is selected randomly during training; at inference, choosing a smaller group removes parameters and computation. The same system also uses a one-million-hour weakly labelled corpus built by a two-transcriber consistency filter, and a multi-task decoder that outputs transcription, language, gender, age, emotion, and sound-event labels. On the SpeechIO Mandarin benchmarks, the single eMoE model reaches 2.45% character error rate at full size and 2.67% at the smallest pruned size, down from 4.98% for the earlier dense baseline. If the elasticity result holds, large speech models become deployable on edge devices at no extra training cost.

What carries the argument

The load-bearing mechanism is a nested power-of-two expert-grouping scheme. In each MoE layer, the $N$ independent experts are arranged so that group $g_1$ has $G$ experts, group $g_2$ has $2G$, and in general $g_i$ has $2^{i-1}G$, up to the full set of $M$ experts, with shared experts always present. Each group gets its own router, and training samples a group with probabilities that favour larger groups, so the larger configurations receive enough updates. Because each group is nested inside the next, parameters active in a small configuration are always part of every larger configuration; at inference, an application merely selects a group, giving an instantaneous accuracy-versus-parameter tradeoff. A second mechanism is the data-validation pipeline: two independently produced transcriptions of each web-audio clip are compared, and clips with word-error rate above 10 or phone-error rate above 5 are discarded, yielding 650K usable hours from 1.26M raw hours.

What would settle it

Train eMoE and a dedicated small MoE on the same data, log how often each nested group is selected, and compare CER on a held-out SpeechIO-style set for the smallest group size: if the smallest eMoE configuration underperforms the dedicated small model by more than the paper's observed margin of about 0.04 percentage points on 500K hours, the elasticity mechanism is not supported. A cheaper check is to inspect the training curve of the smallest group's router loss and see whether it converges to the same value as the full router.

Watch

Extended reading notes

Core claim

The paper claims that one eMoE model trained once produces inference configurations that are comparable to separately trained models of each configuration. Concretely, with 500K hours of data, a single model with $N=3$, $N=7$, or $N=15$ routed experts achieves 3.95%, 3.71%, and 3.68% CER respectively on SpeechIO, against 3.99% for a dedicated smaller model and 3.62% for a dedicated larger model at the two endpoints. With one million hours of data the same trained weights give 2.67% CER at the smallest configuration and 2.45% at the full configuration, and the multi-task extension retains competitive performance on multilingual recognition, language, gender, age, emotion, and sound-event tasks. The paper presents this as evidence that 'train once, deploy elastically' is a viable alternative to training a separate model per device tier.

Load-bearing premise

The claim collapses if a nested expert group sampled only occasionally during training does not learn to route and predict as well as a dedicated model trained exclusively at that size; the paper's evidence for this is a single comparison run, and the group sampling probabilities are not reported.

Editorial extensions

If this is right

  • A single TouchASP model can serve edge, mobile, and cloud deployments by switching expert groups, with no fine-tuning between sizes.
  • Scaling the weakly labelled web corpus from 160K to 1M hours lowers SpeechIO CER from 3.8% to 3.09% under the same architecture, so the two-transcriber consistency filter preserves enough quality to make data scale beneficial.
  • The multi-task token format lets one decoder handle transcription, language identification, gender and age classification, emotion recognition, and sound-event detection, removing the need for separate task-specific models.
  • At full size the model is competitive with strong published and commercial speech systems on the SpeechIO leaderboard, reaching 2.45% CER on that benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report the group sampling distribution or the number of updates each nested router receives; if a re-implementation samples small groups too rarely, the smallest deployment size would underperform a dedicated small model, so tracking per-group update counts is a testable design rule.
  • The elasticity claim is demonstrated with a single model run on one benchmark family; evaluating eMoE across languages, domains, and multiple seeds, and comparing every intermediate size against dedicated models, would strengthen it into a general recipe.
  • The 51.6% retention rate of the data pipeline implies that the discarded half of the raw audio may still contain learnable signal, and a self-training or noise-tolerant objective trained on all raw audio could potentially beat the cleaned-only result.
  • Independent routers for each nested group may make small configurations easier to train, since a router only ranks experts inside its own group; this points to a design where routing heads are explicitly conditioned on the target deployment size.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. TouchASP proposes three contributions: (1) an elastic mixture-of-experts (eMoE) architecture that nests expert groups and prunes them at inference time to support different deployment sizes from a single training run; (2) an unsupervised web-data pipeline that yields roughly 650k hours of usable speech data from 1.26M raw hours, combined with other sources into a 1M-hour training set; and (3) a multi-task 'Automatic Speech Perception' (ASP) framework that adds language, dialect, age, gender, emotion, and sound-event prediction to ASR. The paper reports that eMoE trained once at 1B scale matches separately trained DeepSeekMoE models at N=3 and N=15 experts (Table 4), improves SpeechIO CER from 4.98% to 2.45% (Table 5), and shows competitive or strong results on multilingual, dialect, LID, gender/age, SER, and SED benchmarks (Tables 7-9). The work is framed as a technical report aimed at practical, elastic deployment of speech perception models.

Significance. If the central eMoE claim holds, the paper offers a practically valuable result: a single training run could replace multiple differently sized models, reducing training cost and deployment complexity. The data pipeline, with its explicit retention rate and external SpeechIO evaluation, is a useful reference for large-scale weakly supervised speech data construction. The multi-task ASP formulation is also a reasonable extension of the ASR-only paradigm, and the use of external human-annotated benchmarks (SpeechIO, CommonVoice, MELD, ESC-50) is a strength. However, the current evidence for the eMoE claim rests on a single uncharacterized training configuration, and several evaluation protocols contain confounds or test-set overlap with training data, so the significance is conditional on these issues being resolved with additional experiments and transparent reporting.

major comments (4)
  1. [Section 4.2, Table 4] The central claim that a single eMoE training run matches separately trained DeepSeekMoE models at N=3, N=7, and N=15 is supported only by a single run with no variance or seed analysis, and the group-sampling schedule described in Section 3.1 is never specified. The paper says 'we set different sampling probabilities for different groups' but does not report these probabilities, the resulting per-group step counts, or the per-expert gradient counts induced by the nested groups. Since smaller groups are deliberately sampled less often, the N=3 result (CER 3.95 vs. DeepSeekMoE-350M 3.99) could reflect a hand-tuned schedule rather than a robust property of nested-group training. Additionally, the full-size comparison is already loose: eMoE at N=15 (CER 3.68) is worse than the dedicated DeepSeekMoE-1B (CER 3.62). To make the 'train once, deploy elastically' claim load-bearing, the authors should report the sampling distribution, per-group training statistics, and results across multiple seeds or at least provide error bars.
  2. [Section 4.1, Table 2] The data-scaling experiment compares 160K, 500K, and 1M hours of data by training for 100K steps in each condition. This holds the number of optimizer steps constant but changes the number of epochs per dataset, so the CER improvement (3.8 to 3.09) conflates data volume with training repetition and total unique tokens seen. If the batch size is fixed, the 160K-hour model sees each utterance more times than the 1M-hour model, making the comparison a compute-matched rather than a data-matched scaling study. The authors should either match total samples seen, report the number of epochs, or control for the effective training steps to isolate the effect of data volume.
  3. [Section 4.3, Table 9 and Table 6] The SER evaluation on MELD uses a test set that appears in the training data: Table 6 lists 'meld' among the SER training sources, and Table 9 reports TouchASP accuracy on the meld test set. Similarly, the gender test set (Aishell1, kws, kaggle) and age test set (kws, kaggle) overlap with the sources listed in Table 6 as training data (Aishell1, kaggle, wake-up data with gender/age labels). As a result, the reported accuracies are not directly comparable to baselines that did not train on these evaluation sets, and the claims of 'excellent recognition capability' for gender, age, and emotion are not supported by held-out evaluation. The authors should re-evaluate on datasets or splits that are disjoint from training, or clearly state the overlap and discount the comparisons accordingly.
  4. [Section 4.3, Table 7 and accompanying text] The text states that 'TouchASP significantly outperforms its counterparts, in most test sets with the exception of Librispeech,' but Table 7 does not support this. On wenetspeech test_net, TouchASP (5.52) is worse than eMoE-1B (5.32) and Seed-ASR (4.66); on test_meeting, TouchASP (5.94) is worse than eMoE-1B (5.42); and on Librispeech test_other, TouchASP (4.38) is worse than eMoE-1B (4.28). The only clear gains are on Cantonese and dialect sets, where several baselines have missing entries (dashes). The narrative should be rephrased to reflect the actual pattern: TouchASP improves dialect recognition while regressing on Mandarin and English relative to its own eMoE-1B base, which the paper does acknowledge later but the earlier claim is overstated.
minor comments (6)
  1. [Section 3.1] The group indexing is inconsistent: the text defines groups g1 through gZ, but later refers to 'group 0' and 'g0' without defining it. Please clarify whether the smallest group is g0 or g1, and adjust the notation consistently.
  2. [Section 3.1] There is an off-by-one ambiguity in the group definitions: 'g1 = (e0, e1, ..., eG)' suggests G+1 experts, while the text says the smallest group contains G experts. Also, '2^{Z-1} * G = M' should state whether M is the total number of non-shared experts or includes shared experts, since earlier M = S + N is defined.
  3. [Section 4.2, Table 4] The entry 'N=sample[3, 7, 15]' for eMoE training is unclear: it should specify whether sampling is over the group indices, the number of active experts, or the set of nested group sizes, and how K=3 interacts with the smallest group of size 3.
  4. [Section 4.1 and Table 2] The metric name is inconsistent: Section 4.1 and Table 2 use 'CER', but the text says 'evaluated using WER' and later 'weighted average WER' for SpeechIO. Please standardize the terminology to avoid confusing readers.
  5. [Section 4.3, Table 7] Several entries in Table 7 are missing (dashes) for Seed-ASR and eMoE-1B on the Cantonese and dialect test sets, which prevents a direct comparison of TouchASP against those baselines on the exact same data. Reporting the evaluation coverage or adding the missing numbers would strengthen the comparison.
  6. [Section 3.2] The filtering rule 'filter out data with WER greater than 10 and PER greater than 5' is ambiguous: it is not clear whether data are removed when both conditions hold or when either condition holds, nor how WER/PER are computed between the two ASR outputs when the reference is absent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central eMoE and data-pipeline claims are evaluated on external benchmarks and do not reduce to fitted inputs or self-citations.

full rationale

The derivation chain is self-contained. The data-pipeline evaluation (Section 4.1, Table 2) reports CER on the externally annotated SpeechIO test sets, so the claimed improvement from 160K to 1M hours is an external measurement, not a consequence of how the pipeline was defined. The central eMoE claim (Section 4.2, Table 4) compares a single dynamically trained eMoE model against separately trained DeepSeekMoE baselines at equal inference parameter sizes; the reported CER values are measured on SpeechIO, and no equation in Section 3.1 computes these CER values from the group-sampling probabilities. The sampling probabilities are not fitted to the Table 4 outcomes, and the paper does not define the N=3, N=7, or N=15 results in terms of the training objective. Self-citations to U2++ MoE (Ref. 10) and TouchTTS (Ref. 11) are used as baselines or supporting references, but they are not load-bearing for the main eMoE or ASP claims. The unstated sampling schedule and single-run results are reproducibility and robustness concerns, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities beyond the eMoE architecture, which is a model design rather than an invented entity. The core claims rest on standard background assumptions about MoE architectures, pseudo-label quality, and benchmark validity.

free parameters (4)
  • Group sizes (G and Z) = G=4 (or 3 per Table 4); groups N=3/7/15
    Chosen by hand to give roughly doubling capacities; not tuned systematically and the paper's notation is inconsistent.
  • Group sampling probabilities = Not reported
    Authors state smaller groups get lower sampling probabilities but never give the distribution; this determines relative training effort across groups.
  • Filtering thresholds WER/PER = WER > 10, PER > 5
    Hand-set thresholds in the data pipeline that define which pseudo-labeled clips survive; no sensitivity analysis is provided.
  • Top-K routing count = K=3
    Fixed routing count for all expert groups in the experiments; chosen without ablations.
assumptions (4)
  • domain assumption DeepSeekMoE with shared experts and fine-grained experts is a strong ASR backbone
    Taken from prior work [4]; no independent evidence in this paper.
  • domain assumption Agreement between two ASR systems implies high-quality transcription
    The Rover filter assumes consistent transcripts are correct; potential correlated errors between WhisperX and Paraformer are not quantified.
  • ad hoc to paper Nested expert groups with independent routers can be jointly trained to preserve per-group accuracy
    This is the core premise of eMoE; the paper only provides one set of runs as support.
  • domain assumption Weighted average WER on 26 SpeechIO sets is a valid proxy for general Mandarin ASR quality
    SpeechIO test sets are human-transcribed but the aggregation and metric naming are the authors' choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TouchASP: Elastic Automatic Speech Perception that Everyone Can Touch." pith.science (2026). https://pith.science/paper/TXILJREG

@misc{pith2026241215622,
  author       = {Pith},
  title        = {Pith review of: TouchASP: Elastic Automatic Speech Perception that Everyone Can Touch},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TXILJREG}},
  note         = {Machine review of arXiv:2412.15622}
}
read the original abstract

Large Automatic Speech Recognition (ASR) models demand a vast number of parameters, copious amounts of data, and significant computational resources during the training process. However, such models can merely be deployed on high-compute cloud platforms and are only capable of performing speech recognition tasks. This leads to high costs and restricted capabilities. In this report, we initially propose the elastic mixture of the expert (eMoE) model. This model can be trained just once and then be elastically scaled in accordance with deployment requirements. Secondly, we devise an unsupervised data creation and validation procedure and gather millions of hours of audio data from diverse domains for training. Using these two techniques, our system achieves elastic deployment capabilities while reducing the Character Error Rate (CER) on the SpeechIO testsets from 4.98\% to 2.45\%. Thirdly, our model is not only competent in Mandarin speech recognition but also proficient in multilingual, multi-dialect, emotion, gender, and sound event perception. We refer to this as Automatic Speech Perception (ASP), and the perception results are presented in the experimental section.

Figures

Figures reproduced from arXiv: 2412.15622 by the authors.

Figure 1
Figure 1. TouchASP: Train once, Elastic inference with multiple speech perception capabilities [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of Elastic MoE (eMoE). Subfigure (a) showcases the conventional top-K routing [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of Our Data Pipeline obtain data from various fields of videos and audiobooks from the internet, including TV dramas, interviews, commentaries, documentaries, technology, games, sports, arts, cuisine, etc. This results in a very diverse dataset covering a broad distribution of audio from many different recording setups, speakers, environments, and styles. This diversity in audio and fields helps our model t… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Overview of model structure and multi-task training. The encoder-decoder model is trained [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Typical test sets CER on SpeechIO when data scales. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia

    cs.SD 2025-01 conditional novelty 5.0 of 10

    An open, resource-lean speech understanding LLM trained on 50,500 hours matches or beats larger industry models on several Chinese benchmarks, with caveats in its internal evaluation.

Reference graph

Works this paper leans on

45 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017

    arXiv 2017

  2. [2]

    Gshard: Scaling giant models with condi- tional computation and automatic sharding

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020

    arXiv 2006

  3. [3]

    Mixtral of experts

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024

    arXiv 2024

  4. [4]

    Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.06066, 2024

    arXiv 2024

  5. [5]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024

    arXiv 2024

  6. [6]

    Dense training, sparse inference: Rethinking training of mixture- of-experts language models

    Bowen Pan, Yikang Shen, Haokun Liu, Mayank Mishra, Gaoyuan Zhang, Aude Oliva, Colin Raffel, and Rameswar Panda. Dense training, sparse inference: Rethinking training of mixture- of-experts language models. arXiv preprint arXiv:2404.05567, 2024

    arXiv 2024

  7. [7]

    Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts

    Zhao You, Shulin Feng, Dan Su, and Dong Yu. Speechmoe: Scaling to large acoustic models with dynamic routing mixture of experts. arXiv preprint arXiv:2105.03036, 2021

    arXiv 2021

  8. [8]

    Mixture-of-Expert Conformer for Streaming Multilingual ASR

    Ke Hu, Bo Li, Tara N Sainath, Yu Zhang, and Francoise Beaufays. Mixture-of-expert conformer for streaming multilingual asr. arXiv preprint arXiv:2305.15663, 2023

    work page Pith review arXiv 2023

Show all 45 references
  1. [9]

    Language-routing mixture of experts for multilingual and code-switching speech recognition

    Wenxuan Wang, Guodong Ma, Yuke Li, and Binbin Du. Language-routing mixture of experts for multilingual and code-switching speech recognition. arXiv preprint arXiv:2307.05956, 2023

    2023 arXiv

  2. [10]

    U2++ moe: Scaling 4.7 x parameters with minimal impact on rtf

    Xingchen Song, Di Wu, Binbin Zhang, Dinghao Zhou, Zhendong Peng, Bo Dang, Fuping Pan, and Chao Yang. U2++ moe: Scaling 4.7 x parameters with minimal impact on rtf. arXiv preprint arXiv:2404.16407, 2024

    2024 arXiv

  3. [11]

    Touchtts: An embarrassingly simple tts framework that everyone can touch

    Xingchen Song, Mengtao Xing, Changwei Ma, Shengqiang Li, Di Wu, Binbin Zhang, Fuping Pan, Dinghao Zhou, Yuekai Zhang, Shun Lei, et al. Touchtts: An embarrassingly simple tts framework that everyone can touch. arXiv preprint arXiv:2412.08237, 2024

    2024 arXiv

  4. [12]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR, 2023

    2023

  5. [13]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  6. [14]

    Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio

    Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106.06909, 2021

    2021 arXiv

  7. [15]

    Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition

    Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, et al. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and ...

    2022

  8. [16]

    Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model benchmark

    Linhan Ma, Dake Guo, Kun Song, Yuepeng Jiang, Shuai Wang, Liumeng Xue, Weiming Xu, Huan Zhao, Binbin Zhang, and Lei Xie. Wenetspeech4tts: A 12,800-hour mandarin tts corpus for large speech generation model benchmark. arXiv preprint arXiv:2406.05763, 2024. 12

    2024 arXiv

  9. [17]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation

    Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. arXiv preprint arXiv:2407.05361, 2024

    2024 arXiv

  10. [18]

    Google usm: Scaling automatic speech recognition beyond 100 languages

    Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, et al. Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037, 2023

    2023 arXiv

  11. [19]

    Scaling speech technology to 1,000+ languages

    Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. Scaling speech technology to 1,000+ languages. Journal of Machine Learning Research, 25(97):1–52, 2024

    2024

  12. [20]

    Seed-asr: Understanding diverse speech and contexts with llm-based speech recognition

    Ye Bai, Jingping Chen, Jitong Chen, Wei Chen, Zhuo Chen, Chuang Ding, Linhao Dong, Qianqian Dong, Yujiao Du, Kepan Gao, et al. Seed-asr: Understanding diverse speech and contexts with llm-based speech recognition. arXiv preprint arXiv:2407.04675, 2024

    2024 arXiv

  13. [21]

    Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms

    Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, et al. Funaudiollm: V oice understanding and generation foundation models for natural interaction between humans and llms. arXiv preprint arXiv:2407.04051, 2024

    2024 arXiv

  14. [22]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  15. [23]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  16. [24]

    Slimmable neural networks

    Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint arXiv:1812.08928, 2018

    2018 arXiv

  17. [25]

    Whisperx: Time-accurate speech transcription of long-form audio

    Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. Whisperx: Time-accurate speech transcription of long-form audio. INTERSPEECH 2023, 2023

    2023

  18. [26]

    Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition

    Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Yan. Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition. arXiv preprint arXiv:2206.08317, 2022

    2022 arXiv

  19. [27]

    Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit

    Zhuoyuan Yao, Di Wu, Xiong Wang, Binbin Zhang, Fan Yu, Chao Yang, Zhendong Peng, Xiaoyu Chen, Lei Xie, and Xin Lei. Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit. arXiv preprint arXiv:2102.01547, 2021

    2021 arXiv

  20. [28]

    Common voice: A massively-multilingual speech corpus

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019

    1912 arXiv

  21. [29]

    Automatic speech recognition datasets in cantonese: A survey and new dataset

    Tiezheng Yu, Rita Frieske, Peng Xu, Samuel Cahyawijaya, Cheuk Tung Shadow Yiu, Holy Lovenia, Wenliang Dai, Elham J Barezi, Qifeng Chen, Xiaojuan Ma, et al. Automatic speech recognition datasets in cantonese: A survey and new dataset. arXiv preprint arXiv:2201.02419, 2022

    2022 arXiv

  22. [30]

    Minspeech: A corpus of southern min dialect for automatic speech recognition

    Jiayan Lin, Shenghui Lu, Hukai Huang, Wenhao Guan, Binbin Xu, Hui Bu, Qingyang Hong, and Lin Li. Minspeech: A corpus of southern min dialect for automatic speech recognition. In Proc. Interspeech 2024, pages 2330–2334, 2024

    2024

  23. [31]

    Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline

    Hui Bu, Jiayu Du, Xingyu Na, Bengu Wu, and Hao Zheng. Aishell-1: An open-source mandarin speech corpus and a speech recognition baseline. In2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech I/O systems and a...

    2017

  24. [32]

    Librispeech: an asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE, 2015. 13

    2015

  25. [33]

    Kespeech: An open source speech dataset of mandarin and its eight subdialects

    Zhiyuan Tang, Dong Wang, Yanguang Xu, Jianwei Sun, Xiaoning Lei, Shuaijiang Zhao, Cheng Wen, Xingjun Tan, Chuandong Xie, Shuran Zhou, et al. Kespeech: An open source speech dataset of mandarin and its eight subdialects. In Thirty-fifth Conference on Neural Information Processi...

    2021

  26. [34]

    M3ed: Multi-modal multi-scene multi-label emotional dialogue database

    Jinming Zhao, Tenggan Zhang, Jingwen Hu, Yuchen Liu, Qin Jin, Xinchao Wang, and Haizhou Li. M3ed: Multi-modal multi-scene multi-label emotional dialogue database. arXiv preprint arXiv:2205.10237, 2022

    2022 arXiv

  27. [35]

    Meld: A multimodal multi-party dataset for emotion recognition in conversa- tions

    Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversa- tions. arXiv preprint arXiv:1810.02508, 2018

    2018 arXiv

  28. [36]

    Iemocap: Interactive emotional dyadic motion capture database

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335–359, 2008

    2008

  29. [37]

    Recog- nizing emotional speech in persian: A validated database of persian emotional speech (persian esd)

    Niloofar Keshtiari, Michael Kuhlmann, Moharram Eslami, and Gisela Klann-Delius. Recog- nizing emotional speech in persian: A validated database of persian emotional speech (persian esd). Behavior research methods, 47:275–294, 2015

    2015

  30. [38]

    The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english

    Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5):e0196391, 2018

    2018

  31. [39]

    Esc: Dataset for environmental sound classification

    Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015–1018, 2015

    2015

  32. [40]

    Fsd50k: an open dataset of human-labeled sound events

    Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829–852, 2021

    2021

  33. [41]

    Emobox: Multilingual multi-corpus speech emotion recognition toolkit and benchmark

    Ziyang Ma, Mingjie Chen, Hezhao Zhang, Zhisheng Zheng, Wenxi Chen, Xiquan Li, Jiaxin Ye, Xie Chen, and Thomas Hain. Emobox: Multilingual multi-corpus speech emotion recognition toolkit and benchmark. arXiv preprint arXiv:2406.07162, 2024

    2024 arXiv

  34. [42]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition

    Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880–2894, 2020

    2020

  35. [43]

    Masked spectrogram modeling using masked autoencoders for learning general-purpose audio representation

    Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Masked spectrogram modeling using masked autoencoders for learning general-purpose audio representation. In HEAR: Holistic Evaluation of Audio Representations, pages 1–24. PMLR, 2022

    2022

  36. [44]

    Ssast: Self-supervised audio spectrogram transformer

    Yuan Gong, Cheng-I Lai, Yu-An Chung, and James Glass. Ssast: Self-supervised audio spectrogram transformer. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 10699–10709, 2022

    2022

  37. [45]

    Beats: Audio pre-training with acoustic tokenizers

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058, 2022. 14

    2022 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.