Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Metis: A Foundation Speech Generation Model with Masked Generative Pre-training

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Metis claims that masked generative pre-training on 300K hours of unlabeled speech creates a foundation model that fine-tunes to beat task-specific systems on five speech-generation tasks.

desk verdict The pre-training recipe is real and the TTS ablation is convincing, but the SOTA claim is overstated and the voice conversion protocol measures reconstruction rather than conversion. read the letter →

arxiv 2502.03128 v1 pith:CMNEBIVH submitted 2025-02-05 cs.SD cs.AIcs.LGeess.ASeess.SP

Yuancheng Wang , Jiachen Zheng , Junan Zhang , Xueyao Zhang , Huan Liao , Zhizheng Wu This is my paper · ORCID
classification cs.SDcs.AIcs.LGeess.ASeess.SP
keywords speechgenerationmaskedgenerativemodelself-supervisedlearningtokenspre-trainingzero-shottext-to-speechvoiceconversionenhancementfoundation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a single speech-generation model can be pre-trained on large amounts of unlabeled speech and then fine-tuned, with very little task-specific data or trainable parameters, to outperform models built specifically for each individual task. The central claim is that masked generative pre-training on self-supervised speech tokens is the ingredient that makes this transfer work. If true, the field could move away from building separate expert models for every speech-generation task, instead sharing one set of speech priors across text-to-speech, voice conversion, speech enhancement, target speaker extraction, and lip-to-speech. The paper also claims the same pre-trained weights support multimodal inputs, including text, audio, and video.

What would settle it

Replace the voice-conversion condition with a different speaker’s utterance while keeping the prompt from the target speaker, and measure SIM and WER on VCTK; if SIM drops toward the 0.38 baseline level, the voice-conversion claim is refuted.

Watch

Extended reading notes

Load-bearing premise

The voice-conversion result rests on a setup where the model is asked to rebuild the target utterance from a timbre-altered copy of that same utterance, so if that setup does not reflect genuine cross-speaker conversion, the reported voice-conversion advantage would not hold.

Editorial extensions

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper’s voice-conversion evaluation conditions on w2v-bert-2.0 features of a timbre-perturbed copy of the target utterance, with a random prefix of that same target as the prompt; this is closer to reconstructing the target from a transformed version of itself than to converting a different source speaker’s utterance, so the reported SIM advantage may not transfer to a standard cross-speaker V
  • The paper ablates pre-training only for zero-shot TTS (the “w.o. pre-train” row); extending that ablation to all five tasks would test whether pre-training is the causal factor everywhere or whether some tasks benefit mostly from the two-stage token design.
  • If the pre-trained SSL-token prior is genuinely task-agnostic, a similar pre-train-and-fine-tune recipe might transfer to music or general audio generation, provided a unified audio tokenizer can be learned.
  • The strong results on degraded-input tasks (enhancement and extraction) suggest the pre-trained model may have learned a clean-speech prior that helps it “fill in” missing or corrupted tokens; this could be tested by degrading the prompt or condition at inference and observing whether output quality degrades more gracefully than in non-pre-trained baselines.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Metis, a two-stage speech generation framework built on masked generative pre-training over SSL tokens derived from w2v-bert-2.0, followed by task-specific fine-tuning and a shared masked generative acoustic decoder. The authors pre-train on 300K hours of unlabeled multilingual speech and then fine-tune the model for zero-shot TTS, voice conversion, target speaker extraction, speech enhancement, and lip-to-speech, using either full fine-tuning or LoRA. The paper claims that Metis outperforms state-of-the-art task-specific and multi-task systems across all five tasks while using far less training data and, in some configurations, fewer than 20M trainable parameters. The main evidence for the pre-training benefit is a 'without pre-training' control in the zero-shot TTS experiments, where the pre-trained model converges faster and reaches better WER than the non-pre-trained baseline.

Significance. If the central claim is supported, the paper would make a useful contribution by demonstrating that a single masked generative pre-training objective over semantic SSL tokens can be adapted efficiently to several speech generation tasks, including multimodal lip-to-speech. The zero-shot TTS results on 1K-10K hours with a direct 'without pre-training' comparison are a meaningful piece of evidence, and the release of code and checkpoints would facilitate reproducibility. However, the headline claim that Metis 'outperforms state-of-the-art' across all five tasks is not supported by the tables: on several headline metrics Metis is worse than a strong baseline, and the voice conversion evaluation appears to measure same-utterance reconstruction rather than standard cross-utterance conversion. The paper's core idea is not undermined by these issues, but the evidence needs to be realigned with the claims before publication.

major comments (4)
  1. [§4.2.2, Table 2] The voice conversion evaluation does not follow the standard cross-utterance voice conversion protocol. The conditioning input is the w2v-bert-2.0 features of the timbre-perturbed target speech, and the prompt is a prefix of the same target speech. This means the model is asked to reconstruct the target utterance from a perturbed copy of itself plus its own prefix, rather than to convert the content of a separate source utterance to a target speaker's voice. The baseline systems (Vevo, LM-VC, UniAudio) are evaluated on cross-utterance VC, so the reported SIM 0.55 vs 0.38 and WER 4.49 vs 3.48 are not comparable. The evaluation description at the end of §4.2.2 is also ambiguous: 'we randomly select 200 samples from the dataset as source speech, and for each sample, we randomly select another sample from the same speaker as the prompt speech' does not specify whether the prompt is the target speaker or the source speaker. This directly undermines the abstract's claim of outperforming state-of-the-art on voice conversion. Please re-run the evaluation on a standard protocol with content from a source utterance and speaker identity from a separate reference utterance, ideally from a different speaker, and report those results.
  2. [Abstract and §4.2 results (Tables 1, 2, 3, 7)] The statement that 'Metis outperforms state-of-the-art task-specific or multi-task systems across five speech generation tasks' is contradicted by the tables on several headline metrics. On LibriSpeech test-clean (Table 7), Metis-TTS has WER 4.33 vs NaturalSpeech 3's 1.94 and VoiceBox's 2.03. On voice conversion (Table 2), Metis-VC LoRA 16 has WER 4.49 vs Vevo's 3.48. On target speaker extraction (Table 3), Metis-TSE fine-tune has WER 6.31 vs WeSep's 6.19. The paper should either restrict the SOTA claim to the specific metrics and conditions where it holds, or define an explicit aggregate criterion (e.g., a task-level utility function over similarity, intelligibility, and quality) that justifies the unqualified claim.
  3. [§4.2.1 vs §4.2.2-4.2.5] The only reported 'without pre-training' control is for zero-shot TTS (Table 1, 'Metis-TTS w.o. pre-train'). No such control is reported for voice conversion, target speaker extraction, speech enhancement, or lip-to-speech. Since the central claim is that masked generative pre-training provides the foundation benefit, the evidence for these four tasks cannot distinguish pre-training gains from gains due to the architecture, the SSL-token conditioning, or the fine-tuning setup. Please add 'without pre-training' results for at least one additional task (e.g., speech enhancement or target speaker extraction), or alternatively restrict the foundation-model claim to zero-shot TTS and clearly scope the claim for the other tasks.
  4. [Abstract and Table 1] The abstract claims 'fewer than 20M trainable parameters or 300 times less training data,' but the numbers in the paper do not uniformly support this. Table 1 reports Metis-TTS LoRA 32 with 32M trainable parameters, exceeding 20M. For voice conversion, the fine-tune uses 0.4K hours vs Vevo's 60K hours, which is 150 times less, not 300. The '300 times less' figure is not traced to any specific task in the main text. Please correct the numbers or qualify the claim with 'for some tasks and configurations.'
minor comments (5)
  1. [Abstract] There is a typo: 'Audio samples are are available' should be 'Audio samples are available.'
  2. [§4.2.2] The model name 'HireSpeech++' appears to be a typo for 'HierSpeech++' in the text.
  3. [Appendix E] The sentence 'We show more details in Appendix E' appears inside Appendix E itself; it should be removed or reworded.
  4. [Table captions] The caption text 'The best and the second best result is shown in bold and by underlined' should be corrected to 'The best and second-best results are shown in bold and underlined,' respectively.
  5. [Appendix G] The Limitation section lists only two directions (unified audio representation and few-shot task learning) and does not mention the known evaluations gaps, such as the voice conversion protocol and the lack of non-pre-trained controls for most tasks; the authors should acknowledge these limitations explicitly.

Circularity Check

1 steps flagged · score 6.0 of 10

Voice conversion 'SOTA' rests on a same-utterance reconstruction protocol; the VC prediction reduces by construction to its own input.

  1. self definitional [Section 4.2.2 (Voice Conversion), Implementation Details and Evaluation and Baseline; Tables 2 and 8]
    "The perturbed speech features are then used as input to predict the target speech based on the prompt. Specifically, we employ the w2v-bert-2.0 features of the perturbed speech as conditioning inputs and randomly extract a prefix of the target speech as the prompt. ... we randomly select 200 samples from the dataset as source speech, and for each sample, we randomly select another sample from the same speaker as the prompt speech."

    The conditioning input is, by construction, a timbre-perturbed copy of the target utterance, and the prompt is a prefix of that same target utterance. The content to be generated is therefore identical to the content already present in the input; the model only restores the original timbre from the prompt. This is same-utterance reconstruction, not cross-speaker or cross-utterance voice conversion. The reported SIM/WER comparison against Vevo, LM-VC, and UniAudio is not comparable, since those baselines convert a different source utterance to a target speaker. The evaluation additionally pairs the source and prompt from the same speaker, removing any speaker change. The claimed VC superiority is thus built into the task definition rather than demonstrated as voice conversion.

full rationale

Metis's central pre-training claim is otherwise supported by an internal 'w.o. pre-train' control for zero-shot TTS and by external baselines on TTS, SE, TSE, and L2S, so the foundation-model idea is not circular. However, one of the five headline tasks, voice conversion, uses a protocol in which the condition is a perturbed version of the target itself and the prompt is a prefix of the target, making the 'conversion' a reconstruction of the input utterance. The table's SIM/WER numbers and the abstract's SOTA claim for VC therefore reduce to the task's self-definitional setup. The reliance on the authors' own MaskGCT as architecture source and baseline is not load-bearing: MaskGCT is open-source and the ablation row 'Metis-TTS w.o. pre-train' directly isolates the pre-training effect.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or abstract entities; it combines existing representations (SSL tokens, acoustic tokens), a known architecture (MaskGCT), and standard fine-tuning techniques. The free parameters listed are hand-chosen hyperparameters that define the pre-training objective, the multi-task sampling, and the degradation simulation.

free parameters (7)
  • Prompt probability p = 0.8
    Probability of using a prefix prompt during pre-training (Section 3.4). Chosen by hand; affects in-context learning ability.
  • Prompt length range = [0%, 40%] of sequence
    Uniformly sampled prompt length during pre-training (Section 4.1 Training). Chosen by hand.
  • Mask schedule gamma(t) = sin(pi*t/(2T))
    Masking ratio schedule from MaskGIT (Section 3.1). Standard choice, but a design parameter.
  • Multi-task task proportions = {0.5, 0.1, 0.2, 0.2}
    Sampling proportions for TTS, VC, TSE, SE in Metis-Omni (Appendix E). Chosen by hand.
  • Degradation probabilities for SE = noise p=0.9, reverb p=0.35, bandwidth p=0.25; SNR -5 to 20 dB
    Simulation distribution for speech enhancement training (Section 4.2.4). Chosen by hand.
  • LoRA ranks = 4, 16, 32
    Low-rank adaptation ranks for fine-tuning; parameter counts 2M, 9M, 18M. Chosen per task.
  • Classifier-free guidance scale = cfg=2.0 (VC)
    Inference-time guidance for prompted tasks (Section 4.2.2). Chosen manually.
assumptions (5)
  • domain assumption Most speech generation tasks decompose into two stages: condition to SSL tokens, then SSL tokens to acoustic tokens.
    Stated in Section 1 and 3.2 as the basis for the unified framework. If false, a single pre-trained SSL model cannot serve all tasks.
  • domain assumption SSL tokens from VQ-quantized w2v-bert-2.0 features preserve sufficient semantic and prosodic information for downstream generation.
    Section 3.3 and Appendix A. The paper does not measure reconstruction fidelity of these tokens.
  • domain assumption Masked generative pre-training on unlabeled speech transfers to downstream tasks through fine-tuning.
    This is the core hypothesis of the paper; supported by the w.o. pre-train ablation but still an empirical assumption.
  • domain assumption The SSL-to-acoustic MGM decoder is task-independent and works for all five tasks.
    Section 3.6 describes the decoder but gives no training details or per-task validation.
  • domain assumption w2v-bert-2.0 features of degraded or mixed speech are sufficient conditions for enhancement and extraction tasks.
    Used as condition in TSE (Section 4.2.3), SE (4.2.4), VC (4.2.2). No analysis of information sufficiency.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Metis: A Foundation Speech Generation Model with Masked Generative Pre-training." pith.science (2026). https://pith.science/paper/CMNEBIVH

@misc{pith2026250203128,
  author       = {Pith},
  title        = {Pith review of: Metis: A Foundation Speech Generation Model with Masked Generative Pre-training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CMNEBIVH}},
  note         = {Machine review of arXiv:2502.03128}
}
read the original abstract

We introduce Metis, a foundation model for unified speech generation. Unlike previous task-specific or multi-task models, Metis follows a pre-training and fine-tuning paradigm. It is pre-trained on large-scale unlabeled speech data using masked generative modeling and then fine-tuned to adapt to diverse speech generation tasks. Specifically, 1) Metis utilizes two discrete speech representations: SSL tokens derived from speech self-supervised learning (SSL) features, and acoustic tokens directly quantized from waveforms. 2) Metis performs masked generative pre-training on SSL tokens, utilizing 300K hours of diverse speech data, without any additional condition. 3) Through fine-tuning with task-specific conditions, Metis achieves efficient adaptation to various speech generation tasks while supporting multimodal input, even when using limited data and trainable parameters. Experiments demonstrate that Metis can serve as a foundation model for unified speech generation: Metis outperforms state-of-the-art task-specific or multi-task systems across five speech generation tasks, including zero-shot text-to-speech, voice conversion, target speaker extraction, speech enhancement, and lip-to-speech, even with fewer than 20M trainable parameters or 300 times less training data. Audio samples are are available at https://metis-demo.github.io/.

Figures

Figures reproduced from arXiv: 2502.03128 by the authors.

Figure 1
Figure 1. An illustration of Metis. (a) provides an overview of the two-stage speech generation framework, which consists [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Two discrete speech representations for the two [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DELTA-TTS: Adapting Autoregressive Model into Diffusion Language Model for Text-to-Speech

    eess.AS 2026-07 conditional novelty 6.5 of 10

    A LoRA-plus-convolution adaptation converts an AR TTS backbone into a confidence-ordered discrete diffusion model that improves WER and speed on limited data.

  2. Anysynth:Zero-Shot Instrument Cloning via In-Context Learning and Asymmetric Hierarchical Guidance

    cs.SD 2026-07 conditional novelty 6.0 of 10

    A zero-shot instrument cloning system feeds raw reference audio into a flow-matching DiT and uses asymmetric CFG to keep melody and timbre control separate.

  3. GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model

    eess.AS 2025-12 conditional novelty 5.0 of 10

    A two-stage decoder-only language model with continuous embeddings and UTMOS-based preference fine-tuning reports improved target-speaker-extraction scores on Libri2Mix.

  4. SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipeline

    eess.AS 2025-05 conditional novelty 5.0 of 10

    A cascaded pipeline of audio compression, latent diffusion extraction, and generative correction achieves state-of-the-art target speech extraction quality and intelligibility on Libri2Mix and out-of-domain data.

Reference graph

Works this paper leans on

105 extracted references · 13 canonical work pages · cited by 4 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Seed-tts: A family of high-quality versatile speech generation models

    Anastassiou, P., Chen, J., Chen, J., Chen, Y., Chen, Z., Chen, Z., Cong, J., Deng, L., Ding, C., Gao, L., et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.02430, 2024 a

    arXiv 2024

  4. [4]

    Voiceshop: A unified speech-to-speech framework for identity-preserving zero-shot voice editing

    Anastassiou, P., Tang, Z., Peng, K., Jia, D., Li, J., Tu, M., Wang, Y., Wang, Y., and Ma, M. Voiceshop: A unified speech-to-speech framework for identity-preserving zero-shot voice editing. arXiv preprint arXiv:2404.06674, 2024 b

    arXiv 2024

  5. [5]

    M., and Weber, G

    Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M., and Weber, G. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019

    arXiv 1912

  6. [6]

    Voice conversion with just nearest neighbors

    Baas, M., van Niekerk, B., and Kamper, H. Voice conversion with just nearest neighbors. arXiv preprint arXiv:2305.18975, 2023

    arXiv 2023

  7. [7]

    Beit: Bert pre-training of image transformers

    Bao, H., Dong, L., Piao, S., and Wei, F. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021

    arXiv 2021

  8. [8]

    Better speech synthesis through scaling

    Betker, J. Better speech synthesis through scaling. arXiv preprint arXiv:2305.07243, 2023

    arXiv 2023

Show all 105 references
  1. [9]

    Audiolm: a language modeling approach to audio generation

    Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M., et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 31: 0 25...

    2023

  2. [10]

    Soundstorm: Efficient parallel audio generation

    Borsos, Z., Sharifi, M., Vincent, D., Kharitonov, E., Zeghidour, N., and Tagliasacchi, M. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.09636, 2023 b

    2023 arXiv

  3. [11]

    o lge, E., G \

    Casanova, E., Davis, K., G \"o lge, E., G \"o knar, G., Gulea, I., Hart, L., Aljafari, A., Meyer, J., Morais, R., Olayemi, S., et al. Xtts: a massively multilingual zero-shot text-to-speech model. arXiv preprint arXiv:2406.04904, 2024

    2024 arXiv

  4. [12]

    Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 11315--11325, 2022

    2022

  5. [13]

    T., Rubinstein, M., et al

    Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.00704, 2023

    2023 arXiv

  6. [14]

    Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio

    Chen, G., Chai, S., Wang, G., Du, J., Zhang, W.-Q., Weng, C., Su, D., Povey, D., Trmal, J., Zhang, J., et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106.06909, 2021

    2021 arXiv

  7. [15]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing

    Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16 0 (6): 0 1505--1518, 2022

    2022

  8. [16]

    Streaming voice conversion via intermediate bottleneck features and non-streaming teacher guidance

    Chen, Y., Tu, M., Li, T., Li, X., Kong, Q., Li, J., Wang, Z., Tian, Q., Wang, Y., and Wang, Y. Streaming voice conversion via intermediate bottleneck features and non-streaming teacher guidance. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal ...

    2023

  9. [17]

    Self-supervised learning with random-projection quantizer for speech recognition

    Chiu, C.-C., Qin, J., Zhang, Y., Yu, J., and Wu, Y. Self-supervised learning with random-projection quantizer for speech recognition. In International Conference on Machine Learning, pp.\ 3915--3924. PMLR, 2022

    2022

  10. [18]

    Diff-hiervc: Diffusion-based hierarchical voice conversion with robust pitch generation and masked prior for zero-shot speaker adaptation

    Choi, H.-Y., Lee, S.-H., and Lee, S.-W. Diff-hiervc: Diffusion-based hierarchical voice conversion with robust pitch generation and masked prior for zero-shot speaker adaptation. International Speech Communication Association, pp.\ 2283--2287, 2023 a

    2023

  11. [19]

    Choi, J., Kim, M., and Ro, Y. M. Intelligible lip-to-speech synthesis with speech units. arXiv preprint arXiv:2305.19603, 2023 b

    2023 arXiv

  12. [20]

    W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training

    Chung, Y.-A., Zhang, Y., Han, W., Chiu, C.-C., Qin, J., Pang, R., and Wu, Y. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp.\ 244--25...

    2021

  13. [21]

    Librimix: An open-source dataset for generalizable speech separation

    Cosentino, J., Pariente, M., Cornell, S., Deleforge, A., and Vincent, E. Librimix: An open-source dataset for generalizable speech separation. arXiv preprint arXiv:2005.11262, 2020

    2005 arXiv

  14. [22]

    High fidelity neural audio compression

    D \'e fossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438, 2022

    2022 arXiv

  15. [23]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  16. [24]

    I., Waldner, F., Caccetta, P., and Wu, C

    Diakogiannis, F. I., Waldner, F., Caccetta, P., and Wu, C. Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing, 162: 0 94--114, 2020

    2020

  17. [25]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens

    Du, Z., Chen, Q., Zhang, S., Hu, K., Lu, H., Yang, Y., Hu, H., Zheng, S., Gu, Y., Ma, Z., et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  18. [26]

    Icassp 2023 deep noise suppression challenge

    Dubey, H., Aazami, A., Gopal, V., Naderi, B., Braun, S., Cutler, R., Ju, A., Zohourian, M., Tang, M., Golestaneh, M., et al. Icassp 2023 deep noise suppression challenge. IEEE Open Journal of Signal Processing, 2024

    2023

  19. [27]

    Taming transformers for high-resolution image synthesis

    Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12873--12883, 2021

    2021

  20. [28]

    Metricgan+: An improved version of metricgan for speech enhancement

    Fu, S.-W., Yu, C., Hsieh, T.-A., Plantinga, P., Ravanelli, M., Lu, X., and Tsao, Y. Metricgan+: An improved version of metricgan for speech enhancement. arXiv preprint arXiv:2104.03538, 2021

    2021 arXiv

  21. [29]

    Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition

    Gao, Z., Zhang, S., McLoughlin, I., and Yan, Z. Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition. arXiv preprint arXiv:2206.08317, 2022

    2022 arXiv

  22. [30]

    Funasr: A fundamental end-to-end speech recognition toolkit

    Gao, Z., Li, Z., Wang, J., Luo, H., Shi, X., Chen, M., Li, Y., Zuo, L., Du, Z., Xiao, Z., et al. Funasr: A fundamental end-to-end speech recognition toolkit. arXiv preprint arXiv:2305.11013, 2023

    2023 arXiv

  23. [31]

    Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications

    Guo, H.-H., Liu, K., Shen, F.-Y., Wu, Y.-C., Xie, F.-L., Xie, K., and Xu, K.-T. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint arXiv:2409.03283, 2024

    2024 arXiv

  24. [32]

    Didispeech: A large scale mandarin speech corpus

    Guo, T., Wen, C., Jiang, D., Luo, N., Zhang, R., Zhao, S., Li, W., Gong, C., Zou, W., Han, K., et al. Didispeech: A large scale mandarin speech corpus. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 6968--6972. IEEE, 2021

    2021

  25. [33]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation

    He, H., Shang, Z., Wang, C., Li, X., Gu, Y., Hua, H., Liu, L., Yang, C., Li, J., Shi, P., et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. arXiv preprint arXiv:2407.05361, 2024

    2024 arXiv

  26. [34]

    Masked autoencoders are scalable vision learners

    He, K., Chen, X., Xie, S., Li, Y., Doll \'a r, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 16000--16009, 2022

    2022

  27. [35]

    and Salimans, T

    Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  28. [36]

    H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A

    Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29: 0 3451--3460, 2021

    2021

  29. [37]

    J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W

    Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  30. [38]

    Zero-shot accent conversion using pseudo siamese disentanglement network

    Jia, D., Tian, Q., Peng, K., Li, J., Chen, Y., Ma, M., Wang, Y., and Wang, Y. Zero-shot accent conversion using pseudo siamese disentanglement network. arXiv preprint arXiv:2212.05751, 2022

    2022 arXiv

  31. [39]

    Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models

    Ju, Z., Wang, Y., Shen, K., Tan, X., Xin, D., Yang, D., Liu, Y., Leng, Y., Song, K., Tang, S., et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100, 2024

    2024 arXiv

  32. [40]

    Libri-light: A benchmark for asr with limited or no supervision

    Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., Xu, Q., Mazar \'e , P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech...

    2020

  33. [41]

    Libriheavy: a 50,000 hours asr corpus with punctuation casing and context

    Kang, W., Yang, X., Yao, Z., Kuang, F., Yang, Y., Guo, L., Lin, L., and Povey, D. Libriheavy: a 50,000 hours asr corpus with punctuation casing and context. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 10991--10995...

    2024

  34. [42]

    Speak, read and prompt: High-fidelity text-to-speech with minimal supervision

    Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Girgin, S., Pietquin, O., Sharifi, M., Tagliasacchi, M., and Zeghidour, N. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 11: 0...

    2023

  35. [43]

    Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech

    Kim, J., Kong, J., and Son, J. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pp.\ 5530--5540. PMLR, 2021

    2021

  36. [44]

    Kim, M., Hong, J., and Ro, Y. M. Lip-to-speech synthesis in the wild with multi-task learning. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 1--5. IEEE, 2023

    2023

  37. [45]

    C., Lo, W.-Y., et al

    Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 4015--4026, 2023

    2023

  38. [46]

    L., and Khudanpur, S

    Ko, T., Peddinti, V., Povey, D., Seltzer, M. L., and Khudanpur, S. A study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp.\ 5220--5224. IEEE, 2017

    2017

  39. [47]

    High-fidelity audio compression with improved rvqgan

    Kumar, R., Seetharaman, P., Luebs, A., Kumar, I., and Kumar, K. High-fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Systems, 36, 2024

    2024

  40. [48]

    Base tts: Lessons from building a billion-parameter text-to-speech model on 100k hours of data

    ajszczak, M., C \'a mbara, G., Li, Y., Beyhan, F., van Korlaar, A., Yang, F., Joly, A., Mart \' n-Cortinas, \'A ., Abbas, A., Michalski, A., et al. Base tts: Lessons from building a billion-parameter text-to-speech model on 100k hours of data. arXiv preprint arXiv:2402.08093, 2024

    2024 arXiv

  41. [49]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.\ 1207--1216, Stanford, CA, 2000. Morgan Kaufmann

    2000

  42. [50]

    Voicebox: Text-guided multilingual universal speech generation at scale

    Le, M., Vyas, A., Shi, B., Karrer, B., Sari, L., Moritz, R., Williamson, M., Manohar, V., Adi, Y., Mahadeokar, J., et al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36, 2024

    2024

  43. [51]

    Hierspeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis

    Lee, S.-H., Choi, H.-Y., Kim, S.-B., and Lee, S.-W. Hierspeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis. arXiv preprint arXiv:2311.12454, 2023

    2023 arXiv

  44. [52]

    Improved masked image generation with token-critic

    Lezama, J., Chang, H., Jiang, L., and Essa, I. Improved masked image generation with token-critic. In European Conference on Computer Vision, pp.\ 70--86. Springer, 2022

    2022

  45. [53]

    Freevc: Towards high-quality text-free one-shot voice conversion

    Li, J., Tu, W., and Xiao, L. Freevc: Towards high-quality text-free one-shot voice conversion. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 1--5. IEEE, 2023

    2023

  46. [54]

    Masksr: Masked language model for full-band speech restoration

    Li, X., Wang, Q., and Liu, X. Masksr: Masked language model for full-band speech restoration. arXiv preprint arXiv:2406.02092, 2024

    2024 arXiv

  47. [55]

    T., Ben-Hamu, H., Nickel, M., and Le, M

    Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022

    2022 arXiv

  48. [56]

    H., Le, M., Vyas, A., Shi, B., Tjandra, A., and Hsu, W.-N

    Liu, A. H., Le, M., Vyas, A., Shi, B., Tjandra, A., and Hsu, W.-N. Generative pre-training for speech with flow matching. arXiv preprint arXiv:2310.16338, 2023

    2023 arXiv

  49. [57]

    Voicefixer: Toward general speech restoration with neural vocoder

    Liu, H., Kong, Q., Tian, Q., Zhao, Y., Wang, D., Huang, C., and Wang, Y. Voicefixer: Toward general speech restoration with neural vocoder. arXiv preprint arXiv:2109.13731, 2021

    2021 arXiv

  50. [58]

    Voicefixer: A unified framework for high-fidelity speech restoration

    Liu, H., Liu, X., Kong, Q., Tian, Q., Zhao, Y., Wang, D., Huang, C., and Wang, Y. Voicefixer: A unified framework for high-fidelity speech restoration. arXiv preprint arXiv:2204.05841, 2022

    2022 arXiv

  51. [59]

    Liu, H., Xu, X., Yuan, Y., Wu, M., Wang, W., and Plumbley, M. D. Semanticodec: An ultra low bitrate semantic audio codec for general sound. arXiv preprint arXiv:2405.00233, 2024 a

    2024 arXiv

  52. [60]

    Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q., Tian, Q., Wang, Y., Wang, W., Wang, Y., and Plumbley, M. D. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024 b

    2024

  53. [61]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  54. [62]

    Auto-avsr: Audio-visual speech recognition with automatic labels

    Ma, P., Haliassos, A., Fernandez-Lopez, A., Chen, H., Petridis, S., and Pantic, M. Auto-avsr: Audio-visual speech recognition with automatic labels. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 1--5. IEEE, 2023

    2023

  55. [63]

    Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets

    Mittag, G., Naderi, B., Chehadi, A., and M \"o ller, S. Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets. arXiv preprint arXiv:2104.09494, 2021

    2021 arXiv

  56. [64]

    Mohammadi, S. H. and Kain, A. An overview of voice conversion systems. Speech Communication, 88: 0 65--82, 2017

    2017

  57. [65]

    S., and Zisserman, A

    Nagrani, A., Chung, J. S., and Zisserman, A. Voxceleb: a large-scale speaker identification dataset. arXiv preprint arXiv:1706.08612, 2017

    2017 arXiv

  58. [66]

    Selfvc: Voice conversion with iterative refinement using self transformations

    Neekhara, P., Hussain, S., Valle, R., Ginsburg, B., Ranjan, R., Dubnov, S., Koushanfar, F., and McAuley, J. Selfvc: Voice conversion with iterative refinement using self transformations. arXiv preprint arXiv:2310.09653, 2023

    2023 arXiv

  59. [67]

    and Waibel, A

    Nguyen, T.-B. and Waibel, A. Convoifilter: A case study of doing cocktail party speech recognition. In 2024 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pp.\ 565--569. IEEE, 2024

    2024

  60. [68]

    Librispeech: an asr corpus based on public domain audio books

    Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp.\ 5206--5210. IEEE, 2015

    2015

  61. [69]

    Segan: Speech enhancement generative adversarial network

    Pascual, S., Bonafonte, A., and Serra, J. Segan: Speech enhancement generative adversarial network. arXiv preprint arXiv:1703.09452, 2017

    2017 arXiv

  62. [70]

    Voicecraft: Zero-shot speech editing and text-to-speech in the wild

    Peng, P., Huang, P.-Y., Li, D., Mohamed, A., and Harwath, D. Voicecraft: Zero-shot speech editing and text-to-speech in the wild. arXiv preprint arXiv:2403.16973, 2024

    2024 arXiv

  63. [71]

    Mls: A large-scale multilingual dataset for speech research

    Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert, R. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411, 2020

    2012 arXiv

  64. [72]

    Autovc: Zero-shot voice style transfer with only autoencoder loss

    Qian, K., Zhang, Y., Chang, S., Yang, X., and Hasegawa-Johnson, M. Autovc: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning, pp.\ 5210--5219. PMLR, 2019

    2019

  65. [73]

    Openvoice: Versatile instant voice cloning

    Qin, Z., Zhao, W., Yu, X., and Sun, X. Openvoice: Versatile instant voice cloning. arXiv preprint arXiv:2312.01479, 2023

    2023 arXiv

  66. [74]

    Language models are unsupervised multitask learners

    Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  67. [75]

    W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I

    Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pp.\ 28492--28518. PMLR, 2023

    2023

  68. [76]

    K., Gopal, V., and Cutler, R

    Reddy, C. K., Gopal, V., and Cutler, R. Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 6493--6497. IEEE, 2021

    2021

  69. [77]

    Fastspeech: Fast, robust and controllable text to speech

    Ren, Y., Ruan, Y., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y. Fastspeech: Fast, robust and controllable text to speech. Advances in neural information processing systems, 32, 2019

    2019

  70. [78]

    Fastspeech 2: Fast and high-quality end-to-end text to speech

    Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558, 2020

    2006 arXiv

  71. [79]

    Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers

    Shen, K., Ju, Z., Tan, X., Liu, Y., Leng, Y., He, L., Qin, T., Zhao, S., and Bian, J. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09116, 2023

    2023 arXiv

  72. [80]

    Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis

    Siuzdak, H. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306.00814, 2023

    2023 arXiv

  73. [81]

    Tselm: Target speaker extraction using discrete tokens and language models

    Tang, B., Zeng, B., and Li, M. Tselm: Target speaker extraction using discrete tokens and language models. arXiv preprint arXiv:2409.07841, 2024

    2024 arXiv

  74. [82]

    The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings

    Thiemann, J., Ito, N., and Vincent, E. The diverse environments multi-channel acoustic noise database (demand): A database of multichannel environmental noise recordings. In Proceedings of Meetings on Acoustics, volume 19. AIP Publishing, 2013

    2013

  75. [83]

    Llama 2: Open foundation and fine-tuned chat models

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  76. [84]

    Neural discrete representation learning

    Van Den Oord, A., Vinyals, O., et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017

    2017

  77. [85]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  78. [86]

    Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit

    Veaux, C., Yamagishi, J., MacDonald, K., et al. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit. University of Edinburgh. The Centre for Speech Technology Research (CSTR), 6: 0 15, 2017

    2017

  79. [87]

    Audiobox: Unified audio generation with natural language prompts

    Vyas, A., Shi, B., Le, M., Tjandra, A., Wu, Y.-C., Guo, B., Zhang, J., Zhang, X., Adkins, R., Ngan, W., et al. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312.15821, 2023

    2023 arXiv

  80. [88]

    Neural codec language models are zero-shot text to speech synthesizers

    Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023 a

    2023 arXiv

  81. [89]

    A., Weiss, R

    Wang, Q., Muckenhirn, H., Wilson, K., Sridhar, P., Wu, Z., Hershey, J., Saurous, R. A., Weiss, R. J., Jia, Y., and Moreno, I. L. Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking. arXiv preprint arXiv:1810.04826, 2018

    arXiv

  82. [90]

    Wesep: A scalable and flexible toolkit towards generalizable target speaker extraction

    Wang, S., Zhang, K., Lin, S., Li, J., Wang, X., Ge, M., Yu, J., Qian, Y., and Li, H. Wesep: A scalable and flexible toolkit towards generalizable target speaker extraction. arXiv preprint arXiv:2409.15799, 2024 a

    2024 arXiv

  83. [91]

    E., Chen, S., Tang, M., Liu, S., Li, J., and Yoshioka, T

    Wang, X., Thakker, M., Chen, Z., Kanda, N., Eskimez, S. E., Chen, S., Tang, M., Liu, S., Li, J., and Yoshioka, T. Speechx: Neural codec language model as a versatile speech transformer. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024 b

    2024

  84. [92]

    Maskgct: Zero-shot text-to-speech with masked generative codec transformer

    Wang, Y., Zhan, H., Liu, L., Zeng, R., Guo, H., Zheng, J., Zhang, Q., Zhang, S., and Wu, Z. Maskgct: Zero-shot text-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.00750, 2024 c

    2024 arXiv

  85. [93]

    Lm-vc: Zero-shot voice conversion via speech generation based on language models

    Wang, Z., Chen, Y., Xie, L., Tian, Q., and Wang, Y. Lm-vc: Zero-shot voice conversion via speech generation based on language models. IEEE Signal Processing Letters, 2023 b

    2023

  86. [94]

    Selm: Speech enhancement using discrete tokens and language models

    Wang, Z., Zhu, X., Zhang, Z., Lv, Y., Jiang, N., Zhao, G., and Xie, L. Selm: Speech enhancement using discrete tokens and language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 11561--11565. IEEE, 2024 d

    2024

  87. [95]

    Tf-gridnet: Integrating full-and sub-band modeling for speech separation

    Wang, Z.-Q., Cornell, S., Choi, S., Lee, Y., Kim, B.-Y., and Watanabe, S. Tf-gridnet: Integrating full-and sub-band modeling for speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023 c

    2023

  88. [96]

    R., McQuinn, E., Crow, D., Manilow, E., and Roux, J

    Wichern, G., Antognini, J., Flynn, M., Zhu, L. R., McQuinn, E., Crow, D., Manilow, E., and Roux, J. L. Wham!: Extending speech separation to noisy environments. arXiv preprint arXiv:1907.01160, 2019

    1907 arXiv

  89. [97]

    J., Wang, W., Lin, K

    Xie, J., Mao, W., Bai, Z., Zhang, D. J., Wang, W., Lin, K. Q., Gu, Y., Chen, Z., Yang, Z., and Shou, M. Z. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  90. [98]

    Uniaudio: An audio foundation model toward universal audio generation

    Yang, D., Tian, J., Tan, X., Huang, R., Liu, S., Chang, X., Shi, J., Zhao, S., Bian, J., Wu, X., et al. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704, 2023

    2023 arXiv

  91. [99]

    G., Yang, M.-H., Hao, Y., Essa, I., et al

    Yu, L., Cheng, Y., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M.-H., Hao, Y., Essa, I., et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 10459--10469, 2023 a

    2023

  92. [100]

    B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Hauptmann, A

    Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y., Gupta, A., Gu, X., Hauptmann, A. G., et al. Language model beats diffusion--tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023 b

    arXiv

  93. [101]

    Soundstream: An end-to-end neural audio codec

    Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30: 0 495--507, 2021

    2021

  94. [103]

    J., Jia, Y., Chen, Z., and Wu, Y

    Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R. J., Jia, Y., Chen, Z., and Wu, Y. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02882, 2019 b

    1904 arXiv

  95. [104]

    Amphion: An open-source audio, music and speech generation toolkit

    Zhang, X., Xue, L., Wang, Y., Gu, Y., Chen, X., Fang, Z., Chen, H., Zou, L., Wang, C., Han, J., et al. Amphion: An open-source audio, music and speech generation toolkit. arXiv preprint arXiv:2312.09911, 2023 a

    2023 arXiv

  96. [105]

    Vevo: Controllable zero-shot voice imitation with self-supervised disentanglement

    Zhang, X., Zhang, X., Peng, K., Tang, Z., Manohar, V., Liu, Y., Hwang, J., Li, D., Wang, Y., Chan, J., Huang, Y., Wu, Z., and Ma, M. Vevo: Controllable zero-shot voice imitation with self-supervised disentanglement. OpenReview, 2024

    2024

  97. [106]

    Google usm: Scaling automatic speech recognition beyond 100 languages

    Zhang, Y., Han, W., Qin, J., Wang, Y., Bapna, A., Chen, Z., Chen, N., Li, B., Axelrod, V., Wang, G., et al. Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037, 2023 b

    2023 arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.