Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Discl-VC splits speech into discrete content tokens, prosody tokens, and an in-context timbre prompt, making zero-shot voice conversion controllable in both who speaks and how.

desk verdict A credible controllable VC system whose prosody-disentanglement claim needs direct verification; worth peer review. read the letter →

arxiv 2505.24291 v1 pith:S6MZO65W submitted 2025-05-30 cs.SD cs.AIeess.AS

Kaidi Wang , Wenhao Guan , Ziyue Jiang , Hukai Huang , Peijie Chen , Weijie Wu , Qingyang Hong , Lin Li This is my paper · ORCID
classification cs.SDcs.AIeess.AS
keywords zero-shotvoiceconversionin-contextlearningspeechdisentanglementprosodycontrolflowmatchingmaskedgenerativetransformervectorquantizationdiscretetokens
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Discl-VC is a voice-conversion system built on the idea that speech can be separated into three re-combinable streams: content tokens, prosody tokens, and an in-context timbre prompt. The paper argues that once content and prosody are discretized separately, the target speaker's timbre can be rendered by a flow-matching transformer that learns from surrounding context, while a non-autoregressive mask transformer can regenerate the prosody tokens to imitate a reference clip's speaking style. A single frozen pipeline can then convert any utterance to an unseen speaker's voice and independently control how it is spoken, without per-speaker fine-tuning. In the reported comparisons, the system attains the best naturalness score (N-MOS 4.31) and UTMOS (4.079) among the tested systems, together with the smallest parameter count.

What carries the argument

The machinery has two token streams and two non-autoregressive transformers. Content tokens are obtained by clustering the hidden units of a self-supervised speech model into 1024 classes and removing adjacent duplicates, which strips duration-related prosody. Prosody tokens are obtained by passing a speaker-disentangled continuous representation through an inverted length regulator and a vector-quantization bottleneck trained with a SimVQ-style loss and an F0 regression loss; this bottleneck is meant to filter out content and timbre while retaining expressive variation. The flow-matching transformer predicts masked mel-spectrogram regions from the content tokens, the prosody tokens, and the surrounding acoustic context, thereby learning timbre as an in-context task. The prosody mask transformer follows the mask-and-predict paradigm: given content tokens and a reference prompt's prosody tokens, it iteratively unmasks all prosody tokens in parallel, so that a reference speaker's style can be transplanted onto the source content.

What would settle it

A direct test would be to train a speaker classifier on prosody tokens alone, extracted from many speakers' audio without any content or timbre input: if that classifier performs far above chance, the tokens are not speaker-independent and the disentanglement claim weakens. A more behavioral falsifier: hold content fixed, generate the same sentence with prosody tokens taken from several different reference speakers, and measure the speaker-embedding similarity of the outputs to the target voice; if the similarity shifts with the prosody source, prosody is leaking timbre.

Watch

Extended reading notes

Core claim

The paper's central claim is that disentangled discrete tokens give voice conversion a clean control surface. Content is encoded as deduplicated cluster units of a self-supervised speech model, prosody is encoded as vector-quantized codes taken from a speaker-disentangled continuous representation under an F0-supervision constraint, and timbre is never extracted at all: it is left for the flow-matching transformer to reconstruct by in-context learning from a short acoustic prompt. The prosody mask transformer then predicts what the prosody tokens of the source content should be when told to follow a reference speaker's style, and the flow-matching transformer renders those tokens in the target voice. The authors report that this two-stage, fully non-autoregressive design beats the comparison systems on naturalness, UTMOS, and F0 correlation in zero-shot voice conversion, and enables both prosody-preserving and prosody-converted generation.

Load-bearing premise

The load-bearing premise is that the prosody tokens encode speaking style without also hiding speaker identity or content, so that replacing them with tokens predicted from a reference clip changes only the way the words are spoken and not what is said or whose voice it is.

Editorial extensions

If this is right

  • A user can take one recorded sentence and re-speak it in an unseen target voice while instructing the model to copy the happy, sad, or neutral delivery of a different reference clip, all from the same trained checkpoint.
  • Because both transformers generate in parallel rather than token-by-token, controllable voice conversion can run without the latency of autoregressive synthesis.
  • The deduplicated content tokens make duration a predicted quantity, which opens the door to explicit control of speaking rate and rhythm.
  • The reported system uses 131M parameters in zero-shot voice conversion, compared with 922M for the larger baseline, so the control gains are not purchased with a larger model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves the speaker-independence of prosody tokens unmeasured; an explicit probe of that property would either confirm the central control claim or reveal a ceiling on style transfer accuracy.
  • The same content-token/prosody-token split could be lifted into text-to-speech, expressive dubbing, and emotion conversion, where content and style are likewise separable and a single reference clip should suffice as a style prompt.
  • If the prosody tokens are truly content-free, they become an editable layer: one could modify a single pitch or rhythm token and re-render the utterance without re-synthesizing from text, enabling fine-grained prosody editing.
  • The classifier-free guidance drop probability is fixed at 0.2 in both transformers; scanning this value would likely reveal a naturalness-versus-style-fidelity trade-off that the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Discl-VC, a zero-shot voice conversion system that factorizes speech into discrete content tokens (HuBERT k-means), discrete prosody tokens (a trained VQ encoder on ContentVec features), and timbre via a flow-matching transformer with in-context learning. A non-autoregressive mask generative transformer predicts prosody tokens from a reference prompt to enable prosody conversion. The system is trained in two stages and evaluated on VCTK and ESD against FAcodec and Vevo. The paper claims superior zero-shot voice conversion performance and accurate prosody control.

Significance. The proposed architecture is well motivated, and the two-stage token-prediction scheme is a sensible way to add prosody controllability to a zero-shot voice conversion pipeline. If the empirical results were fully substantiated, a 131M-parameter model with state-of-the-art naturalness (N-MOS 4.31, UTMOS 4.079) and competitive speaker similarity (SECS 0.929) would be a useful contribution. The public demo and comparisons to two strong baselines are valuable. However, the current evidence is mixed: WER is worse than FAcodec, SECS is lower than Vevo in the prosody-converted setting, and no significance tests are given for objective metrics.

major comments (3)
  1. [Abstract, §3.2 (Table 1)] The central claim of 'superior performance' in zero-shot voice conversion is not supported by the reported WER: Discl-VC achieves 1.946, clearly worse than FAcodec's 1.341. Since WER is the primary content-preservation metric, the model is not uniformly superior; the statement in §3.2 that 'the WER performance falls between the two baselines, demonstrating the superiority' does not follow. Either temper the claim to a trade-off or show that the WER difference is within noise after statistical testing.
  2. [§3.2, Tables 1 and 2] None of the objective metrics (UTMOS, WER, F0Corr, SECS) is reported with error bars or significance tests. Several decisive comparisons rest on small differences (e.g., F0Corr 0.973 vs 0.968 in Table 1; SECS 0.847 vs 0.892 in the prosody-converted row of Table 2; UTMOS 4.119 vs 4.057). Since these are single-number estimates, the reported advantage of Discl-VC over Vevo and FAcodec on several metrics could be within run-to-run variability. The authors should provide variance estimates (e.g., bootstrap CIs or multiple evaluation seeds) and, if feasible, significance tests.
  3. [§2.3 (Eq. 3) and §3.3] The disentanglement claim—that the VQ prosody tokens carry prosody but not content or speaker information—is load-bearing for the prosody-control application, yet it is never directly measured. The stage-1 objective in Eq. (3) is reconstruction-based (duration, SimVQ, flow matching, F0); nothing penalizes mutual information between prosody tokens and content tokens, and the codebook is optimized for reconstruction, which can incentivize residual phonetic content in prosody tokens. The evidence provided (Figure 3 spectrogram and the w/o ContentVec ablation) is qualitative or indirect. I recommend adding a quantitative probe: train a linear classifier on the prosody tokens to predict either the content token cluster or the speaker identity, and report accuracy versus chance; and/or measure WER when prosody tokens are exchanged between utterances with different content but the content tokens are held fixed.
minor comments (4)
  1. [§2.3] The phrase 'ground true prosody tokens' should be 'ground truth prosody tokens'.
  2. [§3.1.2] The sentence 'We use the pre-trained RMVPE to extract F0' would benefit from a citation to the RMVPE paper.
  3. [Table 3] The 'w/o F0 loss' row shows a lower WER (1.279) than the full model (1.946); this is worth discussing, since it suggests a possible trade-off between content preservation and prosody modeling.
  4. [§2.1] In the SimVQ description, the codebook vectors are randomly initialized and never updated, with a learned linear layer W producing the quantized result; this choice should be explained more clearly, in particular how the codebook is selected and why this avoids collapse.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the system is an empirical pipeline evaluated against external metrics and baselines.

full rationale

Discl-VC is an empirical voice-conversion system, not a derivation from first principles. The content/prosody/timbre decomposition is implemented by frozen self-supervised extractors (HuBERT k-means, ContentVec) plus a learned VQ prosody encoder. The stage-1 loss (Eq. 3) is reconstruction-based with an auxiliary F0 loss, and stage-2 trains a mask transformer to predict VQ prosody tokens from reference prompts. The paper's central claims are evaluated with external, independently pre-trained metrics (UTMOS, Whisper WER, WavLM SECS, F0 Pearson correlation) against two released baselines (FAcodec, Vevo), so the headline numbers are not constructed by the training objectives. The ablation that replaces ContentVec with first-20 mel dims even acknowledges a remaining speaker-information leak ('the extracted prosody tokens inevitably still contain some speaker information'), which is a limitation candidly stated rather than a circular move. The only self-citations ([21]-[23]) justify using flow matching in audio and are not load-bearing for the disentanglement or prosody-control claims. The concern that prosody tokens may retain content or speaker information is a validity/robustness risk, not a definitional circularity: the paper does not define its evaluation metrics in terms of its own token outputs.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the disentanglement of content, prosody, and timbre, which is achieved by combining pre-trained self-supervised models and learned quantization. The paper does not introduce new physical entities or theoretical constructs. The key free parameters are the token vocabularies and guidance scales, which are chosen empirically and affect performance.

free parameters (4)
  • K-means clusters for content tokens = 1024
    The number of clusters in K-means on HuBERT layer 24 is chosen by hand and affects the granularity of content tokens.
  • VQ codebook size = 2048
    The size of the prosody codebook is set to 2048, which directly determines the prosody token vocabulary and thus the granularity of prosody modeling.
  • Guidance scales = 1.0 (flow matching), 2.5 (prosody mask transformer)
    Classifier-free guidance scales are tuned on validation data to improve generation quality.
  • SimVQ loss weight lambda = not specified
    The weight lambda in Eq. 1 is a hyperparameter that balances the VQ loss terms; its value is not reported, making reproduction harder.
assumptions (4)
  • domain assumption HuBERT large layer 24 with K-means discretization yields content tokens that contain primarily semantic information and filter out timbre and prosody.
    This is asserted in Section 2.1 without direct measurement, relying on prior findings that HuBERT representations are content-rich.
  • domain assumption ContentVec output contains all content and prosody information but no speaker information.
    The paper relies on ContentVec's disentanglement property as a starting point, citing prior work [7].
  • standard math The optimal transport flow matching objective with a straight-line path can learn a valid ODE for mel spectrogram generation.
    Flow matching theory is well-established, so this is a standard mathematical framework.
  • domain assumption Masked token modeling with a sine mask schedule and confidence-based iterative decoding produces coherent prosody token sequences.
    This is adopted from MaskGIT/MaskGCT and assumed to transfer to prosody tokens without further justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion." pith.science (2026). https://pith.science/paper/S6MZO65W

@misc{pith2026250524291,
  author       = {Pith},
  title        = {Pith review of: Discl-VC: Disentangled Discrete Tokens and In-Context Learning for Controllable Zero-Shot Voice Conversion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S6MZO65W}},
  note         = {Machine review of arXiv:2505.24291}
}
read the original abstract

Currently, zero-shot voice conversion systems are capable of synthesizing the voice of unseen speakers. However, most existing approaches struggle to accurately replicate the speaking style of the source speaker or mimic the distinctive speaking style of the target speaker, thereby limiting the controllability of voice conversion. In this work, we propose Discl-VC, a novel voice conversion framework that disentangles content and prosody information from self-supervised speech representations and synthesizes the target speaker's voice through in-context learning with a flow matching transformer. To enable precise control over the prosody of generated speech, we introduce a mask generative transformer that predicts discrete prosody tokens in a non-autoregressive manner based on prompts. Experimental results demonstrate the superior performance of Discl-VC in zero-shot voice conversion and its remarkable accuracy in prosody control for synthesized speech.

Figures

Figures reproduced from arXiv: 2505.24291 by the authors.

Figure 1
Figure 1. The overall architecture of our proposed system. clustering on the continuous representations from the 24th layer. The number of clusters is set to 1024. The content tokens ob￾tained from the Content Extractor can be viewed as contain￾ing only semantic information while filtering out most timbre and prosody information. We also perform a deduplication pro￾cess by removing adjacent duplicate tokens, further eliminati… view at source ↗
Figure 2
Figure 2. Results of subjective metrics for prosody related zero￾shot voice conversion. (1) Origin (2) Remove prosody (3) Remove content [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The visualization of speech disentanglement, where the corresponding sentence is: “It would be a hard choice.” from the same sample, the extracted prosody tokens inevitably still contain some speaker information, leading to mismatches between training and inference. When F0 loss is not used, the model performs better in terms of WER, but it shows weaker prosody similarity and other metrics compared to the original m… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models

    eess.AS 2025-10 conditional novelty 6.0 of 10

    A single LLM can do ASR and zero-shot TTS on continuous speech features by switching between causal and bidirectional attention, reaching competitive but not state-of-the-art results.

  2. DiFlow-TTS: Compact and Low-Latency Zero-Shot Text-to-Speech with Discrete Flow Matching

    cs.SD 2025-09 conditional novelty 6.0 of 10

    A compact zero-shot TTS that applies discrete flow matching with separate prediction heads for prosody and acoustic tokens, reporting near-best quality, best prosody/energy metrics, and up to 25.8x faster inference.

Reference graph

Works this paper leans on

39 extracted references · 33 canonical work pages · cited by 2 Pith papers

  1. [1]

    Zero-shot voice conversion [1, 2] generates target speech given the voice of an unseen speaker during training, making the task even more challenging

    Introduction V oice conversion is the task of transforming the voice of a source speaker into that of a target speaker, while preserving the linguistic content information of the source speaker. Zero-shot voice conversion [1, 2] generates target speech given the voice of an unseen speaker during training, making the task even more challenging. A widely ad...

    work page

  2. [2]

    Discl-VC 2.1. Speech disentanglement The overall architecture of our system is shown in Figure 1 (1), where the Content Extractor and Content-Prosody Extractor are pre-trained self-supervised models. In this work, we use Hu- BERT large [29] as the Content Extractor and apply K-means arXiv:2505.24291v1 [cs.SD] 30 May 2025 Flow Matching Transformer VQ Proso...

    work page Pith review arXiv 2025

  3. [3]

    It would be a hard choice

    Experiments 3.1. Experiments setup 3.1.1. dataset We use the Librilight small+medium dataset for training, which contains approximately 6k hours of English speech data. Dur- ing training, we only use speech with a duration between 4 to 20 seconds. We evaluate the model’s performance on voice conver- sion tasks using the VCTK and ESD datasets. Specifically...

    work page

  4. [4]

    And we leverage in-context learning to in- corporate timbre information and generate high-fidelity speech with a flow matching transformer

    Conclusion In this paper, we propose a new controllable voice conversion framework that disentangles content, prosody, and timbre using different methods. And we leverage in-context learning to in- corporate timbre information and generate high-fidelity speech with a flow matching transformer. Additionally, we introduce a non-autoregressive prosody mask t...

    work page

  5. [5]

    Acknowledgements This work was supported in part by the National Natural Science Foundation of China under Grants 62276220 and 62371407 and the Innovation of Policing Science and Technol- ogy, Fujian province (Grant number: 2024Y0068)

  6. [6]

    Speech resynthesis from discrete disentangled self-supervised representations,

    A. Polyak, Y . Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. rahman Mohamed, and E. Dupoux, “Speech resynthesis from discrete disentangled self-supervised representations,” inIn- terspeech, 2021

    2021

  7. [7]

    Sef-vc: Speaker embedding free zero-shot voice conversion with cross attention,

    J. Li, Y . Guo, X. Chen, and K. Yu, “Sef-vc: Speaker embedding free zero-shot voice conversion with cross attention,”ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 12 296–12 300, 2023

    work page 2024

  8. [8]

    Vec-tok-vc+: Residual-enhanced robust zero-shot voice conversion with progressive constraints in a dual-mode training strategy,

    L. Ma, X. Zhu, Y . Lv, Z. Wang, Z. Wang, W. He, H. Zhou, and L. Xie, “Vec-tok-vc+: Residual-enhanced robust zero-shot voice conversion with progressive constraints in a dual-mode training strategy,” inInterspeech 2024, 2024, pp. 2745–2749

    work page 2024

Show all 39 references
  1. [9]

    AutoVC: Zero-shot voice style transfer with only au- toencoder loss,

    K. Qian, Y . Zhang, S. Chang, X. Yang, and M. Hasegawa- Johnson, “AutoVC: Zero-shot voice style transfer with only au- toencoder loss,” inProceedings of the 36th International Confer- ence on Machine Learning, 2019, pp. 5210–5219

    2019

  2. [10]

    Unsupervised speech decomposition via triple information bot- tleneck,

    K. Qian, Y . Zhang, S. Chang, M. Hasegawa-Johnson, and D. Cox, “Unsupervised speech decomposition via triple information bot- tleneck,” inProceedings of the 37th International Conference on Machine Learning, 2020, pp. 7836–7846

    2020

  3. [11]

    Speechsplit2.0: Unsupervised speech disentanglement for voice conversion without tuning autoencoder bottlenecks,

    C. Ho Chan, K. Qian, Y . Zhang, and M. Hasegawa-Johnson, “Speechsplit2.0: Unsupervised speech disentanglement for voice conversion without tuning autoencoder bottlenecks,” inICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2...

    2022

  4. [12]

    Mega-tts: Zero-shot text- to-speech at scale with intrinsic inductive bias,

    Z. Jiang, Y . Ren, Z. Ye, J. Liu, C. Zhang, Q. Yang, S. Ji, R. Huang, C. Wang, X. Yin, Z. Ma, and Z. Zhao, “Mega-tts: Zero-shot text- to-speech at scale with intrinsic inductive bias,”ArXiv, 2023

    2023

  5. [13]

    ContentVec: An improved self- supervised speech representation by disentangling speakers,

    K. Qian, Y . Zhang, H. Gao, J. Ni, C.-I. Lai, D. Cox, M. Hasegawa- Johnson, and S. Chang, “ContentVec: An improved self- supervised speech representation by disentangling speakers,” in Proceedings of the 39th International Conference on Machine Learning, 2022, pp. 18 003–18 017

    2022

  6. [14]

    Diffvc+: Improving diffusion- based voice conversion for speaker anonymization,

    F. Huang, K. Zeng, and W. Zhu, “Diffvc+: Improving diffusion- based voice conversion for speaker anonymization,” inInter- speech 2024, 2024, pp. 4453–4457

    2024

  7. [15]

    Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,

    E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. G ¨olge, and M. A. Ponti, “Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone,” inInternational Con- ference on Machine Learning, 2022, pp. 2709–2720

    2022

  8. [16]

    Diffusion-based voice conversion with fast maxi- mum likelihood sampling scheme,

    V . Popov, I. V ovk, V . Gogoryan, T. Sadekova, M. S. Kudinov, and J. Wei, “Diffusion-based voice conversion with fast maxi- mum likelihood sampling scheme,” inInternational Conference on Learning Representations, 2022

    2022

  9. [17]

    introduces a method to explicitly disentangle content, style, and timbre, enabling separate control over the speaking style and timbre of the generated speech. Recently, Vevo [18] has also introduced a controllable voice conversion model by pro- gressively incorporating style ...

  10. [18]

    Dddm-vc: Decoupled denoising diffusion models with disentangled representation and prior mixup for verified robust voice conversion,

    H.-Y . Choi, S.-H. Lee, and S.-W. Lee, “Dddm-vc: Decoupled denoising diffusion models with disentangled representation and prior mixup for verified robust voice conversion,” inProceed- ings of the AAAI Conference on Artificial Intelligence, 2024, pp. 17 862–17 870

    2024

  11. [19]

    Mega-TTS 2: Boosting prompting mechanisms for zero-shot speech synthesis,

    Z. Jiang, J. Liu, Y . Ren, J. He, Z. Ye, S. Ji, Q. Yang, C. Zhang, P. Wei, C. Wang, X. Yin, Z. MA, and Z. Zhao, “Mega-TTS 2: Boosting prompting mechanisms for zero-shot speech synthesis,” inThe Twelfth International Conference on Learning Representa- tions, 2024

    2024

  12. [20]

    Enhancing expressive voice conversion with discrete pitch-conditioned flow matching model,

    J. Zuo, S. Ji, M. Fang, Z. Jiang, X. Cheng, Q. Yang, W. Liu, G. Zhang, Z. Tu, Y . Guo, and Z. Zhao, “Enhancing expressive voice conversion with discrete pitch-conditioned flow matching model,” inICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal P...

    2025

  13. [21]

    Ace- vc: Adaptive and controllable voice conversion using explic- itly disentangled self-supervised speech representations,

    S. S. Hussain, P. Neekhara, J. Huang, J. Li, and B. Ginsburg, “Ace- vc: Adaptive and controllable voice conversion using explic- itly disentangled self-supervised speech representations,”ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processin...

    2023

  14. [22]

    Diff-hiervc: Diffusion- based hierarchical voice conversion with robust pitch generation and masked prior for zero-shot speaker adaptation,

    H.-Y . Choi, S.-H. Lee, and S.-W. Lee, “Diff-hiervc: Diffusion- based hierarchical voice conversion with robust pitch generation and masked prior for zero-shot speaker adaptation,” inInterspeech 2023, 2023, pp. 2283–2287

    2023

  15. [23]

    Sta- blevc: Style controllable zero-shot voice conversion with condi- tional flow matching,

    J. Yao, Y . Yang, Y . Pan, Z. Ning, J. Ye, H. Zhou, and L. Xie, “Sta- blevc: Style controllable zero-shot voice conversion with condi- tional flow matching,”ArXiv, 2024

    2024

  16. [24]

    Vevo: Controllable zero-shot voice imitation with self-supervised disentanglement,

    X. Zhang, X. Zhang, K. Peng, Z. Tang, V . Manohar, Y . Liu, J. Hwang, D. Li, Y . Wang, J. Chan, Y . Huang, Z. Wu, and M. Ma, “Vevo: Controllable zero-shot voice imitation with self-supervised disentanglement,” inThe Thirteenth International Conference on Learning Representations, 2025

    2025

  17. [25]

    Flow matching for generative modeling,

    Y . Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” inThe Eleventh Inter- national Conference on Learning Representations, 2023

    2023

  18. [26]

    V oicebox: Text-guided multilingual universal speech gen- eration at scale,

    M. Le, A. Vyas, B. Shi, B. Karrer, L. Sari, R. Moritz, M. Williamson, V . Manohar, Y . Adi, J. Mahadeokar, and W.-N. Hsu, “V oicebox: Text-guided multilingual universal speech gen- eration at scale,” inAdvances in Neural Information Processing Systems, 2023, pp. 14 005–14 034

    2023

  19. [27]

    Reflow-tts: A rectified flow model for high-fidelity text-to- speech,

    W. Guan, Q. Su, H. Zhou, S. Miao, X. Xie, L. Li, and Q. Hong, “Reflow-tts: A rectified flow model for high-fidelity text-to- speech,” inICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 10 501–10 505

    2024

  20. [28]

    Lafma: A latent flow matching model for text-to-audio generation,

    W. Guan, K. Wang, W. Zhou, Y . Wang, F. Deng, H. Wang, L. Li, Q. Hong, and Y . Qin, “Lafma: A latent flow matching model for text-to-audio generation,” inInterspeech 2024, 2024, pp. 4813– 4817

    2024

  21. [29]

    Slim- speech: Lightweight and efficient text-to-speech with slim recti- fied flow,

    K. Wang, W. Guan, S. Lu, J. Yao, L. Li, and Q. Hong, “Slim- speech: Lightweight and efficient text-to-speech with slim recti- fied flow,” inICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5

    2025

  22. [30]

    E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts,

    S. E. Eskimez, X. Wang, M. Thakker, C. Li, C.-H. Tsai, Z. Xiao, H. Yang, Z. Zhu, M. Tang, X. Tan, Y . Liu, S. Zhao, and N. Kanda, “E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts,” in2024 IEEE Spoken Language Technology Workshop (SLT), 2024, pp. 682–689

    2024

  23. [31]

    F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,

    Y . Chen, Z. Niu, Z. Ma, K. Deng, C. Wang, J. Zhao, K. Yu, and X. Chen, “F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching,”arXiv preprint arXiv:2410.06885, 2024

    2024 arXiv

  24. [32]

    Maskgit: Masked generative image transformer,

    H. Chang, H. Zhang, L. Jiang, C. Liu, and W. T. Freeman, “Maskgit: Masked generative image transformer,” inThe IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  25. [33]

    Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,

    Z. Ju, Y . Wang, K. Shen, X. Tan, D. Xin, D. Yang, E. Liu, Y . Leng, K. Song, S. Tang, Z. Wu, T. Qin, X. Li, W. Ye, S. Zhang, J. Bian, L. He, J. Li, and sheng zhao, “Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” inForty- first Interna...

    2024

  26. [34]

    MaskGCT: Zero-shot text-to- speech with masked generative codec transformer,

    Y . Wang, H. Zhan, L. Liu, R. Zeng, H. Guo, J. Zheng, Q. Zhang, X. Zhang, S. Zhang, and Z. Wu, “MaskGCT: Zero-shot text-to- speech with masked generative codec transformer,” inThe Thir- teenth International Conference on Learning Representations, 2025

    2025

  27. [35]

    Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdi- nov, and A. Mohamed, “Hubert: Self-supervised speech represen- tation learning by masked prediction of hidden units,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, pp. 3451–3460, 2021

    2021

  28. [36]

    Addressing representation col- lapse in vector quantized models with one linear layer,

    Y . Zhu, B. Li, Y . Xin, and L. Xu, “Addressing representation col- lapse in vector quantized models with one linear layer,”ArXiv, 2024

    2024

  29. [37]

    Fastspeech 2: Fast and high-quality end-to-end text to speech,

    Y . Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “Fastspeech 2: Fast and high-quality end-to-end text to speech,” inInternational Conference on Learning Representations, 2021

    2021

  30. [38]

    Soundstorm: Efficient parallel audio gen- eration,

    Z. Borsos, M. Sharifi, D. Vincent, E. Kharitonov, N. Zeghidour, and M. Tagliasacchi, “Soundstorm: Efficient parallel audio gen- eration,”ArXiv, 2023

    2023

  31. [39]

    BigVGAN: A universal neural vocoder with large-scale train- ing,

    S. gil Lee, W. Ping, B. Ginsburg, B. Catanzaro, and S. Yoon, “BigVGAN: A universal neural vocoder with large-scale train- ing,” inThe Eleventh International Conference on Learning Rep- resentations, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.