GenVC: Self-Supervised Zero-Shot Voice Conversion

Ashi Garg; Henry Li Xinyuan; Kevin Duh; Leibny Paola Garc\'ia-Perera; Matthew Wiesner; Nicholas Andrews; Sanjeev Khudanpur; Zexin Cai

REVIEW 3 major objections 4 minor 1 cited by

GenVC: Self-Supervised Zero-Shot Voice Conversion

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Voice conversion can be built without any supervised speaker encoder, using only unlabeled speech and an autoregressive language model.

desk verdict Credible self-supervised zero-shot VC with a real anonymization win, but the same-utterance training trick needs a cross-dataset test and the naturalness claim is overstated. read the letter →

arxiv 2502.04519 v2 pith:RUSDF3FZ submitted 2025-02-06 eess.AS cs.LG

Zexin Cai , Henry Li Xinyuan , Ashi Garg , Leibny Paola García-Perera , Kevin Duh , Sanjeev Khudanpur , Matthew Wiesner , Nicholas Andrews This is my paper · ORCID

classification eess.AScs.LG

keywords zero-shotvoiceconversionself-supervisedlearningspeechlanguagemodelPerceiverencoderanonymizationdiscretetokensprivacyautoregressivegeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GenVC aims to show that zero-shot voice conversion—converting a source utterance into an unseen target voice without per-speaker retraining—can be built entirely from unlabeled speech, with no externally supervised speaker encoder in the loop. The system feeds a fixed-length style embedding from a Perceiver encoder plus phonetic tokens into an autoregressive Transformer language model that regenerates acoustic tokens, and the paper reports that this self-supervised recipe matches or beats leading supervised systems on speaker similarity while keeping naturalness competitive. The same autoregressive design, which does not copy the source utterance's timing and prosody, gives a strong anonymization side effect: in a VPC2024-style evaluation, GenVC raises the equal error rate for speaker re-identification to 27–29%, against about 9–12% for the parallel-conversion baselines. If the claim holds, voice conversion no longer needs supervised speaker-verification or ASR components to scale, and the same machinery can serve both voice cloning and voice privacy.

What carries the argument

The load-bearing component is the combination of a Perceiver encoder with a causal Transformer language model. The Perceiver encoder maps a variable-length acoustic prompt to a fixed-length style embedding via learned latent queries, so the model conditions generation on a compact speaker/style vector rather than on the source spectrogram. The causal LM then predicts two interleaved token streams—phonetic tokens from a ContentVec-based discrete VAE and acoustic tokens from a mel-spectrogram discrete VAE—using two prediction heads, with the acoustic loss weighted far more heavily. The autoregressive formulation is what breaks the frame-aligned copying of source prosody: acoustic tokens are generated token by token, conditioned on the style embedding and phonetic content, so temporal structure is regenerated rather than preserved. Training is fully self-supervised because the prompt and the clip come from the same utterance, so no external speaker label or speaker encoder is needed.

What would settle it

A reader could falsify the central claim by training GenVC as specified and testing conversion on pairs where the prompt speaker and the source speaker differ, then measuring whether the output is recognized as the target speaker; if speaker similarity to the target is no better than chance, the self-supervised disentanglement does not transfer to unseen speakers. A second check is to run speaker verification on the phonetic tokens: if their equal error rate falls well below the reported 48.5%, the content stream leaks speaker identity and the conversion is not cleanly zero-shot.

Watch

Extended reading notes

Core claim

On the paper's own terms, GenVC establishes that a single self-supervised pipeline can disentangle who is speaking from what is being said, and regenerate speech in a target voice with state-of-the-art similarity. Trained only on unlabeled utterances, with the audio prompt and the audio clip drawn from the same source utterance at random start points, the model learns to use the prompt to capture speaker-specific acoustic style and the phonetic tokens to carry content. At inference, given an unseen target voice as prompt and an unseen source utterance's phonetic tokens, the causal language model autoregressively predicts acoustic tokens, and a HiFiGAN vocoder reconstructs the waveform. In subjective listening, GenVC-Small reaches a similarity MOS of 3.72, ahead of FreeVC's 3.48, while naturalness remains competitive; in the VPC2024-style anonymization evaluation, GenVC achieves an EER of 27–29%, far above the 9–12% of the supervised baselines. The paper also presents evidence for the disentanglement: phonetic tokens give near-chance speaker-verification EER (48.5%), while the Perceiver output carries speaker information (12.2% EER with a PLDA backend).

Load-bearing premise

Everything rests on the leap from same-speaker training pairs, where the prompt and audio clip are cut from one utterance, to cross-speaker inference, where the prompt is an unseen target voice; the self-supervised loss never directly sees a different speaker as the target.

Editorial extensions

If this is right

Zero-shot voice conversion can be trained on thousands of hours of unlabeled, in-the-wild speech, since no ASR, ASV, or TTS labels are needed for the conversion pipeline.
Autoregressive generation gives a privacy advantage: source prosody is not preserved, so speaker re-identification drops to 27–29% EER compared with about 10% for parallel baselines.
The approach removes the need for a separately trained speaker encoder, simplifying the architecture while keeping similarity competitive or better.
Because the Perceiver embedding is a fixed-length vector, the same model can generate pseudo-voices for anonymization by sampling or altering the style vector rather than cloning an existing speaker.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the reported EER results replicate, GenVC-style autoregressive conversion could become a default building block for VoicePrivacy pipelines, replacing the frame-aligned conversion modules that leak prosody.
The same architecture could be applied to cross-lingual conversion or emotion transfer by redefining the style embedding to carry those attributes, though the paper does not test this.
The gap between GenVC-Small and GenVC-Large on similarity suggests data quality matters more than scale for the style path; a testable extension would be filtering large corpora for recording quality before fine-tuning.
Because the Perceiver embedding is trained only on about 2,300 LibriTTS speakers, its speaker-discriminative EER of 12.2% with PLDA may underestimate what the embedding can do; training the Perceiver on a much larger speaker pool could improve both cloning fidelity and anonymization strength.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: GenVC is a legitimate step for self-supervised zero-shot voice conversion. The architecture—ContentVec features through a DVAE into phonetic tokens, a Perceiver encoder producing a fixed-length style embedding, GPT-2 generating acoustic tokens, and a HiFiGAN vocoder—is a new combination worth knowing. The anonymization result is the most notable: EER of 27–29% versus 9–12% for the baselines is a large gap, measured with the VPC2024 pipeline. They also release code and checkpoints, and the subjective listening test uses significance testing, which many VC papers skip. This is a solid engineering contribution.

Soft spots, in proportion. First, the abstract claims naturalness 'on par with leading zero-shot approaches'; Table III doesn't support that. FreeVC's NMOS is 3.96, GenVC-Small 3.57, GenVC-Large 3.43. The authors note the baselines were trained on VCTK while GenVC used noisier data, but that explains the gap without closing it. The claim should be tempered to 'competitive but lower.' Second, there is an internal inconsistency in Section IV-B: the text first says both DVAEs use a codebook size of 512, then later says the phonetic codebook is 256 and the acoustic codebook is 1024. A reader cannot tell which numbers are correct. That needs fixing.

Third, the same-utterance training trick is the load-bearing assumption the paper doesn't fully address. During training, prompt and clip come from the same recording, so the Perceiver never has to discard content or channel cues—they are always consistent. At inference the prompt is a different utterance of an unseen speaker, often from a different recording environment. The disentanglement experiments show the Perceiver retains speaker information (EER 17.7% with ECAPA), but they don't measure content leakage or robustness to channel mismatch. The evaluation uses CMU Arctic and EMIME, which are clean read speech, so that mismatch isn't stressed. This is a missing experiment, not a demonstrated failure—the strong similarity scores on unseen EMIME speakers suggest the model generalizes at least in clean conditions—but a cross-dataset conversion test with mismatched recording conditions would settle it. The limitation section acknowledges robustness to recording environments is open.

Fourth, minor: scaling from 452 hours to roughly 46k hours did not improve similarity or naturalness much, undercutting the 'large-scale training' story. The authors attribute this to data quality, but that deserves more than a hand-wave.

Who this is for: people working on LM-based VC, speech anonymization, and self-supervised representation learning. It deserves a serious referee; the reviewer should push for the codebook-size clarification, a tempered abstract, and ideally the cross-dataset experiment. I would accept it for peer review.

Referee Report

3 major / 4 minor

Summary. GenVC proposes a fully self-supervised zero-shot voice conversion system that avoids externally supervised speaker encoders. It trains discrete VAE tokenizers for phonetic features (from ContentVec) and acoustic features (from mel-spectrograms), then trains a GPT-2-style autoregressive language model to predict acoustic tokens conditioned on a fixed-length Perceiver-derived style embedding and a sequence of phonetic tokens; a HiFiGAN vocoder reconstructs the waveform. Two model sizes are trained: GenVC-Small on LibriTTS and GenVC-Large on additional CommonVoice/MLS data. Evaluations use EMIME/CMU Arctic conversion pairs, objective WavLM and UTMOS metrics, subjective NMOS/SMOS listening tests, the VPC2024 anonymization pipeline, and speaker-disentanglement probes. The reported results show GenVC-Small achieving the highest speaker similarity (SMOS 3.72) and both GenVC variants yielding much higher anonymization EER (27--29%) than the baselines (9--12%), while subjective naturalness is lower than FreeVC (NMOS 3.43--3.57 vs. 3.96).

Significance. The main contribution is architectural: replacing supervised speaker encoders with a self-supervised Perceiver-style style embedding and an autoregressive LM, which enables training on large unlabeled corpora and disrupts source prosody for anonymization. The paper also ships code, model checkpoints, and audio samples, and it uses open-source evaluation tools with statistical significance tests, which strengthens reproducibility. The anonymization EER gap is striking and, if confirmed under matched conditions, would be practically important. However, the state-of-the-art claim is weakened by the naturalness gap and by the lack of a direct test of the style embedding's invariance to content and recording conditions; the significance of the result therefore depends on additional evidence.

major comments (3)

[Section III-A, Section IV-B, Table V] The zero-shot generalization claim is not yet fully supported because training uses the audio prompt and the audio clip from the same source utterance (Section III-A: 'both the audio prompt and the audio clip are extracted from the same source utterance using random start points and segment lengths'). This allows the style embedding E_style to encode content, prosody, or recording-channel cues that are correlated at training time but absent or misleading when converting to an unseen target speaker at inference. The disentanglement analysis in Table V shows that Perceiver features retain speaker identity (e.g., EER 12.2% with PLDA), but it does not measure content leakage or robustness to channel/domain mismatch, and the conversion evaluation uses clean read-speech corpora (CMU Arctic/EMIME). Please add a direct test such as cross-domain conversion (e.g., VoxCeleb or noisy source to EMIME target, or vice versa) or an ablation varying whether prompt and clip content overlap, and report content preservation and style-embedding leakage.
[Abstract, Table III] The abstract's claim that GenVC achieves 'naturalness on par with leading zero-shot approaches' is contradicted by the subjective results in Table III: GenVC-Small and GenVC-Large have NMOS 3.57 +/- 0.09 and 3.43 +/- 0.09, respectively, while FreeVC has 3.96 +/- 0.07, and the confidence intervals do not overlap. This is a material overstatement. Please revise the abstract and conclusion to state that naturalness is competitive but lower than the best baseline, or provide a matched comparison that supports parity.
[Section V-A, Table III] The comparison against YourTTS, FreeVC, and NeuralVC is confounded by training-data mismatch: the baselines were trained on the high-quality VCTK dataset, while GenVC was trained on LibriTTS/CommonVoice/MLS, and the paper itself acknowledges that data quality may affect NMOS (Section V-A2). Because the 'state-of-the-art' claim rests partly on these cross-system MOS/SMOS numbers, please either include a baseline trained on comparable data, perform a controlled evaluation with the same source/target pairs and a shared vocoder where possible, or explicitly qualify the claim as 'competitive under different training-data regimes'.

minor comments (4)

[Section IV-B, Phase 1] The text states that 'both models employ a codebook size of 512', but then reports the phonetic codebook size Ko as 256 and the acoustic codebook size Ka as 1024; please correct this internal inconsistency.
[Section V-A2] The subjective evaluation uses 13 listeners, but the paper reports only pairwise Mann-Whitney p-values; please also report the number of ratings, inter-rater variability, and whether any correction for multiple comparisons was applied.
[Equations (1) and (2)] The notation os:e and as:j-1 is not defined; please clarify that these denote the start token to the current position in the token sequence.
[Abstract] The abstract repeats the word 'Abstract' at the start of the text; remove the duplicate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GenVC's training is a reconstruction objective and its central claims are evaluated on held-out external benchmarks.

full rationale

The paper's derivation chain does not reduce to its inputs. Phase 2 training (Section III-A) conditions the causal LM on a Perceiver style embedding extracted from a prompt and phonetic tokens from a clip drawn from the same source utterance; the loss (Eq. 3) is a log-likelihood over phonetic and acoustic tokens, so the model is trained to reconstruct tokens rather than to fit any target metric. No fitted parameter is renamed as a prediction: the headline SMOS, NMOS, WavLM similarity, UTMOS, and VPC2024 EER results (Tables II-IV) are measured on held-out EMIME, LibriSpeech, and VoxCeleb material using independent tools (WavLM-base-sv, UTMOSv2, ECAPA-TDNN). The only self-citations (e.g., refs. [9], [10]) are contextual background statements, not load-bearing premises, and no uniqueness theorem or ansatz is imported from the authors' prior work. The same-utterance prompt/clip setup is a legitimate training-time shortcut that creates a potential train/inference mismatch if the Perceiver embedding leaks content or environment information, but that is an empirical generalization and robustness risk, not a circularity; the paper's own disentanglement experiments (Table V) and anonymization results are externally falsifiable. Therefore no circularity is found.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the training-inference transfer assumption, on the speaker-independence of ContentVec features, and on the validity of proxy evaluation metrics. No new physical entities are introduced; the main ledger entries are hand-set hyperparameters and domain assumptions about self-supervised disentanglement.

free parameters (4)

loss weights alpha, beta = alpha=0.01, beta=1
Chosen by hand to balance phonetic and acoustic token losses; directly affects disentanglement quality.
inference sampling parameters = temperature=0.85, length_penalty=1.0, repetition_penalty=2.0, top_k=15, top_p=0.85
Tuned for generation quality; not fixed by theory.
codebook sizes for phonetic and acoustic DVAEs = phonetic 256, acoustic 1024 (paper also states 512)
Capacity of the discrete tokenizers; chosen empirically and stated inconsistently in Section IV-B.
Perceiver latent query count = 32
Determines the fixed-length style embedding size; chosen by hand.

assumptions (4)

domain assumption ContentVec features are sufficiently phonetic and speaker-independent.
GenVC builds phonetic tokens from ContentVec; if ContentVec retained speaker information, disentanglement would fail.
domain assumption Same-utterance prompt and clip training transfers to cross-speaker inference.
Section III-A; the central generalization assumption of the method.
domain assumption Reconstruction loss without external speaker labels yields useful disentanglement.
Core hypothesis; partially validated by EER experiments on VoxCeleb and conversion tests.
domain assumption Evaluation metrics (WavLM, UTMOS, ECAPA, VPC2024 pipeline) are valid proxies for similarity, naturalness, and privacy.
Standard tools in the field; subjective validation uses only 13 raters.

how reviews work

0 comments

Cite this review

Pith. "Pith review of GenVC: Self-Supervised Zero-Shot Voice Conversion." pith.science (2026). https://pith.science/paper/RUSDF3FZ

@misc{pith2026250204519,
  author       = {Pith},
  title        = {Pith review of: GenVC: Self-Supervised Zero-Shot Voice Conversion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RUSDF3FZ}},
  note         = {Machine review of arXiv:2502.04519}
}

read the original abstract

Most current zero-shot voice conversion methods rely on externally supervised components, particularly speaker encoders, for training. To explore alternatives that eliminate this dependency, this paper introduces GenVC, a novel framework that disentangles speaker identity and linguistic content from speech signals in a self-supervised manner. GenVC leverages speech tokenizers and an autoregressive, Transformer-based language model as its backbone for speech generation. This design supports large-scale training while enhancing both source speaker privacy protection and target speaker cloning fidelity. Experimental results demonstrate that GenVC achieves notably higher speaker similarity, with naturalness on par with leading zero-shot approaches. Moreover, due to its autoregressive formulation, GenVC introduces flexibility in temporal alignment, reducing the preservation of source prosody and speaker-specific traits, and making it highly effective for voice anonymization.

Figures

Figures reproduced from arXiv: 2502.04519 by the authors.

**Figure 1.** System architecture and training scheme of GenVC: Phase 1 involves the Discrete VAEs for audio tokenization. Phase 2 has a causal Transformer-based [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Universal Speech Content Factorization
eess.AS 2026-03 conditional novelty 5.0 of 10

A universal least-squares speech-to-content map plus few-second speaker transforms yields open-set, low-rank, timbre-suppressed features competitive for zero-shot VC and TTS.

Reference graph

Works this paper leans on

77 extracted references · 61 canonical work pages · cited by 1 Pith paper

[1]

AutoVC: Zero-Shot V oice Style Transfer with Only Autoencoder Loss,

K. Qian, Y . Zhang, S. Chang, X. Yang, and M. Hasegawa-Johnson, “AutoVC: Zero-Shot V oice Style Transfer with Only Autoencoder Loss,” in International Conference on Machine Learning , vol. 97, 2019, pp. 5210–5219

work page 2019
[2]

GAZEV: GAN-Based Zero-Shot V oice Conversion Over Non-Parallel Speech Corpus,

Z. Zhang, B. He, and Z. Zhang, “GAZEV: GAN-Based Zero-Shot V oice Conversion Over Non-Parallel Speech Corpus,” in Interspeech 2020 , 2020, pp. 791–795

work page 2020
[3]

SIG-VC: A Speaker Information Guided Zero-Shot V oice Conversion System for Both Human Beings and Machines,

H. Zhang, Z. Cai, X. Qin, and M. Li, “SIG-VC: A Speaker Information Guided Zero-Shot V oice Conversion System for Both Human Beings and Machines,” in IEEE International Conference on Acoustics, Speech and Signal Processing , 2022, pp. 6567–65 571

work page 2022
[4]

An Overview of V oice Conversion and Its Challenges: From Statistical Modeling to Deep Learning,

B. Sisman, J. Yamagishi, S. King, and H. Li, “An Overview of V oice Conversion and Its Challenges: From Statistical Modeling to Deep Learning,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 132–157, 2020

work page 2020
[5]

Prosodic Features for Speaker Veri- fication,

L. Mary and B. Yegnanarayana, “Prosodic Features for Speaker Veri- fication,” in Ninth International Conference on Spoken Language Pro- cessing, 2006

work page 2006
[6]

FreeVC: Towards High-Quality Text- Free One-Shot V oice Conversion,

J. Li, W. Tu, and L. Xiao, “FreeVC: Towards High-Quality Text- Free One-Shot V oice Conversion,” in IEEE International Conference on Acoustics, Speech and Signal Processing , 2023, pp. 1–5

work page 2023
[7]

The Database and Benchmark For the Source Speaker Tracing Challenge 2024,

Z. Li, Y . Lin, T. Yao, H. Suo, P. Zhang, Y . Ren, Z. Cai, H. Nishizaki, and M. Li, “The Database and Benchmark For the Source Speaker Tracing Challenge 2024,” in IEEE Spoken Language Technology Workshop (SLT), 2024, pp. 1254–1261

work page 2024
[8]

NeuralVC: Any-to-Any V oice Conver- sion Using Neural Networks Decoder For Real-Time V oice Conversion,

D. Cao, Z. Zhang, and J. Zhang, “NeuralVC: Any-to-Any V oice Conver- sion Using Neural Networks Decoder For Real-Time V oice Conversion,” IEEE Signal Processing Letters , 2024

work page 2024

Show all 77 references

[9]

Identifying Source Speakers for V oice Conversion based Spoofing Attacks on Speaker Verification Systems,

D. Cai, Z. Cai, and M. Li, “Identifying Source Speakers for V oice Conversion based Spoofing Attacks on Speaker Verification Systems,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2023, pp. 1–5

2023
[10]

Privacy Versus Emotion Preservation Trade-Offs in Emotion-Preserving Speaker Anonymization,

Z. Cai, H. L. Xinyuan, A. Garg, L. P. Garc ´ıa-Perera, K. Duh, S. Khudan- pur, N. Andrews, and M. Wiesner, “Privacy Versus Emotion Preservation Trade-Offs in Emotion-Preserving Speaker Anonymization,” in IEEE Spoken Language Technology Workshop (SLT) , 2024, pp. 409–414

2024
[11]

Zero-Shot V oice Conversion with Adjusted Speaker Embeddings and Simple Acoustic Features,

Z. Tan, J. Wei, J. Xu, Y . He, and W. Lu, “Zero-Shot V oice Conversion with Adjusted Speaker Embeddings and Simple Acoustic Features,” in IEEE International Conference on Acoustics, Speech and Signal Processing, 2021, pp. 5964–5968

2021
[12]

YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot V oice Conversion for Everyone,

E. Casanova, J. Weber, C. D. Shulby, A. C. Junior, E. G ¨olge, and M. A. Ponti, “YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot V oice Conversion for Everyone,” inInternational Conference on Machine Learning , 2022, pp. 2709–2720

2022
[13]

Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations,

H.-S. Choi, J. Lee, W. Kim, J. Lee, H. Heo, and K. Lee, “Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations,” Advances in Neural Information Processing Systems , vol. 34, pp. 16 251–16 265, 2021

2021
[14]

NANSY++: Unified V oice Synthesis with Neural Analysis and Synthesis,

H.-S. Choi, J. Yang, J. Lee, and H. Kim, “NANSY++: Unified V oice Synthesis with Neural Analysis and Synthesis,” in International Confer- ence on Learning Representations , 2023

2023
[15]

Better Speech Synthesis through Scaling,

J. Betker, “Better Speech Synthesis through Scaling,” arXiv preprint arXiv:2305.07243, 2023

2023 arXiv
[16]

LM-VC: Zero-Shot V oice Conversion via Speech Generation Based on Language Models,

Z. Wang, Y . Chen, L. Xie, Q. Tian, and Y . Wang, “LM-VC: Zero-Shot V oice Conversion via Speech Generation Based on Language Models,” IEEE Signal Processing Letters , 2023

2023
[17]

AudioGen: Textually Guided Audio Generation,

F. Kreuk, G. Synnaeve, A. Polyak, U. Singer, A. D ´efossez, J. Copet, D. Parikh, Y . Taigman, and Y . Adi, “AudioGen: Textually Guided Audio Generation,” in International Conference on Learning Representations , 2023

2023
[18]

Towards Audio Language Modeling-An Overview,

H. Wu, X. Chen, Y .-C. Lin, K.-w. Chang, H.-L. Chung, A. H. Liu, and H.-y. Lee, “Towards Audio Language Modeling-An Overview,” arXiv preprint arXiv:2402.13236, 2024

2024 arXiv
[19]

AudioLM: A Language Modeling Approach to Audio Generation,

Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Shar- ifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchiet al., “AudioLM: A Language Modeling Approach to Audio Generation,” IEEE/ACM transactions on audio, speech, and language processing , vol. 31, pp. 2...

2023
[20]

SoundStream: An End-to-End Neural Audio Codec,

N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasac- chi, “SoundStream: An End-to-End Neural Audio Codec,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 30, pp. 495–507, 2021

2021
[21]

Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision,

E. Kharitonov, D. Vincent, Z. Borsos, R. Marinier, S. Girgin, O. Pietquin, M. Sharifi, M. Tagliasacchi, and N. Zeghidour, “Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision,” Transactions of the Association for Computational Linguistics , vol. 11, p...

2023
[22]

V oiceCraft: Zero-shot speech editing and text-to-speech in the wild,

P. Peng, P.-Y . Huang, S.-W. Li, A. Mohamed, and D. Harwath, “V oiceCraft: Zero-shot speech editing and text-to-speech in the wild,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , 2024, pp. 12 442–12 462

2024
[23]

CosyV oice 2: Scalable Streaming Speech Syn- thesis with Large Language Models,

Z. Du, Y . Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y . Yang, C. Gao, H. Wang et al., “CosyV oice 2: Scalable Streaming Speech Syn- thesis with Large Language Models,” arXiv preprint arXiv:2412.10117 , 2024

2024 arXiv
[24]

CosyV oice 3: Towards In-the-wild Speech Generation via Scaling-up and Post- training,

Z. Du, C. Gao, Y . Wang, F. Yu, T. Zhao, H. Wang, X. Lv, H. Wang, C. Ni, X. Shi, K. An, G. Yang, Y . Li, Y . Chen, Z. Gao, Q. Chen, Y . Gu, M. Chen, Y . Chen, S. Zhang, W. Wang, and J. Ye, “CosyV oice 3: Towards In-the-wild Speech Generation via Scaling-up and Post- training,”...

2025 arXiv
[25]

Seed-TTS: A Family of High-Quality Versatile Speech Generation Models,

P. Anastassiou, J. Chen, J. Chen, Y . Chen, Z. Chen, Z. Chen, J. Cong, L. Deng, C. Ding, L. Gao, M. Gong, P. Huang, Q. Huang, Z. Huang, Y . Huo, D. Jia, C. Li, F. Li, H. Li, J. Li, X. Li, X. Li, L. Liu, S. Liu, S. Liu, X. Liu, Y . Liu, Z. Liu, L. Lu, J. Pan, X. Wang, Y . Wang,...

2024 arXiv
[26]

Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers,

S. Chen, C. Wang, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei, “Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers,” IEEE Transactions on Audio, Speech and Language Processing , pp. 1–15, 2025

2025
[27]

High Fidelity Neural Audio Compression,

A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High Fidelity Neural Audio Compression,” Transactions on Machine Learning Research , 2023

2023
[28]

Speak Foreign Languages with Your Own V oice: Cross-Lingual Neural Codec Language Modeling,

Z. Zhang, L. Zhou, C. Wang, S. Chen, Y . Wu, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li et al. , “Speak Foreign Languages with Your Own V oice: Cross-Lingual Neural Codec Language Modeling,”arXiv preprint arXiv:2303.03926, 2023

2023 arXiv
[29]

V ALL-E 2: Neural Codec Language Models are Hu- man Parity Zero-Shot Text to Speech Synthesizers,

S. Chen, S. Liu, L. Zhou, Y . Liu, X. Tan, J. Li, S. Zhao, Y . Qian, and F. Wei, “V ALL-E 2: Neural Codec Language Models are Hu- man Parity Zero-Shot Text to Speech Synthesizers,” arXiv preprint arXiv:2406.05370, 2024

2024 arXiv
[30]

CosyV oice: A Scalable Multilingual Zero-Shot Text-to-Speech Synthesizer Based on Supervised Semantic Tokens,

Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y . Yang, H. Hu, S. Zheng, Y . Gu, Z. Ma et al. , “CosyV oice: A Scalable Multilingual Zero-Shot Text-to-Speech Synthesizer Based on Supervised Semantic Tokens,” arXiv preprint arXiv:2407.05407 , 2024

2024 arXiv
[31]

MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer,

Y . Wang, H. Zhan, L. Liu, R. Zeng, H. Guo, J. Zheng, Q. Zhang, X. Zhang, S. Zhang, and Z. Wu, “MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer,” in International Confer- ence on Learning Representations , 2025

2025
[32]

Simple and Controllable Music Generation,

J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. D ´efossez, “Simple and Controllable Music Generation,” in Neural Information Processing Systems , 2023

2023
[33]

Moshi: A Speech-Text Foundation Model for Real-Time Dialogue,

A. D ´efossez, L. Mazar ´e, M. Orsini, A. Royer, P. P ´erez, H. J ´egou, E. Grave, and N. Zeghidour, “Moshi: A Speech-Text Foundation Model for Real-Time Dialogue,” arXiv preprint arXiv:2410.00037 , 2024

2024 arXiv
[34]

XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model,

E. Casanova, K. Davis, E. G ¨olge, G. G ¨oknar, I. Gulea, L. Hart, A. Aljafari, J. Meyer, R. Morais, S. Olayemi, and J. Weber, “XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model,” inInterspeech 2024, 2024

2024
[35]

StreamV oice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot V oice Conversion,

Z. Wang, Y . Chen, X. Wang, L. Xie, and Y . Wang, “StreamV oice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot V oice Conversion,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , 2024, pp. 7328–7338

2024
[36]

Vevo: Control- lable Zero-Shot V oice Imitation with Self-Supervised Disentanglement,

X. Zhang, X. Zhang, K. Peng, Z. Tang, V . Manohar, Y . Liu, J. Hwang, D. Li, Y . Wang, J. Chan, Y . Huang, Z. Wu, and M. Ma, “Vevo: Control- lable Zero-Shot V oice Imitation with Self-Supervised Disentanglement,” in International Conference on Learning Representations , 2025

2025
[37]

The V oicePri- vacy 2024 Challenge Evaluation Plan,

N. Tomashenko, X. Miao, P. Champion, S. Meyer, X. Wang, E. Vincent, M. Panariello, N. Evans, J. Yamagishi, and M. Todisco, “The V oicePri- vacy 2024 Challenge Evaluation Plan,”arXiv preprint arXiv:2404.02677, 2024

2024 arXiv
[38]

Self-Supervised Speech Representations are More Pho- netic than Semantic,

K. Choi, A. Pasad, T. Nakamura, S. Fukayama, K. Livescu, and S. Watanabe, “Self-Supervised Speech Representations are More Pho- netic than Semantic,” in Interspeech 2024 , 2024, pp. 4578–4582

2024
[39]

S2VC: A Frame- work for Any-to-Any V oice Conversion with Self-Supervised Pretrained Representations,

J.-H. Lin, Y . Y . Lin, C.-M. Chien, and H. yi Lee, “S2VC: A Frame- work for Any-to-Any V oice Conversion with Self-Supervised Pretrained Representations,” in Interspeech 2021 , 2021, pp. 836–840

2021
[40]

S3PRL-VC: Open-Source V oice Conversion Framework with Self-Supervised Speech Representations,

W.-C. Huang, S.-W. Yang, T. Hayashi, H.-Y . Lee, S. Watanabe, and T. Toda, “S3PRL-VC: Open-Source V oice Conversion Framework with Self-Supervised Speech Representations,” in IEEE International Confer- ence on Acoustics, Speech and Signal Processing , 2022, pp. 6552–6556

2022
[41]

Neural Discrete Representation Learning,

A. Van Den Oord, O. Vinyals et al. , “Neural Discrete Representation Learning,” Advances in Neural Information Processing Systems , vol. 30, 2017

2017
[42]

Attention is All you Need,

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is All you Need,” in Advances in Neural Information Processing Systems , vol. 30, 2017

2017
[43]

Flamingo: A Visual Language Model for Few-Shot Learning,

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al. , “Flamingo: A Visual Language Model for Few-Shot Learning,” Advances in Neural Information Processing Systems , vol. 35, pp. 23 716–23 736, 2022

2022
[44]

NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers,

K. Shen, Z. Ju, X. Tan, E. Liu, Y . Leng, L. He, T. Qin, S. Zhao, and J. Bian, “NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers,” in International Confer- ence on Learning Representations , 2024

2024
[45]

HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis,

J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis,” in Advances in Neural Information Processing Systems , vol. 33, 2020, pp. 17 022– 17 033

2020
[46]

LibriTTS: A Corpus Derived from LibriSpeech for Text- to-Speech,

H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “LibriTTS: A Corpus Derived from LibriSpeech for Text- to-Speech,” in Proc. Interspeech 2019 , 2019, pp. 1526–1530

2019
[47]

Common V oice: A Massively-Multilingual Speech Corpus,

R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common V oice: A Massively-Multilingual Speech Corpus,” in Proceedings of the Twelfth Language Resources and Evaluation Conference , 2020, pp. 4218–4222

2020
[48]

MLS: A Large-Scale Multilingual Dataset for Speech Research,

V . Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, “MLS: A Large-Scale Multilingual Dataset for Speech Research,” in Interspeech 2020, 2020, pp. 2757–2761

2020
[49]

CMU ARCTIC Databases for Speech Synthesis,

J. Kominek, A. W. Black, and V . Ver, “CMU ARCTIC Databases for Speech Synthesis,” 2003

2003
[50]

The EMIME Mandarin Bilingual Database,

M. Wester and H. Liang, “The EMIME Mandarin Bilingual Database,” The University of Edinburgh, Tech. Rep., 2011

2011
[51]

Librispeech: An ASR Corpus Based on Public Domain Audio Books,

V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR Corpus Based on Public Domain Audio Books,” in IEEE International Conference on Acoustics, Speech and Signal Processing , 2015, pp. 5206–5210

2015
[52]

V oxCeleb: A Large-Scale Speaker Identification Dataset,

A. Nagrani, J. S. Chung, and A. Zisserman, “V oxCeleb: A Large-Scale Speaker Identification Dataset,” in Proc. Interspeech 2017 , 2017, pp. 2616–2620

2017
[53]

V oxCeleb2: Deep Speaker Recognition,

J. S. Chung, A. Nagrani, and A. Zisserman, “V oxCeleb2: Deep Speaker Recognition,” in Interspeech 2018 , 2018, pp. 1086–1090

2018
[54]

ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers,

K. Qian, Y . Zhang, H. Gao, J. Ni, C.-I. Lai, D. Cox, M. Hasegawa- Johnson, and S. Chang, “ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers,” in International Conference on Machine Learning , 2022, pp. 18 003–18 017

2022
[55]

Language Models Are Unsupervised Multitask Learners,

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al. , “Language Models Are Unsupervised Multitask Learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019

2019
[56]

Multi-Scale Sub-Band Constant-Q Transform Discriminator for High-Fidelity V ocoder,

Y . Gu, X. Zhang, L. Xue, and Z. Wu, “Multi-Scale Sub-Band Constant-Q Transform Discriminator for High-Fidelity V ocoder,” in IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing , 2024, pp. 10 616–10 620

2024
[57]

WavLM: Large-Scale Self-Supervised Pre- Training for Full Stack Speech Processing,

S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao et al., “WavLM: Large-Scale Self-Supervised Pre- Training for Full Stack Speech Processing,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1505–1518, 2022

2022
[58]

The T05 System for The V oiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech,

K. Baba, W. Nakata, Y . Saito, and H. Saruwatari, “The T05 System for The V oiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech,” in IEEE Spoken Language Technology Workshop (SLT) , 2024

2024
[59]

Bias and Statistical Significance in Evaluating Speech Synthesis with Mean Opinion Scores,

A. Rosenberg and B. Ramabhadran, “Bias and Statistical Significance in Evaluating Speech Synthesis with Mean Opinion Scores,” in Interspeech 2017, 2017, pp. 3976–3980

2017
[60]

Good Practices for Evaluation of Synthesized Speech,

E. Cooper, S. L. Maguer, E. Klabbers, and J. Yamagishi, “Good Practices for Evaluation of Synthesized Speech,” arXiv preprint arXiv:2503.03250, 2025

2025
[61]

ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,

B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,” in Interspeech 2020, 2020, pp. 3830–3834

2020
[62]

Scaling Laws for Neural Language Models,

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling Laws for Neural Language Models,” arXiv preprint arXiv:2001.08361 , 2020

2001 arXiv
[63]

Deep Residual Learning for Image Recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 770–778. APPENDIX A. Impact Statement This paper presents work aimed at advancing the field of V o...

2016
[64]

The first two encoder convolutional layers upsample the input dimensionality to the DV AE hidden dimension of 1024

DVAE: Our DV AE encoder and decoder are both made up of a sequence of 1-D convolutional layers. The first two encoder convolutional layers upsample the input dimensionality to the DV AE hidden dimension of 1024. They are followed by 3 Resblocks [63] and a final 1-D convolution...
[65]

The learned queries attends to all input frames, transforming them into fixed- length representations

Perceiver Encoder: Our Perceiver encoder module comprises learned latent queries and a cross-attention mechanism. The learned queries attends to all input frames, transforming them into fixed- length representations. The encoder consists of four multi-head attention blocks, ea...
[66]

1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5 and 5 are allowed on the half-point scale

Naturalness: In this experiment, please listen to the speech sample and rate their naturalness on a scale from 1 (Bad) to 5 (Excellent), with increments of 0.5. 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5 and 5 are allowed on the half-point scale. The scale is defined as follows:
[67]

Bad: Very unnatural speech, completely unrecognizable as hu- man speech
[68]

Poor: Noticeably unnatural speech with many artifacts and discontinuities
[69]

Fair: Moderately unnatural speech with noticeable artifacts, but not entirely unnatural
[70]

Good: Mostly natural speech with only minor artifacts that are noticeable upon careful listening
[71]

Excellent: Perfectly natural speech, indistinguishable from hu- man speech
[72]

Use a scale from 1 (Bad) to 5 (Excellent), with increments of 0.5

Similarity: In this experiment, you will listen to pairs of speech samples and rate how similar the second sample sounds to the reference speech in terms of speaker voice, speaking style, and environmental background, regardless of the content (which will differ). Use a scale ...
[73]

Bad: The two samples sound completely different with barely any similarity
[74]

For example, the speakers may differ in gender or pitch (e.g., a high-pitched female voice vs

Poor: The two samples have some resemblance, but the dif- ferences are significant. For example, the speakers may differ in gender or pitch (e.g., a high-pitched female voice vs. a low-pitched male voice). Speaking style and environmental background also differ noticeably
[75]

It’s clear the speakers are the same gender, but their voices are distinctly different, and their speaking styles differ somewhat

Fair: The two samples sound somewhat similar, but there are still noticeable differences. It’s clear the speakers are the same gender, but their voices are distinctly different, and their speaking styles differ somewhat
[76]

The speaker voices are close, and the speaking styles largely match

Good: The given utterance sounds quite similar to the reference utterance, with only minor differences noticeable upon close listening. The speaker voices are close, and the speaking styles largely match
[77]

The timbre of the speakers is the same, their speaking styles match perfectly, and the environmental background is similar, including any acoustic noise

Excellent: The given utterance sounds identical to the reference utterance, with no perceivable differences. The timbre of the speakers is the same, their speaking styles match perfectly, and the environmental background is similar, including any acoustic noise

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

AutoVC: Zero-Shot V oice Style Transfer with Only Autoencoder Loss,

K. Qian, Y . Zhang, S. Chang, X. Yang, and M. Hasegawa-Johnson, “AutoVC: Zero-Shot V oice Style Transfer with Only Autoencoder Loss,” in International Conference on Machine Learning , vol. 97, 2019, pp. 5210–5219

work page 2019

[2] [2]

GAZEV: GAN-Based Zero-Shot V oice Conversion Over Non-Parallel Speech Corpus,

Z. Zhang, B. He, and Z. Zhang, “GAZEV: GAN-Based Zero-Shot V oice Conversion Over Non-Parallel Speech Corpus,” in Interspeech 2020 , 2020, pp. 791–795

work page 2020

[3] [3]

SIG-VC: A Speaker Information Guided Zero-Shot V oice Conversion System for Both Human Beings and Machines,

H. Zhang, Z. Cai, X. Qin, and M. Li, “SIG-VC: A Speaker Information Guided Zero-Shot V oice Conversion System for Both Human Beings and Machines,” in IEEE International Conference on Acoustics, Speech and Signal Processing , 2022, pp. 6567–65 571

work page 2022

[4] [4]

An Overview of V oice Conversion and Its Challenges: From Statistical Modeling to Deep Learning,

B. Sisman, J. Yamagishi, S. King, and H. Li, “An Overview of V oice Conversion and Its Challenges: From Statistical Modeling to Deep Learning,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 132–157, 2020

work page 2020

[5] [5]

Prosodic Features for Speaker Veri- fication,

L. Mary and B. Yegnanarayana, “Prosodic Features for Speaker Veri- fication,” in Ninth International Conference on Spoken Language Pro- cessing, 2006

work page 2006

[6] [6]

FreeVC: Towards High-Quality Text- Free One-Shot V oice Conversion,

J. Li, W. Tu, and L. Xiao, “FreeVC: Towards High-Quality Text- Free One-Shot V oice Conversion,” in IEEE International Conference on Acoustics, Speech and Signal Processing , 2023, pp. 1–5

work page 2023

[7] [7]

The Database and Benchmark For the Source Speaker Tracing Challenge 2024,

Z. Li, Y . Lin, T. Yao, H. Suo, P. Zhang, Y . Ren, Z. Cai, H. Nishizaki, and M. Li, “The Database and Benchmark For the Source Speaker Tracing Challenge 2024,” in IEEE Spoken Language Technology Workshop (SLT), 2024, pp. 1254–1261

work page 2024

[8] [8]

NeuralVC: Any-to-Any V oice Conver- sion Using Neural Networks Decoder For Real-Time V oice Conversion,

D. Cao, Z. Zhang, and J. Zhang, “NeuralVC: Any-to-Any V oice Conver- sion Using Neural Networks Decoder For Real-Time V oice Conversion,” IEEE Signal Processing Letters , 2024

work page 2024