Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

Enhancing Expressive Voice Conversion with Discrete Pitch-Conditioned Flow Matching Model

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PFlow-VC conditions a flow-matching Mel-spectrogram decoder on speaker-normalized discrete pitch tokens and target-speaker timbre prompts, claiming to transfer both timbre and emotion in zero-shot voice conversion.

desk verdict A novel discrete-pitch-conditioned flow matching VC paper with solid ablations, but the emotion-transfer mechanism is under-supported by a train/inference mismatch in the prompting strategy. read the letter →

arxiv 2502.05471 v1 pith:4ZY76P52 submitted 2025-02-08 cs.SD eess.AS

Jialong Zuo , Shengpeng Ji , Minghui Fang , Ziyue Jiang , Xize Cheng , Qian Yang , Wenrui Liu , Guangyan Zhang
show 3 more authors
Zehai Tu Yiwen Guo Zhou Zhao
This is my paper · ORCID
classification cs.SDeess.AS
keywords expressivevoiceconversiondiscretepitchtokensflowmatchingVQVAEzero-shotemotionstyletransfertimbresimilarityMel-spectrogramsynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a voice conversion model can carry a target speaker's emotion and prosody, not just their timbre, if the target's pitch is turned into discrete tokens and used as an in-context prompt for a flow-matching synthesizer. Most voice conversion systems separate linguistic content from speaker identity but leave prosody tied to the source speaker; PFlow-VC is designed to transplant the target speaker's pitch pattern instead. The model is trained on LibriTTS, then tested on unseen speakers and on the emotional ESD corpus. The reported result is a better emotion-consistency score (0.725 versus 0.650 for the best baseline) and competitive-to-better timbre similarity (SECS 0.920), which amounts to saying that expressive style transfer and zero-shot timbre cloning can live in one decoder.

What carries the argument

The load-bearing objects are two. First, a pretrained pitch VQVAE that quantizes speaker-mean normalized log F0, defined as $\log f - \mathbb{E}[\log f']$ over the speaker's voiced frames, into a 64-entry codebook of 128-dimensional vectors, producing discrete pitch tokens at 25 Hz; this is what separates pitch shape from pitch level. Second, a masked conditional flow matching Mel-spectrogram decoder: during training a random continuous segment of pitch tokens is masked, forcing the decoder to infer missing pitch from context; at inference the target speaker's pitch tokens are prepended as a prompt. The decoder is an optimal-transport conditional flow matching network that regresses a vector field with the loss $\mathbb{E}_{t,q(x_1),p_0(x_0)} \|u^{\mathrm{OT}}_t(\phi^{\mathrm{OT}}_t(x)|x_1) - v_t(\phi^{\mathrm{OT}}_t(x)|\theta)\|^2$, conditioned on the global speaker embedding $e_s$, semantic tokens $\{\mu_t\}_{1:T}$, and masked pitch prompt $\tilde{p}_1$. A multi-scale timbre encoder built on ECAPA-TDNN supplies both an attentive-statistic-pooling global embedding and 64 trainable query vectors producing time-varying timbre tokens via cross-attention; HuBERT semantic tokens quantized with k-means provide the speaker-invariant content stream, and a pretrained HiFi-GAN vocoder turns the generated Mel spectrogram into audio. The combination is what lets one decoder handle both timbre cloning and pitch-driven emotional style transfer.

What would settle it

Replace the target pitch-token prompt in the inference pipeline with (a) the source utterance's own pitch tokens and (b) a randomly permuted version of the target pitch tokens, keeping the timbre prompt fixed, and measure the emotion-consistency score on the same ESD test set. If either condition keeps the score near the reported 0.725, the emotion gain is not caused by the discrete pitch prompt; if the score falls toward the ablation level of the model without pitch tokens, the mechanism is confirmed.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that prosody can be treated as a discrete, speaker-independent sequence and used to prompt a generative decoder, rather than being estimated implicitly from Mel spectrograms or predicted by a separate diffusion pitch generator. The pitch VQVAE is trained on speaker-mean normalized log F0, so each speaker's average pitch level is removed before quantization; this yields tokens that are meant to carry contour, range, and emotional inflection without speaker identity. The masked training of the conditional flow matching decoder makes it learn to reconstruct pitch tokens from surrounding context, and at inference the target speaker's semantic tokens, pitch tokens, and timbre embedding are prepended as a prompt while the source pitch is replaced by null tokens. The paper reports that this design transfers emotions at 0.725 emotion-consistency and achieves a SECS of 0.920 on zero-shot conversion, and argues that the combination of global and time-varying timbre tokens is what preserves speaker identity in that same decoder.

Load-bearing premise

The entire transfer effect rests on the assumption that the pitch VQVAE's discrete tokens, computed from a target speaker's utterance after subtracting that speaker's mean log F0, are speaker-independent enough that a model trained with random token masking can use an unseen speaker's pitch-token prompt to impose emotional intonation on a different speaker's words, without any alignment of duration or global pitch statistics.

Editorial extensions

If this is right

  • A single flow-matching decoder can carry zero-shot timbre conversion and emotional prosody transfer, so the separate diffusion pitch generator used by Diff-HierVC is not necessary; this shortens the inference pipeline.
  • Discrete pitch tokens extracted from a target speaker's utterance act as an interpretable prosody prompt, meaning emotion can be transplanted from any reference speaker without aligning durations or matching global pitch statistics.
  • The masked-token training scheme turns the decoder into an in-context pitch predictor, so the model can also complete missing or corrupted pitch tokens, not only transfer them.
  • The two-part timbre representation (global embedding plus time-varying tokens) is what the paper credits for the SECS 0.920 result, which is higher than the 0.905 reported for Diff-HierVC and 0.911 for SEF-VC.
  • Because the flow matching decoder uses classifier-free guidance and rectified flow, high-quality Mel-spectrogram generation needs few steps, giving a real-time factor of 0.13 versus 0.42 for Diff-HierVC.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same recipe — quantize a speaker-independent expressive attribute, mask it during training, and prepend it as a prompt at inference — should transfer to other prosodic dimensions such as speaking rate, pauses, and loudness, although this paper fixes duration to the source utterance.
  • The headline emotion-consistency numbers come from an automatic emotion classifier (emotion2vec), so a natural extension would be a perceptual listening test to confirm whether the 0.075-point gap over Diff-HierVC is audible.
  • The speaker-independence of the pitch codebook is only tested indirectly; a direct check would be to measure emotion transfer when the target prompt comes from a speaker with a very unusual average pitch, since the SMN normalization assumes the per-speaker mean captures all speaker-related pitch information.
  • Such discrete pitch tokens could be plugged into a text-to-speech or speech-editing decoder, potentially giving those systems the same reference-based emotional style transfer without retraining the whole model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes PFlow-VC, a voice conversion model that combines a discrete pitch-conditioned flow matching Mel-spectrogram decoder with global and time-varying timbre embeddings. A pretrained pitch VQVAE quantizes speaker-mean normalized log-F0 into discrete tokens; during inference, pitch tokens from a target-speaker prompt are prepended to guide prosody and emotion, while source pitch tokens are replaced with null tokens. The model is evaluated on unseen LibriTTS test-clean data for zero-shot timbre conversion and on ESD data for emotion style transfer, reporting competitive or superior WER, SECS, QMOS, SMOS, and emotion consistency scores against YourTTS, Diff-HierVC, and SEF-VC, with ablations on pitch and timbre conditioning.

Significance. If the results hold, PFlow-VC provides a simple and interpretable mechanism for expressive voice conversion: discretized pitch tokens with masked in-context conditioning enable target-speaker prosody and emotion transfer while preserving source content. The paper's strengths include a standard optimal-transport conditional flow matching derivation, external objective evaluators (Whisper, WavLM-based SECS, emotion2vec) that are independent of the proposed model, baselines trained on the same data, and an ablation study isolating the pitch and timbre components. The main risk is that the key mechanism—cross-utterance pitch prompting with fully masked source pitch—is not matched by the training procedure, and the reported objective gains are not accompanied by significance testing.

major comments (3)
  1. [Section II-D vs. Section II-E] The training and inference setups for pitch conditioning do not match. In training, the model randomly masks continuous segments of pitch tokens, leaving unmasked pitch tokens from the same utterance as aligned local context. At inference, the source utterance's entire pitch token sequence is replaced with null tokens and the only pitch context is a prompt prepended from a different, unseen target utterance. The model is never trained under full-sequence masking with a cross-utterance prompt, so the claimed 'in-context pitch modeling' capability may not generalize to the actual inference task. The paper should add ablations with full-sequence masking and cross-utterance pitch prompts during training, and should ablate the pitch prompt while retaining the target speaker embedding and semantic prompt, to establish that the emotion-consistency gain in Table II is attributable to the discrete pitch prompt rather than to other conditioning signals.
  2. [Section III-A and Table II] The objective metrics (WER, SECS, Emo-Consistency Score) are reported as point estimates on 200-sample test sets without variance, confidence intervals, or significance tests. The emotion consistency score is an average over emotion2vec classifier outputs, and the headline gap of 0.725 vs. 0.650 over Diff-HierVC could be within sampling noise. The authors should provide paired significance tests (e.g., bootstrap or paired t-tests), per-emotion breakdowns, and preferably a subjective emotion-consistency evaluation, before claiming that PFlow-VC 'significantly exceeds' the baselines in style transfer.
  3. [Section II-E] The inference description is ambiguous as written: it says 'we first concatenate the target speaker's semantic tokens, pitch tokens, and speaker embedding' and prepends them to the tokens to be synthesized. If semantic tokens from the target speaker are used as a prompt while source semantic tokens form the main conditioning, the content-preservation mechanism is unclear; if this is a typo for the target speaker's pitch tokens only, it should be corrected. A precise specification of the prompt composition (which tokens are source, which are target, how null tokens are defined) is essential for reproducibility and for interpreting the ablation 'w/o pitch token'.
minor comments (4)
  1. [Section II-D] The text refers to 'solving the initial value problem in Eq. (1)', but Eq. (1) is the SMN-logf0 definition; the intended reference appears to be Eq. (2).
  2. [Section II-B] For an unseen target speaker, the speaker mean in Eq. (1) is not defined by a precomputed speaker-level statistic. The paper should state how the speaker mean is estimated at inference (e.g., from the prompt utterance) and whether this affects codebook token distributions.
  3. [Throughout] There are several typographical errors that should be corrected: 'pertub' in Section I, 'Trainging' in the Section II-E heading, 'timber' in Section II-C, and 'w/i corss attention' in Table III.
  4. [Section III-A / Table III] The pitch codebook size (C=64), vector dimension, loss coefficients, and number of timbre queries are fixed without sensitivity analysis; a brief discussion of how these choices affect pitch transfer quality would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PFlow-VC's pitch-prompt mechanism and emotion-transfer claim are evaluated with external benchmarks, and the central derivation does not reduce to its training inputs or to self-citations.

full rationale

The paper's core contribution is an empirical pipeline: pretrain a VQVAE on speaker-mean-normalized log F0, quantize into discrete pitch tokens, and feed a masked-pitch-conditioned optimal-transport conditional flow matching decoder with semantic tokens, timbre embeddings, and a target pitch prompt. Each component is specified with its own training objective (Eqs. 1-6) rather than imported as a black-box conclusion. The evaluation metrics—WER from Whisper-Large, SECS from WavLM-base-plus-sv, and emotion consistency from emotion2vec—are external pretrained models not trained by the authors for this task, so the reported gains are not forced by construction. The cited self-papers ([22], [25]-[27]) are used for architectural inspiration or VQ-VAE training details, not as the load-bearing proof of the pitch-transfer claim; the pitch VQVAE itself is trained and described in this paper. The train/inference mismatch noted by a careful reader (partial same-utterance masking during training versus full cross-utterance prompting at inference) is a legitimate generalization-risk concern that could undermine attribution of the emotion-consistency gain to pitch conditioning, but it is not a circularity: nothing is defined in terms of the evaluation metric, and no fitted parameter is renamed a prediction. Therefore no specific circular step can be quoted, and the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central claim rests on standard generative modeling assumptions plus several domain assumptions about speaker independence of pitch and semantic tokens, and on the validity of the emotion2vec classifier as an emotion measure. The paper does not release code or data, so the empirical grounding is partially verified by reported numbers and a demo page.

free parameters (5)
  • pitch codebook size (C) = 64
    Hand-chosen hyperparameter for the pitch VQVAE; quantization granularity directly controls the expressiveness of pitch conditioning.
  • pitch codebook vector dimension = 128
    Hand-chosen; capacity of each discrete pitch token.
  • pitch VQVAE loss coefficients = reconstruction 1.0, commitment 0.15, VQ 0.05
    Hand-set loss weights reported in Section III-A; no sensitivity analysis is provided.
  • number of trainable timbre queries = 64
    Number of latent vectors used in the timbre encoder cross-attention to produce time-varying timbre tokens (Section II-C).
  • HuBERT k-means cluster count = 500
    Number of discrete semantic units; standard practice but still a user-chosen value.
assumptions (5)
  • domain assumption Speaker-mean normalized log F0 removes speaker-specific pitch information
    Equation (1) assumes that subtracting the speaker's mean log F0 over voiced frames makes pitch tokens speaker-independent; this is not proven and underpins the whole pitch prompting design.
  • domain assumption HuBERT semantic tokens are speaker-independent enough to serve as content
    Section II-A states SSL features have 'minimal speaker variance' per prior work; the model relies on this for content preservation.
  • standard math Flow matching theory provides correct generative modeling
    Section II-D adopts OT-CFM from Lipman et al. [30]; standard, not proved in this paper.
  • domain assumption emotion2vec emotion classifier accurately measures emotion consistency
    Section III-C uses the emotion2vec model's predicted emotion scores as ground truth for style transfer; the paper provides no human validation of this metric.
  • domain assumption ECAPA-TDNN based timbre encoder with cross-attention and ASP captures global and time-varying speaker identity
    Section II-C follows [28] and [19]; the model assumes these learned tokens represent timbre dynamics.
invented entities (2)
  • time-varying timbre tokens
    purpose: Represent dynamic, content-dependent speaker characteristics beyond a static global embedding.
    Introduced in Section II-C; the paper provides only intrinsic ablations (Table III) and does not validate against an external measure of timbre dynamics.
  • discrete pitch token sequence independent evidence
    purpose: Speaker-independent prosody representation used to prompt the flow matching decoder.
    The tokens are derived from SMN-logf0 via a pretrained VQVAE and are used at inference with unseen speakers; the public demo page with audio samples provides a falsifiable handle, since listeners can judge whether pitch transfers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Expressive Voice Conversion with Discrete Pitch-Conditioned Flow Matching Model." pith.science (2026). https://pith.science/paper/4ZY76P52

@misc{pith2026250205471,
  author       = {Pith},
  title        = {Pith review of: Enhancing Expressive Voice Conversion with Discrete Pitch-Conditioned Flow Matching Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ZY76P52}},
  note         = {Machine review of arXiv:2502.05471}
}
read the original abstract

This paper introduces PFlow-VC, a conditional flow matching voice conversion model that leverages fine-grained discrete pitch tokens and target speaker prompt information for expressive voice conversion (VC). Previous VC works primarily focus on speaker conversion, with further exploration needed in enhancing expressiveness (such as prosody and emotion) for timbre conversion. Unlike previous methods, we adopt a simple and efficient approach to enhance the style expressiveness of voice conversion models. Specifically, we pretrain a self-supervised pitch VQVAE model to discretize speaker-irrelevant pitch information and leverage a masked pitch-conditioned flow matching model for Mel-spectrogram synthesis, which provides in-context pitch modeling capabilities for the speaker conversion model, effectively improving the voice style transfer capacity. Additionally, we improve timbre similarity by combining global timbre embeddings with time-varying timbre tokens. Experiments on unseen LibriTTS test-clean and emotional speech dataset ESD show the superiority of the PFlow-VC model in both timbre conversion and style transfer. Audio samples are available on the demo page https://speechai-demo.github.io/PFlow-VC/.

Figures

Figures reproduced from arXiv: 2502.05471 by the authors.

Figure 1
Figure 1. The overall architecture of PFlow-VC. II. PFLOW-VC ARCHITECTURE A. Architecture Overview The overall architecture is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The visualization of emotion representations extracted from unseen [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SpeechAccentLLM: A Unified Framework for Foreign Accent Conversion and Text to Speech

    eess.AS 2025-07 conditional novelty 5.0 of 10

    SpeechAccentLLM jointly trains foreign accent conversion and text-to-speech on CTC-regularized discrete speech tokens, with a BERT-style restorer, and reports improved accent reduction and intelligibility over one baseline.

  2. Rhythm Controllable and Efficient Zero-Shot Voice Conversion via Shortcut Flow Matching

    eess.AS 2025-06 conditional novelty 5.0 of 10

    R-VC performs zero-shot voice conversion in two sampling steps while transferring the target speaker's rhythm, matching or exceeding prior systems in naturalness and intelligibility.

  3. EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion

    cs.SD 2025-05 reject novelty 4.0 of 10

    EZ-VC combines discrete units from a multilingual self-supervised encoder (Xeus) with an F5-TTS flow-matching decoder to achieve zero-shot any-to-any voice conversion, without text labels or multiple disentangling encoders.

Reference graph

Works this paper leans on

38 extracted references · 16 canonical work pages · cited by 3 Pith papers

  1. [1]

    Autovc: Zero-shot voice style transfer with only autoencoder loss,

    K. Qian, Y . Zhang, S. Chang, X. Yang, and M. Hasegawa-Johnson, “Autovc: Zero-shot voice style transfer with only autoencoder loss,” in International Conference on Machine Learning . PMLR, 2019, pp. 5210–5219

    work page 2019

  2. [2]

    V oicemixer: Adver- sarial voice style mixup,

    S.-H. Lee, J.-H. Kim, H. Chung, and S.-W. Lee, “V oicemixer: Adver- sarial voice style mixup,” Advances in Neural Information Processing Systems, vol. 34, pp. 294–308, 2021

    work page 2021

  3. [3]

    One-shot voice conversion by sep- arating speaker and content representations with instance normalization,

    J.-c. Chou, C.-c. Yeh, and H.-y. Lee, “One-shot voice conversion by sep- arating speaker and content representations with instance normalization,” arXiv preprint arXiv:1904.05742 , 2019

    arXiv 1904

  4. [4]

    Again-vc: A one- shot voice conversion using activation guidance and adaptive instance normalization,

    Y .-H. Chen, D.-Y . Wu, T.-H. Wu, and H.-y. Lee, “Again-vc: A one- shot voice conversion using activation guidance and adaptive instance normalization,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 5954–5958

    2021

  5. [5]

    Unsupervised speech decomposition via triple information bottleneck,

    K. Qian, Y . Zhang, S. Chang, M. Hasegawa-Johnson, and D. Cox, “Unsupervised speech decomposition via triple information bottleneck,” in International Conference on Machine Learning . PMLR, 2020, pp. 7836–7846

    work page 2020

  6. [6]

    Neural analysis and synthesis: Reconstructing speech from self-supervised representa- tions,

    H.-S. Choi, J. Lee, W. Kim, J. Lee, H. Heo, and K. Lee, “Neural analysis and synthesis: Reconstructing speech from self-supervised representa- tions,” Advances in Neural Information Processing Systems , vol. 34, pp. 16 251–16 265, 2021

    work page 2021

  7. [7]

    Expressive-vc: Highly expressive voice conversion with attention fusion of bottleneck and perturbation features,

    Z. Ning, Q. Xie, P. Zhu, Z. Wang, L. Xue, J. Yao, L. Xie, and M. Bi, “Expressive-vc: Highly expressive voice conversion with attention fusion of bottleneck and perturbation features,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  8. [8]

    Contentvec: An improved self-supervised speech representation by disentangling speakers,

    K. Qian, Y . Zhang, H. Gao, J. Ni, C.-I. Lai, D. Cox, M. Hasegawa- Johnson, and S. Chang, “Contentvec: An improved self-supervised speech representation by disentangling speakers,” in International Con- ference on Machine Learning . PMLR, 2022, pp. 18 003–18 017

    work page 2022

Show all 38 references
  1. [9]

    Unitspeech: Speaker- adaptive speech synthesis with untranscribed data,

    H. Kim, S. Kim, J. Yeom, and S. Yoon, “Unitspeech: Speaker- adaptive speech synthesis with untranscribed data,” arXiv preprint arXiv:2306.16083, 2023

    2023 arXiv

  2. [10]

    A comparison of discrete and soft speech units for improved voice conversion,

    B. Van Niekerk, M.-A. Carbonneau, J. Za ¨ıdi, M. Baas, H. Seut ´e, and H. Kamper, “A comparison of discrete and soft speech units for improved voice conversion,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022,...

    2022

  3. [11]

    Opensr: Open- modality speech recognition via maintaining multi-modality alignment,

    X. Cheng, T. Jin, L. Li, W. Lin, X. Duan, and Z. Zhao, “Opensr: Open- modality speech recognition via maintaining multi-modality alignment,” arXiv preprint arXiv:2306.06410 , 2023

    2023 arXiv

  4. [12]

    Diff-hiervc: Diffusion-based hier- archical voice conversion with robust pitch generation and masked prior for zero-shot speaker adaptation,

    H.-Y . Choi, S.-H. Lee, and S.-W. Lee, “Diff-hiervc: Diffusion-based hier- archical voice conversion with robust pitch generation and masked prior for zero-shot speaker adaptation,” International Speech Communication Association, pp. 2283–2287, 2023

    2023

  5. [13]

    Transface: Unit-based audio-visual speech synthesizer for talking head translation,

    X. Cheng, R. Huang, L. Li, T. Jin, Z. Wang, A. Yin, M. Li, X. Duan, Z. Zhao et al. , “Transface: Unit-based audio-visual speech synthesizer for talking head translation,” arXiv preprint arXiv:2312.15197 , 2023

    2023 arXiv

  6. [14]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W.-N. Hsu, B. Bolte, Y .-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” IEEE/ACM transactions on audio, speech, and language processing , vol. 29, pp. 3451–3460, 2021

    2021

  7. [15]

    Xls-r: Self-supervised cross-lingual speech representation learning at scale,

    A. Babu, C. Wang, A. Tjandra, K. Lakhotia, Q. Xu, N. Goyal, K. Singh, P. V on Platen, Y . Saraf, J. Pino et al. , “Xls-r: Self-supervised cross-lingual speech representation learning at scale,” arXiv preprint arXiv:2111.09296, 2021

    2021 arXiv

  8. [16]

    Dgc-vector: A new speaker embedding for zero-shot voice conversion,

    R. Xiao, H. Zhang, and Y . Lin, “Dgc-vector: A new speaker embedding for zero-shot voice conversion,” in ICASSP 2022-2022 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2022, pp. 6547–6551

    2022

  9. [17]

    Zero-shot multi-speaker text-to-speech with state-of-the- art neural speaker embeddings,

    E. Cooper, C.-I. Lai, Y . Yasuda, F. Fang, X. Wang, N. Chen, and J. Yamagishi, “Zero-shot multi-speaker text-to-speech with state-of-the- art neural speaker embeddings,” in ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP) . I...

    2020

  10. [18]

    Sef-vc: Speaker embedding free zero-shot voice conversion with cross attention,

    J. Li, Y . Guo, X. Chen, and K. Yu, “Sef-vc: Speaker embedding free zero-shot voice conversion with cross attention,” in ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12 296–12 300

    2024

  11. [19]

    Refxvc: Cross-lingual voice conversion with enhanced reference leveraging,

    M. Zhang, Y . Zhou, Y . Ren, C. Zhang, X. Yin, and H. Li, “Refxvc: Cross-lingual voice conversion with enhanced reference leveraging,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2024

    2024

  12. [20]

    Zero-shot voice conversion via self-supervised prosody representation learning,

    S. Wang and D. Borth, “Zero-shot voice conversion via self-supervised prosody representation learning,” in 2022 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2022, pp. 01–08

    2022

  13. [21]

    Promptvc: Flexible stylistic voice conversion in latent space driven by natural language prompts,

    J. Yao, Y . Yang, Y . Lei, Z. Ning, Y . Hu, Y . Pan, J. Yin, H. Zhou, H. Lu, and L. Xie, “Promptvc: Flexible stylistic voice conversion in latent space driven by natural language prompts,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Proces...

    2024

  14. [22]

    Fluentspeech: Stutter-oriented automatic speech editing with context- aware diffusion models,

    Z. Jiang, Q. Yang, J. Zuo, Z. Ye, R. Huang, Y . Ren, and Z. Zhao, “Fluentspeech: Stutter-oriented automatic speech editing with context- aware diffusion models,” arXiv preprint arXiv:2305.13612 , 2023

    2023 arXiv

  15. [23]

    Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,

    J. Kong, J. Kim, and J. Bae, “Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,” Advances in neural information processing systems , vol. 33, pp. 17 022–17 033, 2020

    2020

  16. [24]

    Speech resynthesis from discrete disentangled self-supervised representations,

    A. Polyak, Y . Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.- N. Hsu, A. Mohamed, and E. Dupoux, “Speech resynthesis from discrete disentangled self-supervised representations,” arXiv preprint arXiv:2104.00355, 2021

    2021 arXiv

  17. [25]

    Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling,

    S. Ji, Z. Jiang, W. Wang, Y . Chen, M. Fang, J. Zuo, Q. Yang, X. Cheng, Z. Wang, R. Li et al. , “Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling,” arXiv preprint arXiv:2408.16532, 2024

    2024 arXiv

  18. [26]

    Ace: A generative cross-modal re- trieval framework with coarse-to-fine semantic modeling,

    M. Fang, S. Ji, J. Zuo, H. Huang, Y . Xia, J. Zhu, X. Cheng, X. Yang, W. Liu, G. Wang et al. , “Ace: A generative cross-modal re- trieval framework with coarse-to-fine semantic modeling,”arXiv preprint arXiv:2406.17507, 2024

    2024

  19. [27]

    Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias,

    Z. Jiang, Y . Ren, Z. Ye, J. Liu, C. Zhang, Q. Yang, S. Ji, R. Huang, C. Wang, X. Yin et al., “Mega-tts: Zero-shot text-to-speech at scale with intrinsic inductive bias,” arXiv preprint arXiv:2306.03509 , 2023

    2023 arXiv

  20. [28]

    Ecapa-tdnn: Em- phasized channel attention, propagation and aggregation in tdnn based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa-tdnn: Em- phasized channel attention, propagation and aggregation in tdnn based speaker verification,” arXiv preprint arXiv:2005.07143 , 2020

    2005 arXiv

  21. [29]

    Neural ordinary differential equations,

    R. T. Chen, Y . Rubanova, J. Bettencourt, and D. K. Duvenaud, “Neural ordinary differential equations,” Advances in neural information process- ing systems , vol. 31, 2018

    2018

  22. [30]

    Flow matching for generative modeling,

    Y . Lipman, R. T. Chen, H. Ben-Hamu, M. Nickel, and M. Le, “Flow matching for generative modeling,” arXiv preprint arXiv:2210.02747 , 2022

    2022 arXiv

  23. [31]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

    Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y . Yang, H. Hu, S. Zheng, Y . Gu, Z. Ma et al. , “Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,” arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  24. [32]

    Libritts: A corpus derived from librispeech for text-to-speech,

    H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “Libritts: A corpus derived from librispeech for text-to-speech,” arXiv preprint arXiv:1904.02882 , 2019

    1904 arXiv

  25. [33]

    Librispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP) . IEEE, 2015, pp. 5206–5210

    2015

  26. [34]

    Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset,

    K. Zhou, B. Sisman, R. Liu, and H. Li, “Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2021, pp. 920–924

    2021

  27. [35]

    Matcha-tts: A fast tts architecture with conditional flow matching,

    S. Mehta, R. Tu, J. Beskow, ´E. Sz ´ekely, and G. E. Henter, “Matcha-tts: A fast tts architecture with conditional flow matching,” in ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 11 341–11 345

    2024

  28. [36]

    emotion2vec: Self-supervised pre-training for speech emotion repre- sentation,

    Z. Ma, Z. Zheng, J. Ye, J. Li, Z. Gao, S. Zhang, and X. Chen, “emotion2vec: Self-supervised pre-training for speech emotion repre- sentation,” arXiv preprint arXiv:2312.15185 , 2023

    2023 arXiv

  29. [37]

    F0- consistent many-to-many non-parallel voice conversion via conditional autoencoder,

    K. Qian, Z. Jin, M. Hasegawa-Johnson, and G. J. Mysore, “F0- consistent many-to-many non-parallel voice conversion via conditional autoencoder,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2020, pp. 6284–6288

    2020

  30. [38]

    Text-free prosody-aware generative spoken language modeling,

    E. Kharitonov, A. Lee, A. Polyak, Y . Adi, J. Copet, K. Lakhotia, T.- A. Nguyen, M. Rivi `ere, A. Mohamed, E. Dupoux et al. , “Text-free prosody-aware generative spoken language modeling,” arXiv preprint arXiv:2109.03264, 2021

    2021 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.