REVIEW 3 major objections 4 minor 27 references
An Exploration of ECAPA-TDNN and x-vector Speaker Representations in Zero-shot Multi-speaker TTS
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Within a fixed zero-shot TTS framework on Czech speech, the original H/ASP speaker encoder outperforms both ECAPA-TDNN and x-vector embeddings on speaker similarity, in both listening and embedding-based evaluations.
desk verdict Useful fixed-framework comparison of speaker encoders for zero-shot TTS, but the abstract overstates the consistency of the results and the objective significance test pools dependent observations. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the controlled swap of a single component: three TTS models ('H/ASP TTS', 'ECAPA-TDNN TTS', 'x-vector TTS') are built on the same YourTTS backbone, trained with speaker consistency loss on the same 1062-speaker Czech corpus, and evaluated on the same 24 out-of-domain target speakers. The three speaker encoders are the only intended difference, though they also differ in pretraining data, embedding dimension (512 vs 192), and in how the 30-second reference is processed: H/ASP averages embeddings from 10 short segments totaling about 26.7 seconds, while the other two read the full reference at once thanks to statistics pooling. Subjective similarity ratings and cosine distances between embeddings of synthesized and natural utterances serve as the measuring instruments.
What would settle it
Equalize the input handling across encoders, for instance by feeding H/ASP the full 30-second reference and feeding x-vector and ECAPA-TDNN the same 10-segment averaged inputs, then re-run the listening test and cosine-distance evaluation; if the H/ASP advantage disappears or reverses, the claim that the encoder family is responsible would be falsified.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that, in this fixed YourTTS configuration, switching the speaker encoder to ECAPA-TDNN or x-vector does not improve and actually degrades speaker similarity relative to the baseline H/ASP encoder. The ordering — H/ASP best, ECAPA-TDNN intermediate, x-vector worst — is consistent across the listening test (statistically significant for all pairwise comparisons) and across the objective cosine-distance evaluation, where the aggregate across four embedding extractors confirms H/ASP better than ECAPA-TDNN (p = 0.02) and both better than x-vector (p < 0.001). The authors frame this as evidence that recognition-optimized embeddings are not automatically better for synthesis, and they provide a controlled, reusable comparison framework.
Load-bearing premise
The comparison treats the three systems as differing only in speaker-encoder architecture, but they also differ in pretraining corpus, embedding dimension, and most importantly in how the reference audio is consumed, so the observed ordering could come from those incidental differences rather than from the encoder family itself.
Editorial extensions
If this is right
- Plug-and-play replacement of the speaker encoder in a fixed TTS framework does not guarantee better speaker similarity; empirical evaluation is required.
- The popularity of ECAPA-TDNN in speaker recognition does not translate to an advantage in zero-shot TTS in this configuration.
- In the considered setup, x-vector embeddings perform worse than ECAPA-TDNN for speaker similarity, despite both being TDNN-based.
- A reusable evaluation protocol, combining a 0–100 similarity listening test with cosine distances from multiple embedding extractors, can serve future comparisons of speaker embeddings in TTS.
Reading between the lines
- The H/ASP encoder's 10-segment averaging may act as a form of test-time augmentation, effectively giving it more diverse views of the reference voice than the full-utterance pass used by the other encoders; this could explain part of its advantage.
- The result may depend on the language and domain mismatch between the VoxCeleb-pretrained encoders and the Czech read-speech training data, so matched-language pretraining could change the ranking.
- A direct test would be to integrate a stronger recognition model like the TitaNet-large used in the objective evaluation as an in-loop speaker encoder with speaker consistency loss; if it also fails to beat H/ASP, the input-handling explanation becomes more plausible.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript compares three speaker encoders—the original H/ASP encoder used in YourTTS, a SpeechBrain x-vector encoder, and a SpeechBrain ECAPA-TDNN encoder—within an otherwise fixed YourTTS zero-shot multi-speaker TTS framework. All three TTS models are trained on the same Czech read-speech corpus and evaluated on 24 out-of-domain radio speakers. The subjective evaluation is a 23-listener similarity listening test with per-listener normalization and Holm-Bonferroni-corrected Wilcoxon tests. The objective evaluation measures cosine distances between embeddings of synthesized and natural utterances using four separate embedding extractors (ECAPA-TDNN, x-vector, Resemblyzer, TitaNet). The paper's central claim is that the original H/ASP encoder consistently outperforms the alternatives in both evaluations, and that ECAPA-TDNN outperforms x-vectors. The experiments are mostly clearly described, but the headline claim is stronger than the data support: Table 3 shows extractor-dependent reversals, and the pooled objective significance test has a repeated-measures dependency problem.
Significance. If the result is appropriately qualified, the paper makes a practically useful contribution: it provides a controlled empirical comparison showing that newer speaker-recognition embeddings do not automatically improve zero-shot TTS speaker similarity, and it demonstrates a reusable evaluation protocol. The paper deserves credit for keeping the TTS backbone fixed, training all models on the same dataset, collecting a listening test with normalization and multiple-comparison correction, and using four evaluation extractors with multiple synthesized and natural utterances per speaker. These are genuine strengths. The main value is negative evidence that should temper the common assumption that ECAPA-TDNN is always superior for TTS conditioning. However, the significance is diminished by the overgeneralized conclusion and by a confound in the encoder comparison (Section 2.3), so the contribution currently requires revision rather than acceptance as is.
major comments (3)
- [Abstract and Section 4] The abstract's claim that the H/ASP encoder 'consistently outperformed the alternatives' is contradicted by the paper's own objective data. In Table 3, under the x-vector evaluation extractor, ECAPA-TDNN TTS has a lower mean cosine distance (0.649) than H/ASP TTS (0.674), so H/ASP is not consistently best across all four extractors. Section 4 itself acknowledges that results are 'not fully consistent' and that relative placements 'fluctuate.' The abstract and Section 5 should be revised to state that H/ASP performs best in the listening test and in most objective extractor comparisons, not that it consistently outperforms the alternatives across both evaluations.
- [Section 4, statistical analysis paragraph] The pooled Wilcoxon signed-rank test over 24 speakers × 4 extractors (96 values per TTS model) treats the four extractor-specific distances for the same speaker as independent observations. These values are correlated because they are computed from the same speakers, the same synthesized utterances, and the same natural reference utterances, so the test overstates the evidence for a cross-extractor ranking. This is load-bearing because the only significant objective H/ASP-vs-ECAPA result (p = 0.02) aggregates data in which the direction reverses under the x-vector extractor (ECAPA 0.649 vs. H/ASP 0.674 in Table 3). The authors should report extractor-specific paired tests or use a model that accounts for the repeated-measures structure, and should base the abstract's conclusion on those results rather than on the pooled test.
- [Section 2.3] The comparison does not fully isolate the speaker encoder type as claimed. Section 2.3 specifies that H/ASP averages embeddings from 10 segments totaling approximately 26.7 seconds of a 30-second reference, while the x-vector and ECAPA models process the entire 30-second file at once. The encoders also differ in pretraining data (VoxCeleb2 for H/ASP vs. VoxCeleb1+2 for the alternatives) and embedding dimension (512 vs. 192). Any of these incidental differences could drive the observed speaker-similarity scores, so the conclusion that 'the original H/ASP encoder still remains the most effective' is an overgeneralization. The authors should either control input segmentation (for example, by using the same segmentation strategy across all encoders) or explicitly limit the conclusion to the specific configurations tested.
minor comments (4)
- [Section 4, paragraph 2] There is a typo: 'usng' should be 'using' in the sentence about Resemblyzer embeddings.
- [Section 2.3 and references] The dataset name is inconsistently spaced: 'V oxCeleb2' appears in Section 2.3 and in reference [4], and 'V oxCeleb 1' is used elsewhere; the spacing should be uniform.
- [Figure 2] The three TTS model lines in each panel are hard to distinguish; using distinct markers or line styles (in addition to color) would improve readability for print readers.
- [Section 5] The suggestion that TitaNet-large could be 'properly compared' in a future study is useful, but the manuscript does not provide code or trained model weights, which would be valuable for reproducing the comparison.
Circularity Check
No significant circularity: the speaker-encoder comparison is an empirical evaluation against external human ratings and independent embedding extractors, with no fitted parameter or self-citation chain driving the result.
full rationale
The paper reports a controlled experimental comparison of three speaker encoder types inside a YourTTS framework. The central claim is a ranking of H/ASP, ECAPA-TDNN, and x-vector for speaker similarity, supported by a listening test and by cosine distances between synthesized and natural utterances computed with four embedding extractors. There is no derivation in which a predicted quantity is equal by construction to an input, no parameter fitted to a target dataset and then renamed as a prediction, and no load-bearing self-citation: the reference list contains no prior work by the present authors that is used to justify the result. The TTS models are trained on the same Czech corpus and evaluated on 24 out-of-domain speakers, and the objective metric compares embeddings of synthesized speech with embeddings of real held-out speech, so the evaluation is external to the training setup. One could note that two of the evaluation extractors share architectures with two of the encoders being compared, and that the strict claim of 'consistently' outperformance is weakened by the paper's own Table 3, where under the x-vector extractor ECAPA-TDNN TTS has a lower mean cosine distance (0.649) than H/ASP TTS (0.674) while the text states the results are 'not fully consistent'. These are correctness or overgeneralization concerns about the empirical conclusion, not instances of circularity. The paper's own equations for the distances are straightforward averages over natural and synthesized utterances, and no result is forced by definition or by an imported uniqueness theorem. Score 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Cosine distance between speaker embeddings is a valid proxy for perceived speaker similarity.
- domain assumption The 24 target speakers were absent from the training set.
- domain assumption Cutting references to 30 seconds equalizes encoder conditions.
- domain assumption Pretrained VoxCeleb encoder models transfer to Czech speech.
Cite this review
Pith. "Pith review of An Exploration of ECAPA-TDNN and x-vector Speaker Representations in Zero-shot Multi-speaker TTS." pith.science (2026). https://pith.science/paper/4JSJX2IO
@misc{pith2026250620190,
author = {Pith},
title = {Pith review of: An Exploration of ECAPA-TDNN and x-vector Speaker Representations in Zero-shot Multi-speaker TTS},
year = {2026},
howpublished = {\url{https://pith.science/paper/4JSJX2IO}},
note = {Machine review of arXiv:2506.20190}
}
read the original abstract
Zero-shot multi-speaker text-to-speech (TTS) systems rely on speaker embeddings to synthesize speech in the voice of an unseen speaker, using only a short reference utterance. While many speaker embeddings have been developed for speaker recognition, their relative effectiveness in zero-shot TTS remains underexplored. In this work, we employ a YourTTS-based TTS system to compare three different speaker encoders - YourTTS's original H/ASP encoder, x-vector embeddings, and ECAPA-TDNN embeddings - within an otherwise fixed zero-shot TTS framework. All models were trained on the same dataset of Czech read speech and evaluated on 24 out-of-domain target speakers using both subjective and objective methods. The subjective evaluation was conducted via a listening test focused on speaker similarity, while the objective evaluation measured cosine distances between speaker embeddings extracted from synthesized and real utterances. Across both evaluations, the original H/ASP encoder consistently outperformed the alternatives, with ECAPA-TDNN showing better results than x-vectors. These findings suggest that, despite the popularity of ECAPA-TDNN in speaker recognition, it does not necessarily offer improvements for speaker similarity in zero-shot TTS in this configuration. Our study highlights the importance of empirical evaluation when reusing speaker recognition embeddings in TTS and provides a framework for additional future comparisons.
Figures
Reference graph
Works this paper leans on
-
[1]
Cai, W., Chen, J., Li, M.: Exploring the encoding layer and loss function in end-to-end speaker and language recognition system. In: The Speaker and Language Recognition Work- ECAPA-TDNN and x-vector Speaker Representations in Zero-shot TTS 11 shop (Odyssey 2018). pp. 74–81 (2018). https://doi.org/10.21437/Odyssey. 2018-11
-
[2]
In: Pro- ceedings of the 39th International Conference on Machine Learning
Casanova, E., Weber, J., Shulby, C.D., Candido Jr., A., Gölge, E., Ponti, M.A.: YourTTS: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone. In: Pro- ceedings of the 39th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 162, pp. 2709–2720. PMLR (2022), https://proceedings. mlr.press/...
work page 2022
-
[3]
IEEE Journal of Selected Topics in Signal Processing 16(6), 1505–1518 (2022)
Chen, S., et al.: WavLM: large-scale self-supervised pre-training for full stack speech pro- cessing. IEEE Journal of Selected Topics in Signal Processing 16(6), 1505–1518 (2022). https://doi.org/10.1109/JSTSP.2022.3188113
arXiv 2022
-
[4]
Chung, J.S., Nagrani, A., Zisserman, A.: V oxCeleb2: deep speaker recognition. In: Interspeech 2018. pp. 1086–1090 (2018). https://doi.org/10.21437/ Interspeech.2018-1929
work page 2018
-
[5]
Cooper, E., Lai, C.I., Yasuda, Y ., Fang, F., Wang, X., Chen, N., Yamagishi, J.: Zero-shot multi-speaker text-to-speech with state-of-the-art neural speaker embeddings. In: ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 6184–6188 (2020). https://doi.org/10.1109/ICASSP40776. 2020.9054535
-
[6]
Dawalatabad, N., Ravanelli, M., Grondin, F., Thienpondt, J., Desplanques, B., Na, H.: ECAPA-TDNN embeddings for speaker diarization. In: Interspeech 2021. pp. 3560–3564 (2021). https://doi.org/10.21437/Interspeech.2021-941
-
[7]
IEEE Transactions on Audio, Speech, and Language Processing 19(4), 788–798 (2011)
Dehak, N., Kenny, P.J., Dehak, R., Dumouchel, P., Ouellet, P.: Front-end factor analysis for speaker verification. IEEE Transactions on Audio, Speech, and Language Processing 19(4), 788–798 (2011). https://doi.org/10.1109/TASL.2010.2064307
-
[8]
Desplanques, B., Thienpondt, J., Demuynck, K.: ECAPA-TDNN: Emphasized chan- nel attention, propagation and aggregation in TDNN based speaker verification. In: Interspeech 2020. pp. 3830–3834 (2020). https://doi.org/10.21437/ Interspeech.2020-2650
work page 2020
Show all 27 references
-
[9]
In: Interspeech 2017
Doddipatla, R., Braunschweiler, N., Maia, R.: Speaker adaptation in DNN-based speech syn- thesis using d-vectors. In: Interspeech 2017. pp. 3404–3408 (2017). https://doi.org/ 10.21437/Interspeech.2017-1038
2017 doi
-
[10]
In: Interspeech 2024
Gusev, A., Avdeeva, A.: Improvement speaker similarity for zero-shot any-to-any voice conversion of whispered and regular speech. In: Interspeech 2024. pp. 2735–2739 (2024). https://doi.org/10.21437/Interspeech.2024-2091
2024 doi
- [11]
-
[12]
ITU-R Recommendation BS.1534-3: Method for the subjective assessment of intermediate quality level of audio systems. Tech. rep., International Telecommunication Union (2015)
2015
-
[13]
In: ICASSP 2025 - 2025 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP)
Jeong, M., Kim, M., Kim, S., Kim, N.S.: Evidential-TTS: high fidelity zero-shot text-to- speech using evidential deep learning. In: ICASSP 2025 - 2025 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1–5 (2025). https: //doi.org/10.1109/...
2025
-
[14]
In: Proceedings of the 38th International Conference on Machine Learning
Kim, J., Kong, J., Son, J.: Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In: Proceedings of the 38th International Conference on Machine Learning. vol. 139, pp. 5530–5540. PMLR (2021), https://proceedings. mlr.press/v139/kim21f.html
2021
-
[15]
In: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Koluguri, N.R., Park, T., Ginsburg, B.: TitaNet: neural model for speaker representation with 1D depth-wise separable convolutions and global context. In: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 8102–8106 (2022)...
2022
- [16]
- [17]
-
[18]
In: ICASSP 2024 - 2024 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP)
Li, H., Zhu, X., Xue, L., Song, Y ., Chen, Y ., Xie, L.: SponTTS: modeling and trans- ferring spontaneous style for TTS. In: ICASSP 2024 - 2024 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP). pp. 12171–12175 (2024). https://doi.org/10.1109/...
2024
-
[19]
In: Interspeech 2024
Loweimi, E., Qian, M., Knill, K., Gales, M.: On the usefulness of speaker embeddings for speaker retrieval in the wild: A comparative study of x-vector and ECAPA-TDNN models. In: Interspeech 2024. pp. 3774–3778 (2024). https://doi.org/10.21437/ Interspeech.2024-161
2024
-
[20]
In: Interspeech 2017
Nagrani, A., Chung, J.S., Zisserman, A.: V oxCeleb: a large-scale speaker identification dataset. In: Interspeech 2017. pp. 2616–2620 (2017). https://doi.org/10.21437/ Interspeech.2017-950
2017
- [21]
-
[22]
In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., Khudanpur, S.: X-vectors: Robust DNN embeddings for speaker recognition. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 5329–5333 (2018). https://doi.org/ 10.1109/ICASSP.2018.8461375
2018
-
[23]
In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Variani, E., Lei, X., McDermott, E., Lopez Moreno, I., Gonzalez-Dominguez, J.: Deep neural networks for small footprint text-dependent speaker verification. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 4052–4056 (2014). https:...
2014
-
[24]
In: 2018 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP)
Wan, L., Wang, Q., Papir, A., Lopez Moreno, I.: Generalized end-to-end loss for speaker verification. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Pro- cessing (ICASSP). pp. 4879–4883 (2018). https://doi.org/10.1109/ICASSP. 2018.8462665
2018
-
[25]
In: Interspeech 2021
Xin, D., Saito, Y ., Takamichi, S., Koriyama, T., Saruwatari, H.: Cross-lingual speaker adaptation using domain adaptation and speaker consistency loss for text-to-speech syn- thesis. In: Interspeech 2021. pp. 1614–1618 (2021). https://doi.org/10.21437/ Interspeech.2021-897
2021
-
[26]
In: 2022 13th International Symposium on Chinese Spoken Language Processing (ISCSLP)
Xue, J., Deng, Y ., Han, Y ., Li, Y ., Sun, J., Liang, J.: ECAPA-TDNN for multi-speaker text-to-speech synthesis. In: 2022 13th International Symposium on Chinese Spoken Language Processing (ISCSLP). pp. 230–234 (2022). https://doi.org/10.1109/ ISCSLP57327.2022.10037956
2022
-
[27]
In: Interspeech 2021
Yang, S., et al.: SUPERB: Speech Processing Universal PERformance Benchmark. In: Interspeech 2021. pp. 1194–1198 (2021). https://doi.org/10.21437/ Interspeech.2021-1775
2021
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.