REVIEW 3 major objections 6 minor 29 references
Active Learning for Text-to-Speech Synthesis with Informative Sample Collection
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper claims that actively selecting informative web speech samples as the model trains builds a TTS corpus that, at the same size, synthesizes more high-quality speakers than a same-sized corpus built by prior selection methods.
desk verdict A sensible active-learning idea for TTS corpus construction, undermined by an evaluation that selects and scores with the same pseudo-MOS proxy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a closed loop with two gates. The candidate pool is split into disjoint chunks, and at each step the TTS model trained on the current corpus is used to score each candidate by zero-shot synthesis: an utterance counts as informative if its synthetic quality falls below the threshold $\theta_{hq}$, meaning the current model cannot already produce that speaker well, and it must also pass a data-quality estimator whose predicted score is above $\theta_{hq}$. The threshold $\theta_{hq}$ is set by the minimum pseudo-MOS observed when synthesizing speakers from a studio-recorded reference corpus, and a diffusion model trained on PCA-reduced x-vectors generates new speaker embeddings so the method can be evaluated on unseen speakers. This loop is what focuses data collection on samples that improve the current model rather than on samples that merely look diverse or high-quality in isolation.
What would settle it
Run a human listening study on a sample of speakers that the pseudo-MOS estimator labels high-quality ($\geq \theta_{hq}$) and low-quality for both the proposed corpus and the same-size baseline. If human preference ratings do not reproduce the reported ordering, or if many pseudo-MOS high-quality speakers are judged poor, the central claim that the active-learning corpus improves synthesis quality would be contradicted.
Extended reading notes
Core claim
The paper's central claim is that a text-to-speech corpus assembled by an active-learning loop, alternating TTS training with selection of the next chunk of web data, yields higher synthesis quality per stored utterance than a corpus of the same size assembled by a feed-forward, model-agnostic selector. Concretely, with a 3,943-utterance corpus the method lets 71.0% of real speakers and 69.2% of generated speakers be synthesized above the quality threshold, versus 66.6% and 62.8% for the evaluation-in-the-loop baseline at the same size. A candidate utterance is added only if it passes a quality filter and is not already synthesizable by the current model; the second condition is what distinguishes the method from prior selection. The authors also claim that the method needs only about 10% of candidate data to train the quality estimator, and that the resulting TTS model generalizes to speakers generated in x-vector space, not only to held-out real speakers.
Load-bearing premise
The whole method and its evaluation rely on the automatic quality predictor correctly telling which synthesized voices sound good to people; if that predictor is biased, the method may be selecting for the predictor rather than for real speech quality.
Editorial extensions
If this is right
- At a fixed corpus size, the active-learning corpus yields more speakers whose synthetic voice passes the quality bar, so storage-constrained TTS deployments can spend their data budget better.
- Because selection re-trains with the current model, the loop adapts to the learner, and the same procedure could apply to other generative tasks where a quality proxy and a conditioning embedding exist.
- The informativeness criterion, cannot yet be synthesized well, acts as a redundancy filter; adding a diversity term among accepted samples could raise coverage further.
- The estimator trained on 10% of candidate data reportedly correlates with one trained on all data, suggesting the upfront compute for corpus construction can be reduced by an order of magnitude.
- Evaluation with generated speaker embeddings indicates the trained model generalizes beyond observed speakers, making speaker-coverage comparisons more meaningful as a quality measure.
Reading between the lines
- Pith inference: the reported advantage is measured on pseudo-MOS rather than human listeners, so a human listening test could reveal a smaller or larger gap; the method's practical edge is only as strong as that proxy.
- Pith inference: the reliance on a single quality threshold $\theta_{hq}$ derived from a studio-recorded corpus may not transfer across domains, languages, or recording conditions, and re-deriving the threshold per use case is a natural test.
- Pith inference: on-demand downloading saves storage but assumes fetching and processing each segment is cheap relative to keeping all data; in bandwidth-constrained settings the trade-off could invert.
- Pith inference: applying the informativeness filter within an already-accepted chunk, not only across chunks, could avoid near-duplicate additions and potentially improve speaker diversity further.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an active-learning framework for constructing a multi-speaker TTS corpus from web-scale YouTube data. The method partitions a candidate video list and, at each iteration, downloads a new segment, trains a TTS model on the currently selected corpus, and selects samples that pass a data-quality filter (pseudo-MOS above θhq) and an informativeness filter (zero-shot pseudo-MOS below θhq). It also trains a diffusion-based speaker generation model on the collected x-vectors. Experiments compare the resulting corpus ('Ours') against an evaluation-in-the-loop baseline of the same size, reporting a higher ratio of speakers whose synthesized speech exceeds θhq, along with analyses of speaker coverage and of the diffusion model's fidelity. The paper additionally reports that 'Ours' is worse than 'Baseline' for pseudo-MOS values at or above 2.6.
Significance. If the reported evaluation is trustworthy, the method is a useful step toward storage-efficient, data-efficient TTS corpus construction: it removes the need to download the full candidate corpus, it includes a sanity check on the data-quality estimator trained on a 10% subset (Fig. 7), and it demonstrates that a diffusion model captures the x-vector distribution better than GMMs in terms of Wasserstein-1 distance. However, the central comparison rests entirely on a pseudo-MOS estimator that is used both to select data and to define the success metric, with no human listening test or independent quality measure. The abstract's claim of 'higher-quality speech synthesis' is stronger than what is demonstrated, given the reported worse performance in the high-quality region. The practical significance is therefore conditional on an independent validation of the pseudo-MOS proxy.
major comments (3)
- [§II-C, §III-A5, Table I] The main claim rests on a selection-on-the-evaluator loop: the informativeness filter in §II-C keeps samples whose zero-shot pseudo-MOS is below θhq, and the evaluation in §III-A5 and Table I counts high-quality speakers using the same pseudo-MOS estimator and the same threshold θhq defined in §II-A2. No human listening test or independently trained quality metric is reported. The 71.0% versus 66.6% difference in Table I may therefore reflect optimization of the pseudo-MOS estimator rather than an improvement in human-perceived synthesis quality. Please provide a human MOS evaluation, or at least an independent quality metric, on a sample of speakers, and report both that metric and pseudo-MOS.
- [Table I] Table I reports each condition as a single point estimate, with no multiple runs, error bars, or significance test. The claimed improvement for real speakers is 4.4 percentage points (71.0% vs. 66.6%), which is small relative to the likely run-to-run variation caused by random data download, model training stochasticity, and pseudo-MOS prediction noise. Please report the mean and standard deviation over several independent runs, or provide a paired significance test, before concluding that the proposed method is superior.
- [§III-B1, Fig. 3a] The paper states in §III-B1 that 'Ours' is worse than 'Baseline' for pseudo-MOS values of 2.6 or higher, and Fig. 3a shows that the cumulative curve for the proposed method falls below the baseline in that region. The abstract's claim of 'higher-quality speech synthesis' is therefore too strong; the demonstrated claim is an increased number of speakers above the threshold, not higher quality across the distribution. Please temper the abstract and conclusion, and explicitly discuss this tradeoff in relation to the proposed objective.
minor comments (6)
- [Fig. 6 caption] The caption contains a typo: 'Genarated' should be 'Generated'.
- [§III-B4] The text reads 'Fig. 7 7' with a duplicated figure number; remove the extra '7'.
- [Reference [22]] The reference title contains 'Pcor' instead of 'Proc.'; please correct it.
- [§II-B, §III-A1] Cross-references use inconsistent styles such as 'Section 2.B' while other references use Roman numerals; please standardize them.
- [Fig. 3] The caption says 'The shaded area corresponds to high-quality speakers,' but the shading is difficult to distinguish in grayscale; please use a clearer visual cue or a legend.
- [§III-A5] The notation 'M = 1, 2, ...,10' should be 'M = 1, 2, ..., 10' for typographical consistency.
Circularity Check
The selection objective and the evaluation metric are the same pseudo-MOS threshold; the reported 'higher-quality speech synthesis' reduces to selection on the evaluator.
-
self definitional
[Sec. II-A.2, Sec. II-C, Sec. III-A.5 / Table I]
"We then define the minimum observed quality score among these speakers as the threshold θhq. Speakers whose synthetic speech exceeds θhq are regarded as achieving a synthesis quality comparable to that of the studio-recorded corpus, and thus are considered capable of generating high-quality speech. ... we construct Ck by adding to Ck−1 the samples from Dk that satisfy both of the following: (1) data quality exceeds θhq, and (2) synthetic quality falls below θhq. ... We examine the number and distribution of speakers whose pseudo-MOS exceeded the threshold θhq."
The paper defines 'high-quality' speakers as those whose pseudo-MOS exceeds θhq (II-A.2). The active-learning data-selection rule (II-C) is explicitly designed to admit only data predicted to make synthesis exceed θhq and to exclude data that already synthesize above θhq, so the corpus is optimized to maximize the number of speakers sitting just above θhq. The evaluation (III-A.5, Table I) then counts exactly this quantity: the fraction of speakers whose pseudo-MOS exceeds θhq. The abstract's claim of 'higher-quality speech synthesis' is therefore a re-statement of the selection objective measured on the same automatic estimator, with no human listening test or independent quality metric reported.
full rationale
The paper's method and evaluation both center on the pseudo-MOS threshold θhq. In II-A.2, a speaker is defined as 'high-quality' exactly when its synthetic speech exceeds θhq, where θhq is a pseudo-MOS score observed on a studio corpus. In II-C, the active-learning filter admits only candidate data predicted to exceed θhq and rejects data that already synthesize above θhq, so the corpus construction is explicitly an optimizer for the number of speakers above θhq. The evaluation in III-A.5 and Table I then measures the fraction of speakers whose pseudo-MOS exceeds θhq, and the abstract concludes 'higher-quality speech synthesis.' Because the same automatic estimator and threshold define both the selection objective and the success metric, the reported gains (71.0% vs 66.6% real; 69.2% vs 62.8% generated) are expected consequences of selection pressure rather than independent evidence of absolute quality. The paper offers no human listening test or external quality metric. The self-citations [9],[19] supply the training-evaluation loop, but the more fundamental issue is the shared metric, not the citations as such. The paper's own admission in III-B.1 that Ours is worse than Baseline for pseudo-MOS ≥ 2.6 confirms that the improvement is confined to the band just above θhq. This is a partial but real circularity: the headline claim is, to a large extent, a re-statement of the optimization target. Score 6.
Assumptions & free parameters
free parameters (6)
- Quality threshold θhq =
Not stated numerically (minimum pseudo-MOS of studio speakers)
- Sampling ratios r1, r2 =
0.1, 0.9
- Number of iterations K =
2
- PCA dimension d' =
28
- Diffusion hyperparameters =
T=200, β1=0.0001, βT=0.05, hidden=56
- Pre-screening thresholds (CTC, x-vector variance) =
Not specified; inherited from [9]
assumptions (5)
- domain assumption The pseudo-MOS estimator accurately predicts human-perceived TTS synthesis quality.
- domain assumption A sample whose zero-shot synthesis quality exceeds θhq is redundant for TTS training.
- domain assumption The data quality estimator trained on 10% of candidate data behaves like one trained on all data.
- domain assumption The diffusion model over PCA-reduced x-vectors captures the true speaker embedding distribution.
- standard math Standard PCA and diffusion model mathematics are valid as used.
Cite this review
Pith. "Pith review of Active Learning for Text-to-Speech Synthesis with Informative Sample Collection." pith.science (2026). https://pith.science/paper/CXCIBRVN
@misc{pith2026250708319,
author = {Pith},
title = {Pith review of: Active Learning for Text-to-Speech Synthesis with Informative Sample Collection},
year = {2026},
howpublished = {\url{https://pith.science/paper/CXCIBRVN}},
note = {Machine review of arXiv:2507.08319}
}
read the original abstract
The construction of high-quality datasets is a cornerstone of modern text-to-speech (TTS) systems. However, the increasing scale of available data poses significant challenges, including storage constraints. To address these issues, we propose a TTS corpus construction method based on active learning. Unlike traditional feed-forward and model-agnostic corpus construction approaches, our method iteratively alternates between data collection and model training, thereby focusing on acquiring data that is more informative for model improvement. This approach enables the construction of a data-efficient corpus. Experimental results demonstrate that the corpus constructed using our method enables higher-quality speech synthesis than corpora of the same size.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,
J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y . Zhang, Y . Wang, R. Skerrv-Ryan et al. , “Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions,” in Proc. ICASSP. IEEE, 2018, pp. 4779–4783
work page 2018
-
[2]
A vector quantized ap- proach for text to speech synthesis on real-world spontaneous speech,
L.-W. Chen, S. Watanabe, and A. Rudnicky, “A vector quantized ap- proach for text to speech synthesis on real-world spontaneous speech,” in Proc. AAAI, 2023, pp. 12 644–12 652
work page 2023
-
[3]
Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models,
Z. Ju, Y . Wang, K. Shen, X. Tan, D. Xin, D. Yang, Y . Liu, Y . Leng, K. Song, S. Tang et al. , “Naturalspeech 3: Zero-shot speech syn- thesis with factorized codec and diffusion models,” arXiv preprint arXiv:2403.03100, 2024
arXiv 2024
-
[4]
JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesis,
R. Sonobe, S. Takamichi, and H. Saruwatari, “JSUT corpus: free large-scale Japanese speech corpus for end-to-end speech synthesis,” arXiv:1711.00354, 2017
arXiv 2017
-
[5]
JVS corpus: free Japanese multi-speaker voice corpus,
S. Takamichi, K. Mitsui, Y . Saito, T. Koriyama, N. Tanji, and H. Saruwatari, “JVS corpus: free Japanese multi-speaker voice corpus,” arXiv:1908.06248, 2019
arXiv 1908
-
[6]
O. Take, S. Takamichi, K. Seki, Y . Bando, and H. Saruwatari, “SaSLaW: Dialogue speech corpus with audio-visual egocentric information toward environment-adaptive dialogue speech synthesis,” in Proc. Interspeech, 2024, pp. 1860–1864
work page 2024
-
[7]
LibriTTS: A Corpus Derived from LibriSpeech for Text- to-Speech,
H. Zen, V . Dang, R. Clark, Y . Zhang, R. J. Weiss, Y . Jia, Z. Chen, and Y . Wu, “LibriTTS: A Corpus Derived from LibriSpeech for Text- to-Speech,” in Proc. Interspeech, 2019, pp. 1526–1530
work page 2019
-
[8]
HUI-Audio-Corpus-German: A high quality TTS dataset,
P. Puchtler, J. Wirth, and R. Peinl, “HUI-Audio-Corpus-German: A high quality TTS dataset,” in German Conference on Artificial Intelligence (K¨unstliche Intelligenz). Springer, 2021, pp. 204–216
work page 2021
Show all 29 references
-
[9]
Text-to-speech synthesis from dark data with evaluation-in-the-loop data selection,
K. Seki, S. Takamichi, T. Saeki, and H. Saruwatari, “Text-to-speech synthesis from dark data with evaluation-in-the-loop data selection,” in Proc. ICASSP. IEEE, 2023, pp. 1–5
2023
-
[10]
Gigaspeech: An evolving, multi- domain asr corpus with 10,000 hours of transcribed audio,
G. Chen, S. Chai, G. Wang, J. Du, W.-Q. Zhang, C. Weng, D. Su, D. Povey, J. Trmal, J. Zhang et al. , “Gigaspeech: An evolving, multi- domain asr corpus with 10,000 hours of transcribed audio,” arXiv preprint arXiv:2106.06909, 2021
2021 arXiv
-
[11]
JTubeSpeech: corpus of Japanese speech collected from YouTube for speech recognition and speaker verification,
S. Takamichi, L. K ¨urzinger, T. Saeki, S. Shiota, and S. Watanabe, “JTubeSpeech: corpus of Japanese speech collected from YouTube for speech recognition and speaker verification,” arXiv:2112.09323, 2021
2021 arXiv
-
[12]
J-CHAT: Japanese large-scale spoken dialogue corpus for spoken dialogue language modeling,
W. Nakata, K. Seki, H. Yanaka, Y . Saito, S. Takamichi, and H. Saruwatari, “J-CHAT: Japanese large-scale spoken dialogue corpus for spoken dialogue language modeling,” arXiv preprint arXiv:2407.15828, 2024
2024
-
[13]
Diversity-based core-set selection for text-to-speech with linguistic and acoustic fea- tures,
K. Seki, S. Takamichi, T. Saeki, and H. Saruwatari, “Diversity-based core-set selection for text-to-speech with linguistic and acoustic fea- tures,” in Proc. ICASSP. IEEE, 2024, pp. 1–5
2024
-
[14]
Deepcore: A comprehensive library for coreset selection in deep learning,
C. Guo, B. Zhao, and Y . Bai, “Deepcore: A comprehensive library for coreset selection in deep learning,” in International Conference on Database and Expert Systems Applications . Springer, 2022, pp. 181– 195
2022
-
[15]
Active learning is a strong baseline for data subset selection,
D. Park, D. Papailiopoulos, and K. Lee, “Active learning is a strong baseline for data subset selection,” in NeurIPS 2022 Workshop, 2022
2022
-
[16]
A survey of deep active learning,
P. Ren, Y . Xiao, X. Chang, P.-Y . Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang, “A survey of deep active learning,” ACM computing surveys (CSUR), vol. 54, no. 9, pp. 1–40, 2021
2021
-
[17]
X-vectors: Robust DNN embeddings for speaker recognition,
D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-vectors: Robust DNN embeddings for speaker recognition,” in Proc. ICASSP. IEEE, 2018, pp. 5329–5333
2018
-
[18]
Ctc- segmentation of large corpora for german end-to-end speech recogni- tion,
L. K ¨urzinger, D. Winkelbauer, L. Li, T. Watzel, and G. Rigoll, “Ctc- segmentation of large corpora for german end-to-end speech recogni- tion,” in Speech and Computer, A. Karpov and R. Potapova, Eds. Cham: Springer International Publishing, 2020, pp. 267–278
2020
-
[19]
TTSOps: A closed- loop corpus optimization framework for training multi-speaker TTS models from dark data,
K. Seki, S. Takamichi, T. Saeki, and H. Saruwatari, “TTSOps: A closed- loop corpus optimization framework for training multi-speaker TTS models from dark data,” arXiv preprint arXiv:2506.15614 , 2025
2025
-
[20]
Speaker generation,
D. Stanton, M. Shannon, S. Mariooryad, R. Skerry-Ryan, E. Battenberg, T. Bagby, and D. Kao, “Speaker generation,” in Proc. ICASSP. IEEE, 2022, pp. 7897–7901
2022
-
[21]
Denoising diffusion probabilistic models,
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Proc. NeurIPS, vol. 33, pp. 6840–6851, 2020
2020
-
[22]
Diffusion models are minimax optimal distribution estimators,
K. Oko, S. Akiyama, and T. Suzuki, “Diffusion models are minimax optimal distribution estimators,” in Pcor. ICML. PMLR, 2023
2023
-
[23]
Diffusion models in vision: A survey,
F.-A. Croitoru, V . Hondru, R. T. Ionescu, and M. Shah, “Diffusion models in vision: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 9, pp. 10 850–10 869, 2023
2023
-
[24]
ITA corpus,
“ITA corpus,” https://github.com/mmorise/ita-corpus
-
[25]
FastSpeech 2: Fast and high-quality end-to-end text to speech,
Y . Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y . Liu, “FastSpeech 2: Fast and high-quality end-to-end text to speech,” Proc. ICLR, 2021
2021
-
[26]
HiFi-GAN: Generative adversarial net- works for efficient and high fidelity speech synthesis,
J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial net- works for efficient and high fidelity speech synthesis,” Proc. NeurIPS, vol. 33, pp. 17 022–17 033, 2020
2020
-
[27]
HiFi-GAN,
“HiFi-GAN,” https://github.com/jik876/hifi-gan
-
[28]
FastSpeech 2-JSUT,
“FastSpeech 2-JSUT,” https://github.com/Wataru-Nakata/ FastSpeech2-JSUT
-
[29]
x-vector,
“x-vector,” https://github.com/sarulab-speech/xvector jtubespeech. 6
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.