REVIEW 4 major objections 7 minor 4 cited by
BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation -- Challenges and Insights
T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A Taiwanese Mandarin TTS uses phonetic-symbol injection and iconic-speaker unit conversion to cut voice-cloning phoneme errors by 61.2% and beat commercial systems in listening tests.
desk verdict Useful Taiwanese Mandarin TTS engineering, but the headline 'outperforms commercial systems' rests on a circular PER metric and a tiny subjective test. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanisms are the phonetic augmentation pipeline and the iconic-unit cloning trick. The augmentation pipeline, styled after masked-language-model noising, applies a sentence-level and character-level decision tree that randomly replaces Chinese characters with Mandarin Phonetic Symbols during training, with a small amount of deliberate noise, so the large language model learns to treat the symbols as the authoritative pronunciation signal; at inference the same g2pW predictor supplies those symbols for the input text. Iconic Unit Augmented Speech Cloning replaces direct conditioning on a noisy speaker's units with a two-stage path: the LLM first produces speech units from a clean iconic speaker given the target text, and the OT-CFM (optimal-transport conditional flow matching) model then converts those units into the target speaker's timbre, separating content generation from voice conversion. These sit on the base infrastructure of a supervised semantic speech tokenizer that turns audio into discrete units and an OT-CFM that turns units into mel-spectrograms.
What would settle it
Take a set of polyphonic sentences where the automatic phoneme predictor is known to be wrong, have human phoneticians transcribe the reference, synthesize BreezyVoice with and without phonetic augmentation, and compute PER against the human reference; if the gain disappears or the model systematically reproduces the predictor's errors whenever the predictor and humans disagree, the claimed disambiguation improvement is an artifact of the evaluation loop.
Extended reading notes
Core claim
The paper argues that BreezyVoice outperforms existing commercial TTS systems in both general and code-switching contexts, and that two design choices carry most of the gain. First, augmenting training text and inference text with Mandarin Phonetic Symbols predicted by a neural Mandarin grapheme-to-phoneme model, g2pW, lets the system disambiguate polyphonic characters: on 23 hard instances the base pipeline failed 8 times, while augmentation corrected all but 1. Second, Iconic Unit Augmented Speech Cloning—generating speech units from an iconic clean speaker and then converting them to the target voice—reduces phoneme error rate from 3.4% to 1.3% across 100 spontaneous speakers, improving 86 of them, at a measured cost of 2.51% in speaker similarity. The paper also reports an average speaker similarity of 92.29% over 115 speakers, and finds that the model follows injected phonetic symbols even when they contradict the character's correct pronunciation, which it reads as evidence that phonetic control drives pronunciation accuracy in this architecture.
Load-bearing premise
The evaluation assumes that the phoneme sequence produced by a large ASR model followed by g2pW is an accurate ground truth; because the same g2pW supplies the phonetic symbols used in inference, the reported error reductions may partly measure how faithfully BreezyVoice copies g2pW's predictions rather than how correctly it pronounces the characters.
Editorial extensions
If this is right
- Polyphone disambiguation in Mandarin TTS can be treated as a controllable input problem: giving the model predicted phonetic symbols removes most pronunciation errors without architecture changes.
- Voice cloning for long-tail or noisy speakers can be stabilized by a two-stage generate-then-convert pipeline, at a small measured cost in speaker similarity.
- The same adapted pipeline handles Chinese-to-English code-switching well, with the main remaining weakness being Chinese toponyms, identified as future work.
- Phonetic symbols override grapheme identity in this model, meaning a user can force a desired pronunciation by editing the symbol sequence—a practical control lever for names, rare words, and loanwords.
- The failure analysis locates the dominant error source in the LLM-predicted speech units rather than the speaker embedding or conditioning spectrogram, guiding where future fixes should be aimed.
Reading between the lines
- Not claimed by the paper: because injected phonetic symbols override character identity, the same mechanism could serve as a user-facing pronunciation editor for names, rare words, and loanwords, letting people correct pronunciations by editing symbol sequences rather than retraining.
- Not claimed by the paper: the reported gains are bounded by the phoneme predictor's own accuracy; a human-annotated set of hard polyphones would separate genuine disambiguation from faithful copying of the predictor.
- Not claimed by the paper: the iconic-unit trick suggests a general design principle for unit-based TTS—decouple content-unit generation from timbre conversion—which could transfer to other languages with large spelling-to-pronunciation variation.
- Not claimed by the paper: since Traditional Chinese characters are shared with Cantonese but pronounced differently, the phonetic-symbol augmentation could in principle be transplanted to Cantonese TTS, though the paper does not test this.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents BreezyVoice, a Taiwanese Mandarin TTS system built on CosyVoice, adding a supervised semantic speech tokenizer, an LLM for text-to-unit generation, an OT-CFM vocoder, and a g2pW phoneme prediction model used to augment the input with Mandarin Phonetic Symbols. The authors claim that BreezyVoice outperforms existing commercial TTS systems in both general and code-switching contexts, and they analyze robustness issues in voice cloning and polyphone disambiguation, proposing an iconic-unit augmented speech cloning pipeline that reportedly reduces phoneme error rate from 3.4% to 1.3% with a small speaker-similarity trade-off. The evaluation includes a small human preference study, PER and SSL-MOS scores against four anonymized commercial services, code-switching accuracy on five categories, and speaker-wise cloning error rates, with additional ablations isolating the CFM and LLM components.
Significance. If the claims were fully supported, the paper would be a useful contribution: it adapts a strong open-source TTS backbone to a lower-resourced language variety, integrates explicit phonetic control for polyphone disambiguation, and provides systematic ablations (Equations 5 and 6) that convincingly identify the LLM-predicted units rather than speaker conditioning or CFM reconstruction as the main error source. The iconic-unit augmented cloning idea is interesting and the reported 86/100 speaker improvement is suggestive even if the PER reference is imperfect. I also credit the authors for using public datasets, building on open-source components, and including an ethics statement on voice spoofing. However, the central 'superior performance' claim is currently supported mainly by a weak subjective test and by objective metrics that do not consistently favor BreezyVoice; the PER metric is also partially circular because the evaluation reference and the system input share the same g2pW model. These issues are load-bearing for the paper's headline, so the work needs a substantive evaluation overhaul before the main claim can be accepted.
major comments (4)
- [§5.1 and Eq. (3)] The PER reference is circular with respect to the system under test. Section 5.1 states that PER is computed from phonetic symbols predicted by a Whisper-large + g2pW cascade, while Eq. (3) shows that g2pW is also used at inference to augment BreezyVoice's input with Mandarin Phonetic Symbols. BreezyVoice is therefore scored against a reference generated by the same model that supplies its pronunciation input; the four commercial systems are not given this information. A low PER can partly reflect faithful copying of g2pW's phoneme choices rather than correct polyphone disambiguation, and Whisper recognition errors can make incorrect pronunciations match the misrecognized reference. The same issue affects the 61.2% PER reduction claimed in Section 6.1.3, which is measured against the same cascade. I request an independent phoneme reference (human phoneme transcriptions or a separately trained G2P model), or at minimum a variant of Table 1 in which BreezyVoice does not receive g2pW augmentation, together with an explicit caveat that the current PER numbers are biased in favor of BreezyVoice.
- [Table 1 and §7] The objective results in Table 1 do not support the unqualified claim that BreezyVoice outperforms existing commercial TTS systems in general contexts. BreezyVoice's PER is 0.8%, which is worse than Service U (0.69%) and Service M (0.43%), and its SSL-MOS is 4.46, which is worse than Service Z (4.63). Thus the conclusion in Section 7 that BreezyVoice 'outperforms existing commercial TTS systems in both general and code-switching contexts' is not supported by the paper's own objective metrics. The authors should either soften the claim to 'competitive on objective metrics and preferred in a small subjective test' or provide additional evidence that establishes a reliable overall superiority.
- [§5.1 and Figure 2] The human preference evidence for superiority is statistically weak. The test uses only three annotators and ten TCMD samples, with no significance testing, no confidence intervals, and no inter-annotator agreement reported. Figure 2 shows aggregate win rates but no measure of variability, so the phrase 'consistent superior performance' is an overstatement. The authors should report per-annotator results, agreement statistics, and a significance test (or clearly frame the result as preliminary and exploratory). Without this, the subjective evaluation cannot carry the weight of the paper's main claim.
- [§5.1.1 and Table 2] The code-switching comparison does not uniformly favor BreezyVoice. In Table 2, BreezyVoice ties with Service Y on General (8 vs 8) and on Sentences (7 vs 7), loses to Service Y on Toponyms (3 vs 7), and clearly wins only on Entities and Abbreviations. The statement in Section 5.1.1 that BreezyVoice shows 'great performance across all categories' is therefore inaccurate, and the conclusion that it outperforms commercial systems in code-switching contexts is not supported for all categories. Please either provide a per-category analysis explaining the toponym and sentence failures, or restrict the claim to the categories where the advantage actually holds.
minor comments (7)
- [§5.1 heading] The heading 'Comperlative Evaluations' contains a typo and should read 'Comparative Evaluations'.
- [Table 2 caption] The caption refers to the 'TCSSD' dataset, but the corpus is introduced in Section 4.1 as TCCSD; please correct the abbreviation.
- [§2.3] There are typos in the sentence 'We perform phonteic augmenetations using g2pW...' and 'allow us to preamptively augment'; these should read 'phonetic augmentations' and 'preemptively augment'.
- [§4.1] The phrase 'from scouring read-world data' should be 'from scouring real-world data'.
- [§4.2, Eq. (3)] Equation (3) defines Y_augmented, but the text does not explicitly state how Y_augmented is combined with the byte-pair-encoded text Y in Equation (1); please clarify whether it replaces Y, is concatenated with it, or is used as an auxiliary conditioning signal.
- [§5.2] The sentence 'over half of the spontaneous samples samples exhibit an error rate of less than 3%' contains a duplicated word and should be corrected.
- [References] The BERT citation appears twice, once as a standalone 'Kenton and Toutanova (2019)' entry and once as part of 'Devlin et al. (2019)'; please unify the citations.
Circularity Check
Objective pronunciation evaluation is self-referential: g2pW both supplies BreezyVoice's phonetic input and generates the PER reference.
-
self definitional
[Section 4.2 Eq. (3); Section 5.1; Section 2.3]
"Additionally, there is the option to augment Mandarin phonetic symbols. Yaugmented = g2pW(Y) ... The PER is calculated with predicted phonetic symbols from cascading Whisper-large with g2pW. ... We perform phonteic augmenetations using g2pW in both training and inference to boost TTS accuracy."
BreezyVoice receives g2pW's phonetic symbols as an auxiliary input at inference (Eq. 3), while the PER reference is also produced by the same g2pW model cascaded after Whisper-large. Therefore the evaluation measures how faithfully BreezyVoice reproduces g2pW's phoneme decisions rather than pronunciation correctness independent of g2pW. A system that simply copied g2pW's output would be scored as correct by construction. The four commercial comparison systems do not receive the g2pW augmentation, so Table 1's PER comparison is biased in BreezyVoice's favor by the metric design.
-
self definitional
[Section 6.1.3, using PER definition from Section 5.1]
"Our benchmark results show that this approach drastically reduces phoneme error rate from 3.4% to 1.3%, which is an overall reduction of 61.2%, accompanied by only a slight trade-off in speaker similarity (-2.51%)."
The 61.2% reduction is measured with the same Whisper-large + g2pW PER reference described in Section 5.1. Since g2pW's phonetic symbols are already provided to BreezyVoice at inference, a lower PER can reflect better adherence to g2pW rather than improved disambiguation of polyphonic characters. If the Whisper-based reference misrecognizes the audio, incorrect pronunciations can also match the misrecognized reference, so the reported improvement is not an independent measure of pronunciation quality.
full rationale
The central objective pronunciation evaluation is partially circular: BreezyVoice is augmented with g2pW phonetic symbols at inference (Eq. 3), and PER is computed against a Whisper-large + g2pW reference (Sec. 5.1). This makes a low PER partly a measure of copying g2pW and disadvantages commercial systems that lack the same augmentation. The 61.2% iconic-unit reduction is measured against the same reference and inherits the bias. Independent evidence exists: the human preference test (Fig. 2), the human-annotated code-switching accuracy (Table 2), and SSL-MOS scores do not depend on the g2pW input/reference loop, and the voice-similarity and flow-matching ablations are not pronunciation claims. However, the headline 'outperforms existing commercial TTS systems in both general and code-switching contexts' relies in part on this self-referential PER metric. Self-citations to g2pW and Breeze-7B are load-bearing but g2pW has independent benchmark standing; the circularity is in the evaluation design, not in the citation itself. Separately, even within the biased metric, Table 1 does not show BreezyVoice with the best PER (M: 0.43, U: 0.69), which further weakens the unqualified superiority claim.
Assumptions & free parameters
free parameters (3)
- Augmentation probabilities in Figure 1 =
50%/15%/1%/98% (sentence/character/replacement/phonetic)
- Rare-word frequency threshold =
500 occurrences
- Number of training epochs and learning rate =
3 epochs, lr 1e-4
assumptions (4)
- domain assumption CosyVoice's supervised semantic tokens and OT-CFM model behave as described in Du et al. (2024a).
- domain assumption g2pW provides accurate Mandarin phonetic symbols for polyphone disambiguation.
- domain assumption Whisper-large cascaded with g2pW gives a valid reference for phoneme error rate.
- domain assumption The curated speech and text datasets are representative of Taiwanese Mandarin usage.
Cite this review
Pith. "Pith review of BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation -- Challenges and Insights." pith.science (2026). https://pith.science/paper/MFHSSBTW
@misc{pith2026250117790,
author = {Pith},
title = {Pith review of: BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation -- Challenges and Insights},
year = {2026},
howpublished = {\url{https://pith.science/paper/MFHSSBTW}},
note = {Machine review of arXiv:2501.17790}
}
abstract
We present BreezyVoice, a Text-to-Speech (TTS) system specifically adapted for Taiwanese Mandarin, highlighting phonetic control abilities to address the unique challenges of polyphone disambiguation in the language. Building upon CosyVoice, we incorporate a $S^{3}$ tokenizer, a large language model (LLM), an optimal-transport conditional flow matching model (OT-CFM), and a grapheme to phoneme prediction model, to generate realistic speech that closely mimics human utterances. Our evaluation demonstrates BreezyVoice's superior performance in both general and code-switching contexts, highlighting its robustness and effectiveness in generating high-fidelity speech. Additionally, we address the challenges of generalizability in modeling long-tail speakers and polyphone disambiguation. Our approach significantly enhances performance and offers valuable insights into the workings of neural codec TTS systems.
Figures
Forward citations
Cited by 4 Pith papers
-
The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation
Emotion embedding similarities are unsuitable for zero-shot evaluation of emotional expressiveness in speech generation due to confounding by non-emotional acoustic features.
-
On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
Global token perplexity mis-ranks spoken language models; localized/normalized likelihood scores and an embedding judge track human MOS better and make the best model look much closer to human.
-
Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative
Fake-Mamba reports EERs of 0.97%, 1.74%, and 5.85% on three speech deepfake benchmarks, but the provided full text is an unrelated paper, so the claims cannot be verified.
-
A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data
Fine-tuning Whisper-large-v2 on 10,000 hours of synthesized Mandarin plus small real English/code-switching sets yields Twister, cutting mixed error rate by up to 56% on code-switching and 19% on Taiwanese Mandarin.
Reference graph
Works this paper leans on
-
[1]
David Bradley. 1992. Chinese as a pluricentric language. Pluricentric languages: Differing norms in different nations, pages 305--324
work page 1992
-
[2]
Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. 2024. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.05370
arXiv 2024
-
[3]
Yi-Chang Chen, Yu-Chuan Steven, Yen-Cheng Chang, and Yi-Ren Yeh. 2022. https://doi.org/10.21437/Interspeech.2022-216 g2pw: A conditional weighted softmax bert for polyphone disambiguation in mandarin . In Interspeech 2022, pages 1926--1930
-
[4]
Alexandre D \'e fossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438
arXiv 2022
-
[5]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...
-
[6]
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. 2024 a . Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05407
arXiv 2024
-
[7]
Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng. 2024 b . Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 591--595. IEEE
work page 2024
-
[8]
Chan-Jan Hsu, Yi-Chang Chen, Feng-Ting Liao, Pei-Chen Ho, Yu-Hsiang Wang, Po-Chun Hsu, and Da-shan Shiu. 2024 a . Let's fuse step by step: A generative fusion decoding algorithm with llms for multi-modal text recognition. arXiv preprint arXiv:2405.14259
arXiv 2024
Show all 25 references
-
[9]
Chan-Jan Hsu, Chang-Le Liu, Feng-Ting Liao, Po-Chun Hsu, Yi-Chang Chen, and Da-Shan Shiu. 2024 b . Breeze-7b technical report. arXiv e-prints, pages arXiv--2403
2024
-
[10]
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276
2024 arXiv
-
[11]
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. 2024. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100
2024 arXiv
-
[12]
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2. Minneapolis, Minnesota
2019
-
[13]
Yuan-Fu Liao, Chia-Yu Chang, Hak-Khiam Tiun, Huang-Lan Su, Hui-Lu Khoo, Jane S Tsay, Le-Kun Tan, Peter Kang, Tsun-guan Thiann, Un-Gian Iunn, et al. 2020. Formosa speech recognition challenge 2020 and taiwanese across taiwan corpus. In 2020 23rd Conference of the Oriental COCOS...
2020
-
[14]
Guan-Ting Lin, Chan-Jan Hsu, Da-Rong Liu, Hung-Yi Lee, and Yu Tsao. 2022. Analyzing the robustness of unsupervised speech recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8202--8206. IEEE
2022
-
[15]
dialect/topolect
Victor H Mair. 1991. What is a chinese" dialect/topolect"?: Reflections on some key sino-english linguistic terms. Sino-Platonic Papers
1991
-
[16]
Shivam Mehta, Ruibo Tu, Jonas Beskow, \'E va Sz \'e kely, and Gustav Eje Henter. 2024. Matcha-tts: A fast tts architecture with conditional flow matching. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11341--11345. IEEE
2024
-
[17]
Kyubyong Park and Seanie Lee. 2020. https://arxiv.org/abs/2004.03136 A neural grapheme-to-phoneme conversion package for mandarin chinese based on a new open benchmark dataset . Proc. Interspeech 2020
2020 arXiv
-
[18]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492--28518. PMLR
2023
-
[19]
Liang-Hsuan Tseng, Zih-Ching Chen, Wei-Shun Chang, Cheng-Kuang Lee, Tsung-Ren Huang, and Hung-yi Lee. 2024. Leave no knowledge behind during knowledge distillation: Towards practical and effective knowledge distillation for code-switching asr using realistic data. arXiv prepri...
2024 arXiv
-
[20]
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. 2023. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111
2023 arXiv
-
[21]
Chih-Kai Yang, Yu-Kuan Fu, Chen-An Li, Yi-Cheng Lin, Yu-Xiang Lin, Wei-Chih Chen, Ho Lam Chung, Chun-Yi Kuan, Wei-Ping Huang, Ke-Han Lu, et al. 2024. Building a taiwanese mandarin spoken language model: A first attempt. arXiv preprint arXiv:2411.07111
2024 arXiv
-
[22]
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2021. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495--507
2021
-
[23]
Zezario, Yu-Wen Chen, Szu-Wei Fu, Yu Tsao, Hsin-Min Wang, and Chiou-Shann Fuh
Ryandhimas E. Zezario, Yu-Wen Chen, Szu-Wei Fu, Yu Tsao, Hsin-Min Wang, and Chiou-Shann Fuh. 2023. https://arxiv.org/abs/2309.12766 A study on incorporating whisper for robust speech assessment . Preprint, arXiv:2309.12766
2023 arXiv
-
[24]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[25]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.