{"id":"a3506870-d65d-4222-99ae-0c16b0d00f11","arxiv_id":"2501.17790","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BreezyVoice adapts CosyVoice to Taiwanese Mandarin with g2pW-based phonetic augmentation and a two-stage iconic-unit voice cloning pipeline, improving pronunciation accuracy and cloning robustness.","lead":"BreezyVoice is a text-to-speech system built for Taiwanese Mandarin that adds phonetic symbols to guide pronunciation of ambiguous Chinese characters. It reportedly beats four commercial TTS services on human preference and voice cloning tests, while the paper also reveals common failure modes in neural codec TTS.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PER evaluation is circular: g2pW both supplies BreezyVoice's phonetic input and generates the evaluation reference, biasing objective comparisons in favor of BreezyVoice.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the objective pronunciation metric is generated by a model that is also used to control BreezyVoice at inference. This is not a minor methodological quibble; it directly undermines the evidence for the paper's central claim that BreezyVoice outperforms commercial TTS systems. If the reference were independent human-verified phonemes, the comparison would be meaningful, but as reported the PER numbers partly measure self-consistency between g2pW and BreezyVoice. The subjective preference test is also small, but the circular PER is the more specific and fixable flaw. Because the reader has already marked the paper CONDITIONAL on this basis, my stress-test does not change the verdict: the authors should add an independent, human-verified evaluation before the superiority claim can be accepted. I would not move to REJECT because the paper's architectural insights and the iconic-unit ablation may still be valid; they simply need to be re-measured with an unbiased reference. The proposed concrete test would settle the circularity concern directly by recomputing PER against human labels, and it would also test whether the 61.2% reduction persists under an unbiased metric.","tokens_in":8456,"tokens_out":3563,"duration_ms":38063,"concrete_test":"Construct a held-out set of roughly 100 Taiwanese Mandarin sentences with human-verified pronunciation annotations, enriched for polyphonic characters and rare words, and recompute PER for BreezyVoice and the four commercial systems against these human labels instead of the Whisper+g2pW cascade. Also recompute the iconic-unit ablation PER with the same human-labelled reference; if BreezyVoice's advantage over the best commercial system disappears or shrinks materially, the claimed superiority and the 61.2% reduction are artifacts of the shared g2pW reference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 states that PER is computed from phonetic symbols predicted by a Whisper-large + g2pW cascade. Section 4.2 shows that the same g2pW model is used at inference to augment BreezyVoice's input with Mandarin Phonetic Symbols (Y_augmented = g2pW(Y)). Thus BreezyVoice is effectively given the answer key for pronunciation, while the four commercial systems are not. A low PER therefore partly measures how faithfully BreezyVoice copies g2pW's phoneme choices, not how correctly it disambiguates polyphonic characters. Even if g2pW were perfect, the comparison is biased: any system that reproduces g2pW's output is scored as correct by construction. The headline 'superior performance in general contexts' rests on Table 1, where BreezyVoice's PER is not even best, and on a small subjective preference test (three annotators, ten TCMD samples, no significance testing). In addition, the 61.2% iconic-unit reduction in Section 6.1.3 is measured against this same Whisper+g2pW reference, so the improvement could reflect improved adherence to g2pW rather than improved pronunciation accuracy; Whisper recognition errors can also make incorrect pronunciations match the misrecognized reference. The code-switching scores in Table 2 use human annotations and are less affected, but BreezyVoice does not win all categories there, further weakening the unqualified 'outperforms existing commercial TTS' claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents BreezyVoice, a Taiwanese Mandarin TTS system built on CosyVoice, adding a supervised semantic speech tokenizer, an LLM for text-to-unit generation, an OT-CFM vocoder, and a g2pW phoneme prediction model used to augment the input with Mandarin Phonetic Symbols. The authors claim that BreezyVoice outperforms existing commercial TTS systems in both general and code-switching contexts, and they analyze robustness issues in voice cloning and polyphone disambiguation, proposing an iconic-unit augmented speech cloning pipeline that reportedly reduces phoneme error rate from 3.4% to 1.3% with a small speaker-similarity trade-off. The evaluation includes a small human preference study, PER and SSL-MOS scores against four anonymized commercial services, code-switching accuracy on five categories, and speaker-wise cloning error rates, with additional ablations isolating the CFM and LLM components.","tokens_in":8773,"tokens_out":6267,"duration_ms":60086,"significance":"If the claims were fully supported, the paper would be a useful contribution: it adapts a strong open-source TTS backbone to a lower-resourced language variety, integrates explicit phonetic control for polyphone disambiguation, and provides systematic ablations (Equations 5 and 6) that convincingly identify the LLM-predicted units rather than speaker conditioning or CFM reconstruction as the main error source. The iconic-unit augmented cloning idea is interesting and the reported 86/100 speaker improvement is suggestive even if the PER reference is imperfect. I also credit the authors for using public datasets, building on open-source components, and including an ethics statement on voice spoofing. However, the central 'superior performance' claim is currently supported mainly by a weak subjective test and by objective metrics that do not consistently favor BreezyVoice; the PER metric is also partially circular because the evaluation reference and the system input share the same g2pW model. These issues are load-bearing for the paper's headline, so the work needs a substantive evaluation overhaul before the main claim can be accepted.","major_comments":[{"comment":"The PER reference is circular with respect to the system under test. Section 5.1 states that PER is computed from phonetic symbols predicted by a Whisper-large + g2pW cascade, while Eq. (3) shows that g2pW is also used at inference to augment BreezyVoice's input with Mandarin Phonetic Symbols. BreezyVoice is therefore scored against a reference generated by the same model that supplies its pronunciation input; the four commercial systems are not given this information. A low PER can partly reflect faithful copying of g2pW's phoneme choices rather than correct polyphone disambiguation, and Whisper recognition errors can make incorrect pronunciations match the misrecognized reference. The same issue affects the 61.2% PER reduction claimed in Section 6.1.3, which is measured against the same cascade. I request an independent phoneme reference (human phoneme transcriptions or a separately trained G2P model), or at minimum a variant of Table 1 in which BreezyVoice does not receive g2pW augmentation, together with an explicit caveat that the current PER numbers are biased in favor of BreezyVoice.","section":"§5.1 and Eq. (3)"},{"comment":"The objective results in Table 1 do not support the unqualified claim that BreezyVoice outperforms existing commercial TTS systems in general contexts. BreezyVoice's PER is 0.8%, which is worse than Service U (0.69%) and Service M (0.43%), and its SSL-MOS is 4.46, which is worse than Service Z (4.63). Thus the conclusion in Section 7 that BreezyVoice 'outperforms existing commercial TTS systems in both general and code-switching contexts' is not supported by the paper's own objective metrics. The authors should either soften the claim to 'competitive on objective metrics and preferred in a small subjective test' or provide additional evidence that establishes a reliable overall superiority.","section":"Table 1 and §7"},{"comment":"The human preference evidence for superiority is statistically weak. The test uses only three annotators and ten TCMD samples, with no significance testing, no confidence intervals, and no inter-annotator agreement reported. Figure 2 shows aggregate win rates but no measure of variability, so the phrase 'consistent superior performance' is an overstatement. The authors should report per-annotator results, agreement statistics, and a significance test (or clearly frame the result as preliminary and exploratory). Without this, the subjective evaluation cannot carry the weight of the paper's main claim.","section":"§5.1 and Figure 2"},{"comment":"The code-switching comparison does not uniformly favor BreezyVoice. In Table 2, BreezyVoice ties with Service Y on General (8 vs 8) and on Sentences (7 vs 7), loses to Service Y on Toponyms (3 vs 7), and clearly wins only on Entities and Abbreviations. The statement in Section 5.1.1 that BreezyVoice shows 'great performance across all categories' is therefore inaccurate, and the conclusion that it outperforms commercial systems in code-switching contexts is not supported for all categories. Please either provide a per-category analysis explaining the toponym and sentence failures, or restrict the claim to the categories where the advantage actually holds.","section":"§5.1.1 and Table 2"}],"minor_comments":[{"comment":"The heading 'Comperlative Evaluations' contains a typo and should read 'Comparative Evaluations'.","section":"§5.1 heading"},{"comment":"The caption refers to the 'TCSSD' dataset, but the corpus is introduced in Section 4.1 as TCCSD; please correct the abbreviation.","section":"Table 2 caption"},{"comment":"There are typos in the sentence 'We perform phonteic augmenetations using g2pW...' and 'allow us to preamptively augment'; these should read 'phonetic augmentations' and 'preemptively augment'.","section":"§2.3"},{"comment":"The phrase 'from scouring read-world data' should be 'from scouring real-world data'.","section":"§4.1"},{"comment":"Equation (3) defines Y_augmented, but the text does not explicitly state how Y_augmented is combined with the byte-pair-encoded text Y in Equation (1); please clarify whether it replaces Y, is concatenated with it, or is used as an auxiliary conditioning signal.","section":"§4.2, Eq. (3)"},{"comment":"The sentence 'over half of the spontaneous samples samples exhibit an error rate of less than 3%' contains a duplicated word and should be corrected.","section":"§5.2"},{"comment":"The BERT citation appears twice, once as a standalone 'Kenton and Toutanova (2019)' entry and once as part of 'Devlin et al. (2019)'; please unify the citations.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a competent systems paper with useful engineering insights, but the headline claim of superiority over commercial TTS is not supported by the current evaluation. The PER circularity and the small subjective test are load-bearing, and they can be addressed with an independent phoneme reference, more annotators, and softened claims. I would be willing to look at a revised version, but in its present form the paper reads more like a technical report than a paper whose central claim is fully established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful engineering paper—adapting CosyVoice to Taiwanese Mandarin, with a phonetic augmentation pipeline and an iconic-unit cloning trick that appear to help. But the strong claim that BreezyVoice 'outperforms existing commercial TTS systems' is not backed by the evidence as reported. The PER metric is circular in a way that matters: g2pW supplies phonetic symbols at inference and also generates the reference for PER. Any system that copies g2pW gets a good PER by construction. So Table 1's PER numbers and the 61.2% reduction in Section 6.1.3 measure adherence to g2pW, not pronunciation accuracy. On that same table, BreezyVoice's PER (0.8) is not the best (M is 0.43, U is 0.69), and on the code-switching set it loses to Y on Toponyms and Sentences. The subjective test is three annotators on ten samples—a pilot, not a headline.\n\nThe paper's real strengths are the clear description of the augmentation decision tree (Figure 1) and the conditioning ablations (CFM vs LLM, with vs without speaker conditions). Those are the kind of results practitioners will actually use. The iconic-unit cloning result, even with the questionable PER reference, shows a consistent effect across 86 of 100 speakers and only a small similarity trade-off. The failure-mode analysis—stopword insertion, stuttering, hallucinated utterances—is honest and useful.\n\nThe circularity is the load-bearing flaw. It can be fixed by scoring PER against human-verified phoneme labels, or at least by adding an ASR-based metric like character error rate with a separate recognizer, and by including a CosyVoice baseline trained on the same data. The SSL-MOS score is independent of g2pW but is still a predictor, not a human judgment; it does not rescue the headline.\n\nWho should read this: anyone building TTS for Taiwanese Mandarin or adapting neural codec TTS to a lower-resourced language. The engineering insights are real, but the claims need to be scaled back or the evaluation redone. I would send it to peer review with a clear request for a stronger evaluation before acceptance.","headline":"Useful Taiwanese Mandarin TTS engineering, but the headline 'outperforms commercial systems' rests on a circular PER metric and a tiny subjective test.","tokens_in":9352,"tokens_out":3160,"would_cite":true,"duration_ms":28134,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Taiwanese Mandarin TTS uses phonetic-symbol injection and iconic-speaker unit conversion to cut voice-cloning phoneme errors by 61.2% and beat commercial systems in listening tests.","keywords":["Taiwanese Mandarin TTS","polyphone disambiguation","voice cloning","grapheme-to-phoneme","code-switching","phonetic augmentation","neural codec TTS","flow matching"],"falsifier":"Take a set of polyphonic sentences where the automatic phoneme predictor is known to be wrong, have human phoneticians transcribe the reference, synthesize BreezyVoice with and without phonetic augmentation, and compute PER against the human reference; if the gain disappears or the model systematically reproduces the predictor's errors whenever the predictor and humans disagree, the claimed disambiguation improvement is an artifact of the evaluation loop.","tokens_in":8267,"feed_emoji":"🗣️","tokens_out":9365,"duration_ms":87651,"temperature":0.7,"pith_summary":"BreezyVoice is a Taiwanese Mandarin text-to-speech system built by adapting an existing neural codec TTS stack, and the paper's central claim is that this adapted system beats four commercial TTS services on pronunciation and naturalness, in both ordinary Chinese text and Chinese sentences that switch into English. The two headline fixes are phonetic-symbol injection for polyphone disambiguation and a two-stage cloning method that routes speech through a clean reference speaker's units; the latter cuts the phoneme error rate in cloned speech from 3.4% to 1.3%, a 61.2% relative reduction, with a 2.51% dip in speaker similarity. A sympathetic reading of the paper is that adapting a strong multilingual TTS model to a local variety can be done with targeted patches rather than full retraining, and that pronunciation accuracy and speaker fidelity can be engineered separately. That matters because Taiwanese Mandarin is under-served by commercial TTS, and the methods point toward controllable, high-fidelity synthetic speech for other written forms with ambiguous character-to-sound mappings.","feed_headline":"TTS for Taiwanese Mandarin cuts cloning errors 61% via iconic units","feed_subtitle":"Feeding the model phonetic symbols and clean reference units fixes the two failure modes that break Mandarin synthesis.","key_machinery":"The load-bearing mechanisms are the phonetic augmentation pipeline and the iconic-unit cloning trick. The augmentation pipeline, styled after masked-language-model noising, applies a sentence-level and character-level decision tree that randomly replaces Chinese characters with Mandarin Phonetic Symbols during training, with a small amount of deliberate noise, so the large language model learns to treat the symbols as the authoritative pronunciation signal; at inference the same g2pW predictor supplies those symbols for the input text. Iconic Unit Augmented Speech Cloning replaces direct conditioning on a noisy speaker's units with a two-stage path: the LLM first produces speech units from a clean iconic speaker given the target text, and the OT-CFM (optimal-transport conditional flow matching) model then converts those units into the target speaker's timbre, separating content generation from voice conversion. These sit on the base infrastructure of a supervised semantic speech tokenizer that turns audio into discrete units and an OT-CFM that turns units into mel-spectrograms.","core_discovery":"The paper argues that BreezyVoice outperforms existing commercial TTS systems in both general and code-switching contexts, and that two design choices carry most of the gain. First, augmenting training text and inference text with Mandarin Phonetic Symbols predicted by a neural Mandarin grapheme-to-phoneme model, g2pW, lets the system disambiguate polyphonic characters: on 23 hard instances the base pipeline failed 8 times, while augmentation corrected all but 1. Second, Iconic Unit Augmented Speech Cloning—generating speech units from an iconic clean speaker and then converting them to the target voice—reduces phoneme error rate from 3.4% to 1.3% across 100 spontaneous speakers, improving 86 of them, at a measured cost of 2.51% in speaker similarity. The paper also reports an average speaker similarity of 92.29% over 115 speakers, and finds that the model follows injected phonetic symbols even when they contradict the character's correct pronunciation, which it reads as evidence that phonetic control drives pronunciation accuracy in this architecture.","pith_inferences":["Not claimed by the paper: because injected phonetic symbols override character identity, the same mechanism could serve as a user-facing pronunciation editor for names, rare words, and loanwords, letting people correct pronunciations by editing symbol sequences rather than retraining.","Not claimed by the paper: the reported gains are bounded by the phoneme predictor's own accuracy; a human-annotated set of hard polyphones would separate genuine disambiguation from faithful copying of the predictor.","Not claimed by the paper: the iconic-unit trick suggests a general design principle for unit-based TTS—decouple content-unit generation from timbre conversion—which could transfer to other languages with large spelling-to-pronunciation variation.","Not claimed by the paper: since Traditional Chinese characters are shared with Cantonese but pronounced differently, the phonetic-symbol augmentation could in principle be transplanted to Cantonese TTS, though the paper does not test this."],"forward_implications":["Polyphone disambiguation in Mandarin TTS can be treated as a controllable input problem: giving the model predicted phonetic symbols removes most pronunciation errors without architecture changes.","Voice cloning for long-tail or noisy speakers can be stabilized by a two-stage generate-then-convert pipeline, at a small measured cost in speaker similarity.","The same adapted pipeline handles Chinese-to-English code-switching well, with the main remaining weakness being Chinese toponyms, identified as future work.","Phonetic symbols override grapheme identity in this model, meaning a user can force a desired pronunciation by editing the symbol sequence—a practical control lever for names, rare words, and loanwords.","The failure analysis locates the dominant error source in the LLM-predicted speech units rather than the speaker embedding or conditioning spectrogram, guiding where future fixes should be aimed."],"supporting_citations":[{"why":"Supplies the CosyVoice base architecture and inference pipeline that BreezyVoice adapts, including the S3 tokenizer and LLM stage.","marker":"Du et al., 2024a"},{"why":"Provides g2pW, the neural polyphone disambiguator whose phonetic symbols are injected during training and at inference.","marker":"Chen et al., 2022"},{"why":"FormosaSpeech supplies the iconic speaker and evaluation utterances for voice-cloning tests.","marker":"Liao et al., 2020"},{"why":"Matcha-TTS-style OT-CFM converts LLM speech units into mel-spectrograms and enables the iconic-unit speaker conversion.","marker":"Mehta et al., 2024"},{"why":"The large ASR model is the first half of the cascade whose outputs are converted to phonemes for PER measurement.","marker":"Radford et al., 2023"},{"why":"TMHINT-QI is the Taiwanese Mandarin speech-quality corpus used to train the SSL-MOS predictor for objective quality scores.","marker":"Zezario et al., 2023"},{"why":"Masked-language-model noising motivates the phonetic augmentation decision tree during training.","marker":"Devlin et al., 2019"}],"fun_headline_variants":["BreezyVoice slashes polyphone errors 61% in Taiwanese TTS","Phonetic symbols cut TTS polyphone errors by 61%","Iconic units slash TTS cloning errors from 3.4% to 1.3%","BreezyVoice beats commercial TTS on Taiwanese Mandarin","Phonetic control fixes polyphone errors in Taiwanese TTS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the phoneme sequence produced by a large ASR model followed by g2pW is an accurate ground truth; because the same g2pW supplies the phonetic symbols used in inference, the reported error reductions may partly measure how faithfully BreezyVoice copies g2pW's predictions rather than how correctly it pronounces the characters.","fun_headline_variants_meta":{"raw":{"variants":["BreezyVoice slashes polyphone errors 61% in Taiwanese TTS","Phonetic symbols cut TTS polyphone errors by 61%","Iconic units slash TTS cloning errors from 3.4% to 1.3%","BreezyVoice beats commercial TTS on Taiwanese Mandarin","Phonetic control fixes polyphone errors in Taiwanese TTS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000583,"raw_usage":{"total_tokens":2730,"prompt_tokens":918,"completion_tokens":1812,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":1713}},"tokens_in":534,"tokens_out":1812,"duration_ms":11130,"temperature":1.0,"reasoning_tokens":1713,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:32:40.580416+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of polyphonic sentences where the automatic phoneme predictor is known to be wrong, have human phoneticians transcribe the reference, synthesize BreezyVoice with and without phonetic augmentation, and compute PER against the human reference; if the gain disappears or the model systematically reproduces the predictor's errors whenever the predictor and humans disagree, the claimed disambiguation improvement is an artifact of the evaluation loop.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides g2pW, the neural polyphone disambiguator whose phonetic symbols are injected during training and at inference."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FormosaSpeech supplies the iconic speaker and evaluation utterances for voice-cloning tests."}],"review_version":1}