{"id":"656b22a0-aee6-43e7-b008-9627d4ccb1d0","arxiv_id":"2505.05159","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FlexSpeech is a zero-shot TTS system that predicts phoneme durations autoregressively, renders speech with flow matching, and applies direct preference optimization to durations for fast style transfer.","lead":"FlexSpeech splits speech synthesis into an autoregressive duration predictor and a non-autoregressive flow-matching sound renderer, then tunes only the duration module to change speaking style. The paper reports strong word-error rates and style transfer from about 100 samples, but the evaluation has important protocol gaps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported zero-shot WER and SMOS may be inflated if the prompt durations in Section 4.1 are the target utterance's own ground-truth durations; the paper never states that the prompt is a different utterance, so the strongest quantitative claim is not yet verifiable.","rationale":"The reader's weakest assumption is correct and is the most load-bearing because it targets the paper's strongest quantitative result. The architecture itself is not internally contradictory: the decomposition into an AR duration predictor and a NAR flow-matching acoustic model is coherent, the DPO-on-durations idea is plausible, and the ablations in Table 2 are internally consistent with the narrative that SFT and DPO both help. Those components count as support for the engineering contribution. However, the headline claim of 'SOTA stability and naturalness in zero-shot TTS' depends on Table 1, and Table 1's validity depends on the reference-prompt protocol. The phrase 'target speaker's phonemes and durations' is ambiguous in exactly the way that determines whether the WER numbers reflect zero-shot generalization or oracle-rhythm-assisted synthesis. This is not an accusation of misconduct; it is a request for a missing procedural detail and a simple control experiment. The verdict should remain CONDITIONAL: the paper is publishable in principle, but the central quantitative claim cannot be accepted as stated until the prompt protocol is clarified and the same-utterance leak is ruled out or quantified.","tokens_in":16956,"tokens_out":10173,"duration_ms":107161,"concrete_test":"Run Table 1 under two prompt conditions for the same test sets: (A) a different utterance from the same speaker as prompt, which is the standard zero-shot reference setup, and (B) the target utterance's own phonemes, durations, and mel-spectrogram as prompt. If WER and SMOS in condition (B) are substantially better than in (A), the reported numbers must be reported for condition (A) only, and the zero-shot claim adjusted accordingly. If the two conditions hardly differ, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states: 'During inference, we use the target speaker's phonemes and durations as prompts, with their mel-spectrogram providing a reference for timbre.' In the pipeline (Section 3.3), the duration model conditions on d_prompt, p_prompt, and m_ref to predict the target durations. If, for each test sample, the prompt is the very utterance being synthesized, then the model receives oracle durations for the target sentence through an in-context prompt, and the acoustic model receives a speaker embedding extracted from the ground-truth target audio. The reported WER 1.81% (test-en) and 1.20% (test-zh) and SMOS around 3.9 would no longer be zero-shot results, and the comparison with F5-TTS, MaskGCT, and other baselines would be unfair because those baselines use a separate reference utterance. The manuscript does not specify how reference utterances were selected for Common Voice, DiDiSpeech, or LibriSpeech-PC, and no evaluation code is released. This ambiguity is load-bearing because the headline stability and naturalness claim rests on these numbers. If the prompt is a separate utterance, the concern does not land; the paper needs to state that explicitly and, ideally, quantify the difference.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"FlexSpeech is a two-component text-to-speech system: a flow-matching DiT acoustic model that converts a duration-expanded phoneme sequence plus an ECAPA-TDNN speaker embedding into a mel-spectrogram, and an autoregressive transformer duration model that predicts phoneme durations by next-token prediction while conditioning on prompt durations, prompt phonemes, and a reference mel-spectrogram clip. The duration model is aligned with direct preference optimization (DPO), and the authors report state-of-the-art intelligibility (WER 1.81% on Seed-TTS test-en and 1.20% on test-zh), competitive CMOS/SMOS scores, and rapid style transfer using roughly 100 DPO pairs without retraining the acoustic model. The paper also includes ablations isolating the contributions of supervised fine-tuning and DPO in both modules. The main unresolved point is whether the evaluations are truly zero-shot, since the manuscript does not specify whether the prompt durations and reference mel-spectrogram are taken from the target utterance itself.","tokens_in":17174,"tokens_out":9066,"duration_ms":86269,"significance":"The decoupled design, with an explicitly autoregressive duration predictor feeding a non-autoregressive acoustic model, is a plausible and potentially valuable way to combine synthesis stability with prosodic expressiveness. The claim that style adaptation can be achieved with roughly 100 preference pairs by optimizing only the duration module is interesting and, if the protocol is sound, would be a practical contribution. The paper uses standard objective recognizers (Whisper and Paraformer), standard zero-shot benchmarks, and a clear ablation matrix, and it reports the main architectural hyperparameters. However, the central zero-shot and style-transfer claims are not yet verifiable because the prompt-construction protocol is ambiguous; if target-utterance durations and target-audio references are used as prompts, the reported WER and SMOS numbers would be oracle-conditioned and the comparisons to baselines would be unfair.","major_comments":[{"comment":"The zero-shot evaluation protocol is not verifiable as written. Section 4.1 states that during inference 'we use the target speaker's phonemes and durations as prompts, with their mel-spectrogram providing a reference for timbre', and Section 3.3 conditions the duration model on d_prompt, p_prompt, and m_ref. If, for each test sample, the prompt is the target utterance itself, then the duration model receives oracle durations at inference and the acoustic model receives a speaker embedding extracted from the ground-truth target audio; in that case the WER of 1.81% on test-en and 1.20% on test-zh, together with the SMOS values in Table 1, are not zero-shot results and are not comparable to baselines that use a separate reference utterance. The authors must state explicitly whether every test prompt is a distinct non-target utterance, describe how reference utterances were selected for LibriSpeech-PC, Common Voice, and DiDiSpeech, and, if target-utterance prompts were used, rerun the evaluation with held-out prompt utterances.","section":"§4.1, §3.3, Table 1"},{"comment":"The style-transfer experiment leaves the inference protocol underspecified. Section 5 does not state what durations are used as prompts for the 100 test sentences; if these are the ground-truth durations of the test sentences, the WER and SMOS curves in Figure 2 are oracle-conditioned and do not demonstrate transfer to unseen text. In addition, Section 3.4 describes the preference pairs as 'manually annotated preference duration pairs', whereas Section 5 constructs pairs by treating predicted durations as losers and ground-truth durations as winners; these are different data-construction protocols. The paper should clarify which protocol produced the main model in Table 1, how many human-annotated pairs were used, and how the test-set prompts for Figure 2 were obtained.","section":"§5, Figure 2"},{"comment":"The objective speaker-similarity results contradict the subjective similarity claim and are not discussed. FlexSpeech has the lowest SIM-O on Seed-TTS test-en (0.61/0.62 versus 0.67 for F5-TTS and 0.717 for MaskGCT), on test-zh (0.68 versus 0.76 for F5-TTS and 0.774 for MaskGCT), and on LibriSpeech-PC (0.60 versus 0.66 for F5-TTS), yet the text says 'we achieved superior SMOS scores compared to all baselines' and does not mention SIM-O. The authors should report SIM-O and SMOS together, explain the discrepancy (for example, by noting that SMOS includes prosodic pattern while SIM-O measures timbre only), or temper the speaker-similarity claim.","section":"§4.4, Table 1"},{"comment":"No confidence intervals, significance tests, or sampling variability are reported for any objective or subjective metric. The headline WER gaps are small on Seed-TTS test-en (1.81 versus 1.83 for F5-TTS), and the CMOS comparisons to non-open-source baselines rely on samples downloaded from demo pages whose matching conditions are unverified. The paper should provide error bars or significance tests for WER, SMOS, and CMOS, describe the exact procedure for generating the parallel demo-page samples, and release the evaluation code and prompt lists.","section":"§4.4, Table 1"}],"minor_comments":[{"comment":"There is a typo: 'geeneration' should be 'generation'.","section":"Footnote 1"},{"comment":"The vocoder is inconsistently referred to as both 'BigGAN' and 'BiGVGAN'; the correct name is BigVGAN.","section":"§4.2"},{"comment":"The axes of Figure 2 are not labeled; the caption and text explain that '0' means no DPO, but the figure should include unit labels for the number of DPO pairs and for the WER and SMOS scales.","section":"Figure 2"},{"comment":"The sentence 'outperformed all previous SOTA models of 1.81 in Seed-TTS test-en' is grammatically unclear and should read 'and 1.81% in Seed-TTS test-en'.","section":"§4.4"},{"comment":"The abstract says the acoustic model is trained 'given reference audio prosody and phone durations', but Section 3.1 describes inputs consisting only of the expanded phoneme sequence and a speaker embedding; please clarify that reference prosody enters through the duration predictor while the acoustic model receives only duration-expanded phonemes and timbre.","section":"Abstract and §3.1"},{"comment":"The sentence 'Since annotators only need to listen to each audio sample once to make a preference judgment' appears twice in the same paragraph; one occurrence should be removed.","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the prompt-protocol ambiguity: if the evaluations use target-utterance durations and target-audio references as prompts, the paper's central results would not be zero-shot and the quantitative comparisons would be invalid. Please require the authors to disclose the exact prompt construction for every benchmark, clarify the DPO data-construction protocol, and make the evaluation code and prompt lists available before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a credible engineering contribution, and the reader's conditional verdict is about right. The genuinely new piece is applying DPO to the autoregressive duration tokens while keeping the flow-matching acoustic model frozen, then showing the style-transfer plateau around 50-100 preference pairs. That is a useful training-time trick, and the paper gives enough detail to reproduce the idea even without code. The architecture is mostly a recombination of known pieces — flow matching, ECAPA embeddings, BigVGAN, DPO, in-context duration prompting — but the combination is not in the cited prior work, including the MegaTTS line. The WER numbers are strong if they survive scrutiny, and the ablation shows DPO and SFT both matter, which is evidence that the mechanism is doing something.\n\nSoft spots, in rough order of importance. First, the zero-shot evaluation is under-specified. Section 4.1 says 'we use the target speaker's phonemes and durations as prompts.' For a zero-shot claim, the prompt/reference must be a different utterance from the one being synthesized. If the durations come from the target utterance itself, the model is getting oracle rhythm through the prompt, and the reported 1.20/1.81 WER and SMOS are not comparable to baselines that use separate references. The stress-test note is right: this ambiguity is load-bearing. It may be that FlexSpeech uses separate utterances and the numbers are honest, but the paper has to say so and ideally quantify the gap. Second, no code or data are released, and several CMOS comparisons against closed baselines use demo-page downloads with unverified matching conditions. That makes the 'SOTA naturalness' claim hard to audit. Third, no confidence intervals or significance tests on WER or subjective scores; at 1.81 vs 1.83, that matters. None of this is internally contradictory, and I don't see a load-bearing flaw in the method itself. The circularity burden is low: DPO is empirical optimization on separate pairs, and the style-transfer test set is disjoint from DPO pairs by construction. Self-citation is background.\n\nWho this is for: TTS researchers working on zero-shot synthesis or preference alignment; useful as a source of the duration-DPO idea. It deserves serious peer review, not desk rejection, despite the evaluation gaps. I'd send it out with a request for clearer reference/prompt specification, confidence intervals, and ideally code.","headline":"A credible decoupled AR-duration + NAR-acoustic TTS with duration-level DPO; the architecture is a solid recombination, but the zero-shot evaluation needs to prove the prompt is not the target utterance.","tokens_in":17740,"tokens_out":2621,"would_cite":true,"duration_ms":27140,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FlexSpeech separates rhythm from sound to break the TTS stability-expressiveness trade-off, reaching word error rates as low as 1.20%.","keywords":["text-to-speech","zero-shot TTS","autoregressive duration prediction","non-autoregressive acoustic model","flow matching","direct preference optimization","style transfer","speech stability"],"falsifier":"Run the Seed-TTS English and Chinese benchmarks under three prompt conditions: durations from a different sentence by the same speaker, durations predicted by the model itself, and the target sentence's own ground-truth durations. If the last condition reproduces the paper's WER of 1.20% and 1.81% and clearly beats the other two, the zero-shot numbers are contaminated by oracle rhythm; if all three match, the decoupling claim is confirmed under strict zero-shot conditions.","tokens_in":16719,"feed_emoji":"🗣️","tokens_out":10194,"duration_ms":90055,"temperature":0.7,"pith_summary":"FlexSpeech claims that the long-standing trade-off between stability and expressiveness in text-to-speech can be broken by relocating the autoregressive part of generation to duration prediction alone. The model predicts each phoneme's duration one step at a time with an autoregressive encoder-decoder, then hands a duration-expanded phoneme sequence to a flow-matching acoustic model that renders a mel-spectrogram in parallel. Because audibility no longer depends on the autoregressive path getting every acoustic token right, the system reports word error rates of 1.20% on a Chinese test set and 1.81% on an English test set while matching or beating strong baselines on naturalness and speaker similarity. The same separation makes style adaptation cheap: direct preference optimization on win/lose duration pairs, using about 100 samples, transfers a new speaking style to unseen speakers without retraining the acoustic model.","feed_headline":"New TTS hits 1.20% word error, learns a style from 100 samples","feed_subtitle":"Separating duration prediction from audio generation keeps words intelligible and restyles voice from about 100 samples.","key_machinery":"The load-bearing mechanism is the decoupling of duration prediction from acoustic rendering: an autoregressive duration predictor, which treats each phoneme's duration as a discrete label (the mel-spectrogram frame count) and predicts it by next-token prediction with causal attention, and a non-autoregressive flow-matching acoustic model that generates a mel-spectrogram of exactly the length implied by the expanded phoneme sequence. The duration predictor also reads a reference mel-spectrogram clip through cross-attention and accepts prompt phoneme-duration pairs for in-context style transfer, while the acoustic model receives only a speaker embedding for timbre. This arrangement lets a lightweight DPO pass on duration pairs change rhythm and style without touching the acoustic model, which is what makes the reported fast style transfer possible.","core_discovery":"The central discovery claimed is that duration is the right carrier for both Markov dependencies and human-preference optimization in a TTS pipeline. FlexSpeech decomposes synthesis into an autoregressive duration predictor and a non-autoregressive acoustic model. The duration predictor is an encoder-decoder transformer whose decoder uses causal attention and next-token prediction to emit each phoneme's duration as a discrete frame-count label, conditioned on reference-mel prosody through cross-attention. The acoustic model is a diffusion-transformer flow-matching network (about 330M parameters) that maps Gaussian noise to a mel-spectrogram of exactly the length implied by the duration-expanded phoneme sequence, conditioned only on a speaker embedding for timbre. Because the acoustic model never estimates rhythm, it cannot accumulate token-level errors, and because the duration predictor is small, DPO on dozens of winner/loser duration pairs can align predicted rhythms with human preferences; the paper reports the lowest word error rates among the systems compared (1.20% on Seed-TTS test-zh, 1.81% on test-en) and says a storyteller style transfers with roughly 100 pairs while the acoustic model stays frozen.","pith_inferences":["If the zero-shot evaluation uses the target sentence's own ground-truth durations as prompts, as Section 4.1's wording suggests, the reported scores measure rhythm-aided synthesis rather than open-loop zero-shot synthesis; re-running the benchmarks with prompt durations drawn from a different sentence would settle the gap.","The decoupling recipe generalizes beyond speech: any generative task where a small discrete variable controls high-level structure (rhythm, timing, layout) and a separate model renders details could use the same autoregressive-control-plus-parallel-renderer split.","A practical system could cache several DPO-tuned duration models for different styles and swap them at inference time while sharing one acoustic model, making style a cheap switchable input rather than a retraining burden."],"forward_implications":["If the central claim is right, intelligibility no longer has to be traded against expressiveness: the reported WER of 1.20% (Chinese) and 1.81% (English) comes with CMOS and SMOS scores at or near the top of the compared systems.","Style transfer becomes a data-efficient operation: performance plateaus around 50–100 DPO pairs, so a production system could adapt to a new narrative or storytelling style with a few minutes of listening data and no acoustic-model retraining.","Because durations are explicit and controllable, users can reliably slow or lengthen individual phonemes or whole sentences by scaling the predicted durations, giving phoneme- and sentence-level rate control as a direct corollary.","Ablation results indicate both supervised fine-tuning and DPO contribute to the gain, so the full pipeline—pretraining on internet-scale data, cleaning with SFT, then aligning durations—is part of the claimed recipe, not just the architecture alone."],"supporting_citations":[{"why":"Provides the direct preference optimization objective used to align duration predictions with human preferences.","marker":"[39]"},{"why":"Supplies the conditional flow-matching loss that trains the non-autoregressive acoustic model.","marker":"[31]"},{"why":"BigVGAN is the vocoder that converts predicted mel-spectrograms into 48 kHz waveforms.","marker":"[28]"},{"why":"F5-TTS is the main comparison baseline and the in-context-learning approach the authors deliberately avoid in favor of a speaker embedding.","marker":"[9]"},{"why":"Mega-TTS 2 is the closest prompt-based predecessor and the comparison baseline motivating the DPO extension.","marker":"[20]"},{"why":"StoryTTS supplies the storytelling-style data used to test rapid style transfer with about 100 DPO pairs.","marker":"[32]"},{"why":"Seed-TTS provides the test-en and test-zh benchmarks and serves as a strong autoregressive baseline.","marker":"[1]"},{"why":"FastSpeech 2 establishes the explicit-duration NAR paradigm whose stability guarantees FlexSpeech preserves.","marker":"[40]"}],"fun_headline_variants":["Duration-first TTS: 1.20% WER, restyles from 100 samples","Split rhythm and timbre: FlexSpeech stable, style in 100 samples","TTS with AR duration+ NAR audio: 1.2% errors, fast style transfer","Stable, expressive TTS: separate duration, learn style from 100 samples","FlexSpeech: decouple duration from audio for stable, controllable speech"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The zero-shot evaluation assumes that the phoneme and duration prompts fed to the duration model at inference come from a different utterance than the one being synthesized, so the reported WER and SMOS are not inflated by oracle rhythm; Section 4.1 says the target speaker's own phonemes and durations are used as prompts, which leaves this assumption unstated.","fun_headline_variants_meta":{"raw":{"variants":["Duration-first TTS: 1.20% WER, restyles from 100 samples","Split rhythm and timbre: FlexSpeech stable, style in 100 samples","TTS with AR duration+ NAR audio: 1.2% errors, fast style transfer","Stable, expressive TTS: separate duration, learn style from 100 samples","FlexSpeech: decouple duration from audio for stable, controllable speech"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000922,"raw_usage":{"total_tokens":4015,"prompt_tokens":1070,"completion_tokens":2945,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":686,"completion_tokens_details":{"reasoning_tokens":2836}},"tokens_in":686,"tokens_out":2945,"duration_ms":20115,"temperature":1.0,"reasoning_tokens":2836,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:10:47.452478+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Seed-TTS English and Chinese benchmarks under three prompt conditions: durations from a different sentence by the same speaker, durations predicted by the model itself, and the target sentence's own ground-truth durations. If the last condition reproduces the paper's WER of 1.20% and 1.81% and clearly beats the other two, the zero-shot numbers are contaminated by oracle rhythm; if all three match, the decoupling claim is confirmed under strict zero-shot conditions.","supporting_citations":[{"cited_title":"Manning, Stefano Ermon, and Chelsea Finn","cited_arxiv_id":null,"evidence_quote":"Provides the direct preference optimization objective used to align duration predictions with human preferences."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the conditional flow-matching loss that trains the non-autoregressive acoustic model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Mega-TTS 2 is the closest prompt-based predecessor and the comparison baseline motivating the DPO extension."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FastSpeech 2 establishes the explicit-duration NAR paradigm whose stability guarantees FlexSpeech preserves."}],"review_version":1}