{"id":"79e21a3d-eb5a-468e-9a81-5d81b3d8085a","arxiv_id":"2412.03074","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SSL discrete tokens outperform ASR transcripts on naturalness and acoustic quality, but not intelligibility, in a two-language comparison of text-free TTS.","lead":"The paper compares speech synthesized from text transcripts, from ASR transcripts, and from self-supervised speech tokens in English and Japanese. Text inputs keep words clearer, while SSL tokens give slightly more natural and prosodically rich speech than ASR transcripts, though true text still scores best.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim not yet supported: WARP-Q/SDR in §IV-C use GT-text-synthesized speech as reference, not original source audio, so they do not directly measure preservation of original prosody/intonation.","rationale":"The reader correctly identifies a weakness in the ASR baseline, and that is a real issue. However, the single most load-bearing problem is different: the metrics used to support the central claim about acoustic-content preservation are not anchored to the original audio. §IV-C explicitly sets the reference for WARP-Q as 'synthesized speech using correct scripts,' so WARP-Q measures similarity to a canonical text-driven output, not fidelity to source prosody. This does not invalidate the empirical comparison between SSL and ASR inputs, which may still be useful, but it means the headline claim is overinterpreted without an original-audio-referenced evaluation. The reader's CONDITIONAL verdict remains appropriate; my concern strengthens the reasons for requiring additional analysis rather than changing the verdict. The paper's strengths—bilingual comparison, codebook-size and layer ablations, and transparent tables—are acknowledged, but they do not resolve this reference-mismatch issue.","tokens_in":8868,"tokens_out":7583,"duration_ms":78170,"concrete_test":"Recompute Table II's WARP-Q and SDR with the original LibriSpeech-dev/JSSS test utterances as the reference, instead of GT-text-synthesized speech, keeping all other settings fixed; optionally add F0 and duration correlation between source and synthesized outputs. If SSL's advantage over ASR disappears or reverses, the §IV-C/V conclusion that SSL better preserves acoustic content, including prosody, is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline conclusion—SSL discrete units preserve acoustic content, including prosody, better than ASR text—rests on Table II's UTMOS, WARP-Q, and SDR columns. UTMOS is a naturalness predictor, not a content-preservation metric. More importantly, §IV-C explicitly states that WARP-Q used 'synthesized speech using correct scripts' as the reference and ASR/SSL outputs as hypotheses; SDR is described relative to 'the input audio,' but no separate original-audio reference is defined. Thus, the higher WARP-Q/SDR values for SSL conditions show that SSL-Tacotron output is closer to a text-driven canonical rendering (or less noisy relative to it), not that it preserves the source utterance's acoustic content. The original test audio is available; without using it as the reference, the specific claim about prosodic and intonational preservation is untested. This is more load-bearing than the Whisper-CER concern: even a perfect ASR baseline would not make these metrics measure acoustic-content preservation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether discrete symbol representations extracted from a self-supervised learning (SSL) model (HuBERT with k-means quantization) can replace conventional text as input to a Tacotron 2 speech synthesizer, in a text-free speech synthesis setup. The authors build three input conditions—ground-truth text (GT), Whisper ASR transcripts (ASR), and SSL discrete units—for English and Japanese. They evaluate synthesized speech on intelligibility (WER/CER/PER), naturalness (UTMOS), and acoustic quality (WARP-Q, SDR), varying the SSL codebook size (50/200/1000), HuBERT layer (6/12), and language match between the SSL encoder and the synthesizer. The central claim, stated in the abstract and conclusion, is that SSL discrete units preserve acoustic content, including prosodic and intonational information, better than ASR text, whereas ASR text better preserves semantic/linguistic content.","tokens_in":9002,"tokens_out":4747,"duration_ms":42032,"significance":"If the central claim were fully supported, the paper would be a useful empirical reference for zero-resource and textless speech synthesis. Its main strengths are the systematic ablation across codebook sizes, Transformer layers, and language-matched/unmatched conditions, judged on a common benchmark with publicly available components. The rich results table (Table II) provides a useful comparison of many configurations. However, the headline claim about acoustic content preservation, specifically prosody and intonation, rests on metrics whose references do not actually measure preservation of the original utterance's acoustic properties, as detailed in the major comments. The study is therefore a useful data point, but its main conclusion is currently overstated.","major_comments":[{"comment":"The WARP-Q evaluation uses synthesized speech from correct scripts as the reference, not the original test audio, as stated in §IV-C: 'synthesized speech using correct scripts was used as a reference.' Therefore, a higher WARP-Q value for an SSL condition means that the SSL output is closer to a text-driven canonical rendering of the same sentence, not that it preserves the original utterance's prosody, intonation, or other acoustic content. The abstract's claim that SSL discrete symbols are 'superior for preserving acoustic content, including prosodic and intonational information' is not supported by the WARP-Q results, and the conclusion repeats this overstatement. Use the original source audio as the reference if acoustic-content preservation is the intended claim, or explicitly reframe the metric as 'closeness to text-conditioned synthesis.'","section":"§IV-C, Table II"},{"comment":"The SDR evaluation is described as measuring degradation 'relative to the input audio,' but the reference signal is never defined in the text. Table II reports SDR only for ASR and SSL conditions, with no SDR value for the GT condition, so the reference cannot be the GT-text-synthesized speech (which would make the GT SDR trivially 0). If the reference is the original test audio, the paper must state this explicitly and explain how the temporal alignment between the synthesized speech and the original audio is handled, given that Tacotron 2 outputs are not time-aligned to the source. As written, the SDR column is uninterpretable, and the conclusion that SSL output is less noisy than ASR output is not substantiated.","section":"§IV-C, Table II"},{"comment":"No significance tests or confidence intervals are provided for the WER, CER, and PER results, although several decisive comparisons rely on very small differences. For example, in English, the ASR condition has WER 5.41% and the best SSL condition (match-1000-L12) has WER 5.45%; these are effectively identical. Similarly, the UTMOS differences between ASR and the best SSL conditions are around 0.1–0.3 points, and the quoted confidence intervals in footnote 2 are explicitly derived from the model output standard deviation, not from human ratings, so they do not support claims of statistically significant naturalness differences. The paper should report error bars or statistical tests for all intelligibility and naturalness metrics, or weaken the claims accordingly.","section":"§IV-A and Table II"},{"comment":"The paper reports that Whisper-base has a Character Error Rate of 19.35% on the Japanese GT test audio. This makes the ASR condition in Japanese a degraded label rather than a reliable text representation. Consequently, the SSL-versus-ASR comparisons in Japanese, which contribute to the conclusion that SSL representations are superior for acoustic content, compare against an unusually weak baseline. The authors should either use a stronger ASR model for the Japanese condition, report the acoustic comparisons separately for English and Japanese while acknowledging the baseline quality, or temper the cross-language generality of the conclusion.","section":"§IV-A, Table II"}],"minor_comments":[{"comment":"The phrase 'the proposed speech synthesis is conducted using discrete symbol representations' is misleading because no new synthesis model is proposed; the paper is a comparative study. Consider rewording to 'we conduct speech synthesis using discrete symbol representations.'","section":"Abstract"},{"comment":"The typesetting of 'W ARP-Q' and 'V ocoder' (with spaces) appears to be a LaTeX artifact; these should read 'WARP-Q' and 'Vocoder' throughout.","section":"Throughout"},{"comment":"The Japanese morphological analyzer is referred to as 'Mecab'; the standard name is 'MeCab'.","section":"§III-A"},{"comment":"In the first paragraph, it is not immediately clear whether WER/CER are computed by feeding the synthesized speech into Whisper and comparing its output to the reference transcript, or by comparing Whisper's output on the input transcripts. Please clarify the exact evaluation procedure.","section":"§IV-A"},{"comment":"The sentence 'while some errors are within confidence levels' is vague; specify which comparisons are within the confidence intervals and which are not.","section":"§IV-B"},{"comment":"For the GT rows, WARP-Q and SDR are listed as '−'. The caption should note that these metrics are undefined for GT because GT speech serves as the reference condition (or state the actual reason).","section":"Table II"}],"recommendation":"major_revision","confidential_remarks":"The core empirical work is solid and the ablation is well organized, but the paper's headline claim about acoustic content preservation, especially prosody and intonation, is not supported by the reported metrics. The WARP-Q reference issue and the ambiguous SDR reference are load-bearing and require a careful re-analysis or a significant rewriting of the claims. The lack of statistical testing and the weak Japanese ASR baseline are additional concerns that should be addressed in the revision. With these fixes, the paper could become a valuable contribution to the textless speech synthesis literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick read: this paper compares ground-truth text, Whisper ASR transcripts, and HuBERT discrete units as inputs to Tacotron 2, in English and Japanese, with ablations over codebook size (50/200/1000), HuBERT layer (6/12), and matched/unmatched language pairs. The setup is clean, the corpus choices are sensible, and the table gives a lot of data. As a reference point for zero-resource TTS behavior, it has value.\n\nThe trouble is that the central conclusion — SSL units preserve acoustic content, including prosody, better than ASR text — is not actually measured. In Section IV-C, WARP-Q uses \"synthesized speech using correct scripts\" as the reference, not the original test audio. SDR is described relative to \"the input audio,\" but no original-audio reference is defined in the table. So these metrics show that SSL-conditioned output is closer to a text-driven canonical rendering than ASR-conditioned output, not that it retains the source utterance's prosody. The original audio is available; using it as reference would have made the claim testable. This is a load-bearing gap, not a quibble.\n\nThe secondary issues are real but less severe. Whisper-base's 19.35% CER on Japanese GT makes the Japanese ASR baseline a degraded label, which inflates SSL's apparent advantage there. Using the same Whisper model to produce the ASR transcripts and to compute WER/CER adds a mild self-scoring bias. UTMOS confidence intervals come from the predictor's output spread, not human ratings, as the paper itself notes in footnote 2. None of this sinks the comparison, but it calls for significance testing and re-scoped claims.\n\nWhat is genuinely new: a unified two-language comparison with codebook-size and layer ablations, plus WARP-Q/SDR added to the textless TTS evaluation toolkit. People working on low-resource or textless TTS will want to know these numbers.\n\nThis deserves a serious referee. The experiment is structurally reproducible, the numbers are transparent, and the flaw is fixable: recompute the acoustic metrics against original audio and revise the prosody claim. I'd accept it as a conference paper conditionally.","headline":"Useful empirical map of SSL-token vs ASR-text TTS, but the prosody-preservation claim is untested because WARP-Q/SDR use GT-synthesized speech as reference, not the original audio.","tokens_in":9590,"tokens_out":3167,"would_cite":false,"duration_ms":27043,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using discrete self-supervised speech tokens as the input to Tacotron 2 preserves prosody and acoustic quality better than using ASR transcripts, at the cost of some intelligibility.","keywords":["speech synthesis","self-supervised learning","discrete speech representations","text-free synthesis","zero-resource speech","Tacotron 2","HuBERT","prosody"],"falsifier":"Run the Japanese condition again with a high-accuracy ASR (character error rate below 5% on the test set) and compare the same SSL configurations against its transcripts; if the SSL advantage in UTMOS, WARP-Q, and SDR shrinks to within confidence intervals, the paper's claim that SSL tokens preserve acoustic content better than text would not hold for that language.","tokens_in":8617,"feed_emoji":"🔊","tokens_out":8487,"duration_ms":74144,"temperature":0.7,"pith_summary":"This paper asks whether speech synthesis can be driven directly by raw audio, without any transcribed text, using discrete tokens extracted by a self-supervised speech model. It builds three systems with identical Tacotron 2 backbones and varying input: ground-truth text, transcripts produced by an ASR model, and discrete symbol sequences from a self-supervised model (HuBERT features quantized with k-means). Across English and Japanese, the paper reports that ASR text better preserves semantic and phonetic content, while the self-supervised discrete tokens better preserve prosodic, intonational, and general acoustic quality, as measured by UTMOS, WARP-Q, and SDR. The finding matters because it suggests a route to zero-resource, multilingual speech synthesis that needs no text annotations.","feed_headline":"SSL tokens beat ASR text at preserving prosody","feed_subtitle":"Untranscribed audio can drive synthesis that keeps prosody better than ASR text does.","key_machinery":"The load-bearing object is the discrete symbol sequence produced by a self-supervised speech model: raw audio is fed into HuBERT, features from a chosen Transformer layer (6th or 12th) are clustered with k-means into a codebook of 50, 200, or 1000 symbols, and the resulting token stream is used in place of text as input to Tacotron 2. This representation is what isolates the comparison: the synthesis network, vocoder, and training data are held fixed, so any difference between the SSL-token system and the ASR-text system is attributable to the information content of the input representation. The paper also varies whether the SSL encoder was trained on the same language as the synthesizer and whether repeated tokens are collapsed, and reports that larger codebooks, later layers, and language-matched encoders improve metrics.","core_discovery":"Using discrete symbol representations from the SSL model instead of ASR-derived text as input to Tacotron 2 yields higher UTMOS, WARP-Q, and SDR scores, and slightly lower intelligibility metrics. In the English condition, the best SSL configuration (language-matched, 1000-cluster codebook, 12th HuBERT layer) reaches UTMOS 3.72, WARP-Q 2.93, and SDR -17.05 dB, compared with UTMOS 3.41, WARP-Q 2.71, and SDR -18.55 dB for the ASR baseline, while the ASR baseline has lower error rates in WER, PER, and CER. The paper concludes that text representations carry an advantage for semantic content, whereas discrete SSL representations are superior for acoustic content including prosody and intonation. The effect holds for both English and Japanese, with Japanese showing smaller SSL advantages and a weaker overall intelligibility level.","pith_inferences":["A direct testable extension would feed both ASR text and SSL tokens into the synthesizer; if the complementarity is real, the combined input should raise both intelligibility and acoustic quality above either alone.","Because the Japanese ASR baseline itself has a 19.35% character error rate on ground-truth test audio, the reported SSL advantage in Japanese may be partly an artifact of a weak text baseline; repeating the comparison with a stronger recognizer would show whether the acoustic advantage survives.","The quality metrics WARP-Q and SDR compare SSL/ASR outputs against synthesis from ground-truth scripts as reference; thus the 'acoustic superiority' of SSL tokens is relative to the reference generation pipeline, and a different reference could shift the ranking.","If SSL tokens encode prosodic contours well, the finding suggests using untranscribed audiobook or conversational data to build expressive synthesis systems in low-resource languages without transcription campaigns."],"forward_implications":["Speech synthesis from untranscribed audio is feasible with only small quality losses relative to text-driven synthesis, and with gains in naturalness and acoustics over ASR-derived text.","Larger discrete codebooks improve both intelligibility and naturalness, with the largest jump occurring between 50 and 200 tokens, suggesting a point of diminishing returns.","The SSL encoder's training language matters: language-matched speech2unit/unit2speech pairs produce more intelligible and more natural synthetic speech than cross-language pairs.","Choosing a deeper Transformer layer (12th vs 6th) yields more semantic content, so layer selection is a practical control knob for the linguistic/acoustic trade-off in SSL-token synthesis.","ASR-based text remains the better input when intelligibility is the priority, so the two representation families are complementary rather than one universally dominant."],"supporting_citations":[{"why":"Supplies the speech2unit/unit2speech pipeline that turns raw audio into discrete symbols and back into speech.","marker":"[6]"},{"why":"Provides the SSL model whose hidden-unit features are clustered into discrete symbols for the hypothesis input.","marker":"[9]"},{"why":"The synthesis architecture shared by all three input-representation systems, so it fixes the decoder side of the comparison.","marker":"[5]"},{"why":"The ASR model that produces the text-baseline transcripts from the same audio.","marker":"[24]"},{"why":"Prior zero-resource evaluation results showing SSL-derived units can rival text-based topline naturalness, motivating the hypothesis tested here.","marker":"[13]"},{"why":"Previous analysis of HuBERT layer properties showing later layers carry more semantic information, motivating the layer comparison.","marker":"[25]"},{"why":"Defines the metric used to evaluate acoustic similarity and quality between reference and resynthesized speech.","marker":"[29]"}],"fun_headline_variants":["SSL discrete tokens keep prosody better than ASR text","Discrete SSL beats text for acoustic fidelity in synthesis","Prosody preserved better by SSL tokens than ASR text","SSL codebooks trump text for prosody in speech synthesis","Acoustic info survives better with SSL discrete tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that ASR transcripts are a representative text baseline, but the ASR model errs on roughly one in five Japanese characters, so the text condition is a noisy proxy for true text input.","fun_headline_variants_meta":{"raw":{"variants":["SSL discrete tokens keep prosody better than ASR text","Discrete SSL beats text for acoustic fidelity in synthesis","Prosody preserved better by SSL tokens than ASR text","SSL codebooks trump text for prosody in speech synthesis","Acoustic info survives better with SSL discrete tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000532,"raw_usage":{"total_tokens":2512,"prompt_tokens":847,"completion_tokens":1665,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":1587}},"tokens_in":463,"tokens_out":1665,"duration_ms":9658,"temperature":1.0,"reasoning_tokens":1587,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:47:17.695650+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Japanese condition again with a high-accuracy ASR (character error rate below 5% on the test set) and compare the same SSL configurations against its transcripts; if the SSL advantage in UTMOS, WARP-Q, and SDR shrinks to within confidence intervals, the paper's claim that SSL tokens preserve acoustic content better than text would not hold for that language.","supporting_citations":[{"cited_title":"Self- supervised language learning from raw audio: Lessons from the zero resource speech challenge,","cited_arxiv_id":null,"evidence_quote":"Prior zero-resource evaluation results showing SSL-derived units can rival text-based topline naturalness, motivating the hypothesis tested here."},{"cited_title":"Investi- gation of robustness of hubert features from different layers to domain, accent and language variations,","cited_arxiv_id":null,"evidence_quote":"Previous analysis of HuBERT layer properties showing later layers carry more semantic information, motivating the layer comparison."},{"cited_title":"Speech quality assessment with W ARP-Q: From sim- ilarity to subsequence dynamic time warp cost,","cited_arxiv_id":null,"evidence_quote":"Defines the metric used to evaluate acoustic similarity and quality between reference and resynthesized speech."}],"review_version":1}