{"id":"f60e8183-2ca2-4c50-9ec8-4f0f1555aca6","arxiv_id":"2505.05056","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper releases Teochew-Wild, the first publicly available Teochew speech corpus with orthographic and pinyin annotations, and shows it supports ASR and TTS training.","lead":"The authors built a public speech dataset for the Teochew dialect, containing 18.9 hours of real-world audio with Chinese character and pinyin transcriptions. It targets automatic speech recognition and text-to-speech for a language spoken by about 30 million people but with almost no public data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Orthographic accuracy rests on unvalidated, partly author-invented labels; no inter-annotator or external check is reported.","rationale":"The reader's weakest assumption correctly identifies the unverified accuracy of the orthographic and pinyin annotations as the key load-bearing element. The paper does contribute a genuinely scarce resource: 18.9 hours of in-the-wild Teochew speech, 20 speakers, 12,500 utterances, a community-derived G2P component, and a public repository. The DNSMOS comparison and MOS confidence intervals are real supporting evidence, and the Whisper fine-tuning result (10.01% CER on the test set) is non-trivial, so I am not arguing that the claim is false. The concern is that the central assertion of 'accurate orthographic annotations' is not demonstrated. The paper's own Section III-E admits that part of the writing system was supplemented by the authors assigning new meanings and pronunciations to rare characters. Without an external audit, independent reannotation, or inter-annotator agreement statistics, there is no way to distinguish a high-quality corpus from a self-consistent but idiosyncratic annotation scheme. The proposed concrete test would settle the question: if two independent Teochew linguists reproduce the same labels with high agreement, the claim is substantially strengthened; if they do not, the paper would need to report accuracy statistics, publish the supplemented character list, and justify the orthographic choices before the firstness claim can be accepted. This does not change the reader's conditional verdict; it reinforces it with a specific falsifiable check.","tokens_in":8950,"tokens_out":6430,"duration_ms":69678,"concrete_test":"Randomly sample 100 utterances from the released test split and have two Teochew-speaking linguists who are not authors independently transcribe each clip into Teochew characters and pinyin using only the public annotation rules in Section III-D, while explicitly marking any character they believe is an author-invented assignment. Compute character-level and pinyin/tone-level agreement with Cohen's kappa. If unweighted kappa is below 0.8 on either layer, or if the independent linguists reject a substantial fraction of the Section III-E supplemented characters, the 'accurate orthographic annotations' claim is not reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim 'first publicly available Teochew dataset with accurate orthographic annotations' depends on the ground-truth labels being accurate. The paper never measures that accuracy. Section III-D describes a two-stage process with four native transcribers and two experts, but reports no inter-annotator agreement, no independent dictionary check, and no error analysis. Section III-E states that when a pronunciation had no corresponding character, the authors selected rare characters from ancient texts and assigned them new pronunciations and meanings; those labels are therefore not independently recoverable from any external standard. In addition, the annotation pipeline in Section III-C relies on the authors' own G2P and polyphone-disambiguation tools, so the reported ASR/TTS experiments cannot detect systematic annotation errors: the models are trained and evaluated on labels produced by the same conventions. The rules also permit subjective adjustments ('adjust phonetic transcriptions based on the speaker's regional accent'), making exact labels hard to reproduce. If the annotations contain systematic errors, the reported MOS/CER/WER numbers are not evidence of a usable accurate orthographic resource. This is a missing-evidence concern rather than a demonstrated error, but it is exactly the load-bearing support for the paper's firstness and accuracy claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Teochew-Wild, a publicly released 18.9-hour Teochew speech corpus containing 12,500 utterances from 20 speakers of in-the-wild audio, annotated with both Chinese characters and Teochew pinyin. The construction pipeline includes data collection from internet multimedia, preprocessing (standardization, source separation, denoising, VAD, SNR filtering), and a two-stage annotation process. The paper also presents supplementary text front-end tools (G2P conversion, polyphone disambiguation, Teochew-Mandarin mapping) and describes a supplementation of the Teochew writing system. Experiments report TTS MOS scores of 3.52 (Tacotron2) and 3.22 (FastSpeech2), and ASR CER/WER values ranging from about 9.6% to 39.4% for Fairseq S2T and fine-tuned Whisper-medium, which the authors interpret as validating the corpus for ASR and TTS.","tokens_in":9188,"tokens_out":5636,"duration_ms":56936,"significance":"If the annotation quality is verified, Teochew-Wild would be a valuable contribution to a severely under-resourced language: it is a publicly available corpus with orthographic and pinyin labels covering formal and colloquial speech, and it ships text front-end resources that are genuinely missing for Teochew. The speech-quality comparison and the intelligible TTS output are encouraging, and the Whisper fine-tuning results suggest the corpus can support modern model adaptation. The corpus statistics are internally consistent, and the dataset release is a concrete and reusable artifact. However, the central claim of 'accurate orthographic annotations' is currently supported only by an assertion about expert annotation, not by measurement, so the corpus's value to the community depends on the validation the manuscript does not yet provide.","major_comments":[{"comment":"The abstract's central claim that Teochew-Wild is a dataset with 'accurate orthographic annotations' is load-bearing, but the manuscript reports no inter-annotator agreement, no independent dictionary check, and no error analysis for the annotations. Section III-D describes a two-stage process ending with two experts, and Section III-E states that rare characters were assigned new pronunciations and meanings when no character existed; these labels are not independently recoverable from an external standard. Additionally, the annotation pipeline in Section III-C relies on the authors' own G2P and polyphone-disambiguation tools, so the ASR/TTS experiments cannot by themselves detect systematic annotation errors. I request at least one of the following: (i) an inter-annotator agreement measure (e.g., character-level and pinyin-level agreement, Cohen's kappa) on a held-out subset; (ii) an independent expert audit of a random sample with a quantified error rate; or (iii) a public list of the supplemented characters with their assigned pronunciations and meanings, together with counts of how many utterances/characters are affected. Without such evidence, 'accurate' is an assertion rather than a demonstrated property of the resource.","section":"III-D, III-E, Abstract"},{"comment":"The SNR filtering threshold is described only as 'predefined' and is never given, even though it is one of the quality gates that determines which clips enter the corpus. The threshold must be reported for reproducibility and for users to understand the data distribution. Relatedly, the VAD merging rule is stated only as 'no shorter than 1 second and no longer than 20 seconds'; the precise algorithm for merging consecutive segments should be specified, since this parameter affects utterance boundaries and, indirectly, all downstream statistics and experiments.","section":"III-B.e"},{"comment":"The ASR effectiveness claims rest on point estimates from a 700-utterance test set without confidence intervals, significance tests, or per-speaker/per-domain breakdowns. For example, the Whisper Chinese-character CER of 10.01% versus pinyin WER of 15.03% may reflect a difference in evaluation granularity rather than model behavior, and a change of a few dozen utterances could move these numbers noticeably. I ask for bootstrap confidence intervals or an equivalent uncertainty estimate, the number of test utterances contributing to each number, and a clarification of what 'WER' means for pinyin annotations: if pinyin tokens are syllables, this should be called a pinyin syllable error rate or a word segmentation criterion must be defined.","section":"V-C, Table V"}],"minor_comments":[{"comment":"The abstract uses 'precise orthographic annotations' while the body and conclusion use 'accurate'; choose one term and define what accuracy means for this resource.","section":"Abstract and Section III"},{"comment":"The 'community-created online Teochew dictionary' is not versioned or dated; please add a retrieval date and, if possible, a snapshot so the G2P lexicon is reproducible.","section":"III-C.1"},{"comment":"The phrase 'about 12.13% of the characters in the Teochew dataset are made up of the 10 most common characters' is unclear; it should read 'the 10 most common characters account for 12.13% of character occurrences.'","section":"IV-B"},{"comment":"The MOS setup should report how many ratings each of the 50 sampled utterances received; with 12 testers each listening to 20 samples, the 95% confidence intervals are only interpretable if the per-utterance rating distribution is specified.","section":"V-B.1"},{"comment":"State clearly whether the DNSMOS scores for Teochew-Wild were computed by the authors, and note that DNSMOS is trained mainly on English and Mandarin data, so cross-language comparisons should be interpreted cautiously.","section":"Table II"},{"comment":"The supplementation of the writing system by assigning new pronunciations and meanings to rare characters should cite the specific ancient texts used and discuss how these choices become stable conventions; as written, the procedure appears ad hoc and hard for others to reproduce.","section":"III-E"},{"comment":"The dataset URL in the footnote contains a space ('teochew wild') and should be percent-encoded or replaced with a working link.","section":"Footnote 1"}],"recommendation":"major_revision","confidential_remarks":"The missing annotation validation is the key gap and is fixable within the scope of a revision; I do not see a reason to reject. The 'first publicly available' claim is strong, so I recommend that the authors verify the related-work search covers non-English publications and Southern Min/Taiwanese resources. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know up front: this is the first publicly available Teochew speech corpus with orthographic and pinyin annotations, and it is a real contribution. For a dialect with perhaps 30 million speakers and next to nothing public, 18.9 hours of in-the-wild speech across 20 speakers with Chinese-character and pinyin labels is a meaningful start. The paper also ships a G2P lexicon, polyphone disambiguation, and a Teochew-Mandarin mapping dictionary—all useful scaffolding for future work. The internal statistics (duration, character counts, phoneme coverage) are consistent, the DNSMOS comparison shows audio quality in a reasonable range, and the TTS MOS numbers come with confidence intervals. On those grounds, I take the resource as usable for prototyping.\n\nThe soft spot is the load-bearing claim: \"accurate orthographic annotations.\" There is no inter-annotator agreement, no independent dictionary check, no error analysis. Section III-E is the real problem: when a pronunciation had no corresponding character, the authors selected rare characters from ancient texts and assigned them new pronunciations and meanings. Those labels are not recoverable from any external standard. Worse, the annotation pipeline relies on the authors' own G2P and polyphone-disambiguation tools, so the ASR/TTS experiments cannot detect systematic annotation errors: the models are trained and evaluated on labels produced by the same conventions. This is missing evidence, not a demonstrated error, but it is exactly the evidence needed for a paper whose headline is \"first ... with accurate orthographic annotations.\" I would want a held-out sample re-annotated by an independent Teochew linguist or a second team, with character-level agreement reported.\n\nMinor issues: the SNR threshold for filtering is unspecified, ASR results have no confidence intervals, and the frontend tools are described but not released as runnable code. The 20-speaker count limits generalizability, though that is an inherent low-resource constraint, not a flaw.\n\nWho should read this: anyone working on Southern Min ASR/TTS, low-resource speech resource creation, or dialect orthography documentation. This is a resource paper; judge it on the resource's usability and the annotations' reliability.\n\nMy recommendation: send it to peer review. The resource is valuable enough to warrant referee time, but the revision should be required to add an independent annotation-quality check and release the tools.","headline":"A genuinely useful first resource for a low-resource dialect, whose central 'accurate annotations' claim needs independent validation before you can lean on it.","tokens_in":9684,"tokens_out":3153,"would_cite":true,"duration_ms":27465,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper constructs the first publicly available Teochew speech corpus with accurate orthographic and pinyin annotations and reports ASR and TTS experiments that support its usefulness.","keywords":["Teochew dialect","speech corpus","low-resource ASR","text-to-speech","orthographic annotation","pinyin","tone sandhi","grapheme-to-phoneme"],"falsifier":"Have an independent Teochew linguist re-annotate a random one-hour subset of the same audio with the same segmentation, then compute character- and pinyin-level agreement with the released annotations; agreement below roughly 90 percent on pinyin tones would show the ground truth is not reproducible, and the reported error rates would not be a trustworthy measure of the corpus's usefulness.","tokens_in":8759,"feed_emoji":"🗣️","tokens_out":7911,"duration_ms":69276,"temperature":0.7,"pith_summary":"Teochew-Wild supplies 18.9 hours of real-world Teochew speech from 20 speakers, with each utterance annotated in Chinese characters and Teochew pinyin. The paper's central claim is that this is the first publicly available Teochew corpus with accurate orthographic annotations, and that it is usable for both automatic speech recognition and text-to-speech. Native-speaker listening tests give synthesized speech a mean opinion score of roughly 3.2 to 3.5, and recognition experiments reach about 10 percent character error with a fine-tuned large model, suggesting that a previously unserved dialect spoken by roughly 30 million people can now be tackled by standard speech toolkits. The surrounding text-processing tools, including polyphone disambiguation, Mandarin-to-Teochew vocabulary mapping, and grapheme-to-phoneme conversion, address the missing front end for Teochew speech synthesis.","feed_headline":"First in-the-wild Teochew speech corpus with orthographic labels","feed_subtitle":"Dual character and pinyin labels make a 30-million-speaker dialect trainable for speech recognition and synthesis.","key_machinery":"The central mechanism is the dual annotation scheme: each utterance is first transcribed into semantically equivalent Mandarin, then mapped to Teochew orthography and Teochew pinyin with a rule-based front end that includes a grapheme-to-phoneme lexicon, a 10,000-plus-word polyphone disambiguation lexicon, and a 1,500-entry Mandarin-to-Teochew dialect mapping. Where a Teochew pronunciation has no standard Chinese character, the annotators select rarely used characters and assign them new pronunciations and meanings. This gives every utterance a consistent character-to-sound target, which is what allows the same corpus to drive character-based ASR, pinyin-based ASR, and TTS.","core_discovery":"The paper constructs Teochew-Wild, an 18.9-hour, 20-speaker, 12,500-utterance corpus of in-the-wild Teochew speech, annotated with both Chinese characters and Teochew pinyin. It reports that TTS models trained on the corpus receive mean opinion scores of 3.52 (Tacotron2) and 3.22 (FastSpeech2), and that ASR models reach a 16.88 percent word error rate on pinyin annotations and a 10.01 percent character error rate with a fine-tuned large model, arguing the corpus works for both tasks. The paper also claims this is the first publicly available Teochew dataset with accurate orthographic annotations, and it supplements the Teochew writing system by assigning rare Chinese characters new pronunciations and meanings where spoken words lack established written forms.","pith_inferences":["The paper's choice to assign new pronunciations and meanings to rare characters is a de facto orthographic proposal; its long-term viability depends on whether fluent speakers accept and reproduce those characters in writing.","The dataset makes possible a direct benchmark of Teochew-versus-Hokkien transfer, letting future work quantify how much cross-dialect data actually helps or hurts low-resource speech systems.","The same two-stage annotation pipeline, Mandarin semantic transcription followed by dialect orthography and pinyin, could be applied to other undocumented Sinitic dialects with a similar written-character gap.","The Whisper fine-tuning result of about 10 percent CER is an empirical claim about large-model adaptation to low-resource dialects; re-running with a matched training schedule on a different large model would test its generality."],"forward_implications":["Pinyin-based ASR on the corpus reaches 16.88 percent word error rate with a small transformer and 10.01 percent character error with a fine-tuned large model, so the annotations are usable for recognition.","Tacotron2 and FastSpeech2 trained on the corpus produce intelligible speech with mean opinion scores of 3.52 and 3.22, so the corpus supports text-to-speech.","The corpus covers 94.8 percent of Teochew's 882 initial-final combinations, providing a broad phonological basis for model training.","The released text frontend tools fill the missing Teochew front end for speech synthesis, including polyphone disambiguation and Mandarin-to-Teochew mapping.","The remaining unlabeled in-the-wild audio can support semi-supervised expansion of the corpus in future work."],"supporting_citations":[{"why":"supplies the in-the-wild preprocessing pipeline the paper adapts and the DNSMOS comparison scores for four corpora.","marker":"[2]"},{"why":"provides the Whisper-medium pre-trained model that is fine-tuned to reach the reported character error rates.","marker":"[4]"},{"why":"reports that adding Taiwanese Hokkien data did not improve Teochew speech synthesis, motivating the need for a dedicated Teochew corpus.","marker":"[13]"},{"why":"is the source of the DNSMOS scores for Aishell-3 and LibriTTS used in the quality comparison.","marker":"[20]"},{"why":"defines the DNSMOS P.835 OVRL metric used to measure the corpus's speech quality.","marker":"[26]"},{"why":"is the autoregressive TTS model used in the listening-test evaluation.","marker":"[27]"},{"why":"is the non-autoregressive TTS model used in the listening-test evaluation.","marker":"[28]"},{"why":"is the vocoder that converts predicted mel-spectrograms into waveforms for the TTS experiments.","marker":"[29]"},{"why":"provides the Fairseq S2T Transformer XS model used for the ASR experiments.","marker":"[30]"}],"fun_headline_variants":["First in-the-wild Teochew corpus with character and pinyin labels","18.9-hour Teochew corpus with dual annotations enables ASR and TTS","Teochew-Wild: first dataset with accurate orthographic labels","In-the-wild Teochew speech, 18.9h, with characters and pinyin","First Teochew-Wild corpus: ASR and TTS from orthographic labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The corpus's value rests on the expert annotations being consistent and faithful to real Teochew, including the rare characters that were assigned new pronunciations and meanings when no standard written form existed.","fun_headline_variants_meta":{"raw":{"variants":["First in-the-wild Teochew corpus with character and pinyin labels","18.9-hour Teochew corpus with dual annotations enables ASR and TTS","Teochew-Wild: first dataset with accurate orthographic labels","In-the-wild Teochew speech, 18.9h, with characters and pinyin","First Teochew-Wild corpus: ASR and TTS from orthographic labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001612,"raw_usage":{"total_tokens":6371,"prompt_tokens":848,"completion_tokens":5523,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":5415}},"tokens_in":464,"tokens_out":5523,"duration_ms":33943,"temperature":1.0,"reasoning_tokens":5415,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:13:20.690870+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have an independent Teochew linguist re-annotate a random one-hour subset of the same audio with the same segmentation, then compute character- and pinyin-level agreement with the released annotations; agreement below roughly 90 percent on pinyin tones would show the ground truth is not reproducible, and the reported error rates would not be a trustworthy measure of the corpus's usefulness.","supporting_citations":[{"cited_title":"Fastspeech 2: Fast and high-quality end-to-end text to speech,","cited_arxiv_id":null,"evidence_quote":"is the non-autoregressive TTS model used in the listening-test evaluation."},{"cited_title":"Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis,","cited_arxiv_id":null,"evidence_quote":"is the vocoder that converts predicted mel-spectrograms into waveforms for the TTS experiments."},{"cited_title":"Experiments on speech synthesis for teochew, can taiwanese help?,","cited_arxiv_id":null,"evidence_quote":"reports that adding Taiwanese Hokkien data did not improve Teochew speech synthesis, motivating the need for a dedicated Teochew corpus."},{"cited_title":"Autoprep: An automatic preprocessing framework for in-the-wild speech data,","cited_arxiv_id":null,"evidence_quote":"is the source of the DNSMOS scores for Aishell-3 and LibriTTS used in the quality comparison."},{"cited_title":"Dnsmos p. 835: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors,","cited_arxiv_id":null,"evidence_quote":"defines the DNSMOS P.835 OVRL metric used to measure the corpus's speech quality."},{"cited_title":"Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,","cited_arxiv_id":null,"evidence_quote":"is the autoregressive TTS model used in the listening-test evaluation."}],"review_version":1}