{"id":"2a4ff210-45ed-4df8-be5b-dd556bbe7c44","arxiv_id":"2506.02166","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Dhvani adapts a weakly-supervised model to Hindi pronunciation error detection using synthetic mispronunciations, reporting 82% F1 on the synthetic test set but no validation on real non-native speech.","lead":"This paper introduces Dhvani, a system that detects Hindi pronunciation errors and gives personalized articulatory feedback. It trains on synthetic mispronounced speech generated by a text-to-speech model, and evaluates on its own synthetic test data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic-only evaluation: the 82% F1 rests on unvalidated assumptions that generated mispronunciations match real non-native errors.","rationale":"The strongest claim is the 82% F1 detection benchmark. The only support for this is Table 1, which reports performance on the authors' synthetic test set described in Section 3.2. The synthetic data generation perturbs each phoneme with probability 0.05 and uses bark-small TTS. This assumes uniform, context-independent errors; real L2 speech exhibits structured transfer errors. The human study in Section 4.2 uses self-ratings and does not measure detection accuracy, so it is orthogonal. Thus the central claim rests on an untested assumption. A real-speech evaluation with expert annotations is the minimal check that would validate it. Until then, the claim cannot be accepted. The reader's REJECT verdict is appropriate; our analysis does not change it.","tokens_in":7093,"tokens_out":4782,"duration_ms":39948,"concrete_test":"Evaluate the trained Dhvani model on a newly collected, expert-annotated corpus of real non-native Hindi speech (at least two L1 backgrounds, e.g., 30–60 minutes), and compare F1 to the reported 82%; a substantial drop would show the synthetic-data evaluation does not support the central detection claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of an 82% F1 detection benchmark (Section 4.1, Table 1) is computed exclusively on synthetic data whose mispronunciations are generated by random phoneme perturbation (addition/deletion/modification, p=0.05 per phoneme) and synthesized with suno/bark-small (Section 3.2). The paper provides no evidence that this synthetic error distribution matches errors produced by real non-native Hindi learners, who typically show L1-transfer patterns (e.g., dental/retroflex substitution, loss of aspiration) rather than uniform per-phoneme corruption. If real errors differ, the reported F1 is not an estimate of real-world detection performance. The human evaluation (Section 4.2) is also self-ratings only; it does not measure whether the detector correctly identifies errors, so it cannot support the detection claim. Additionally, the 5-fold split (Section 4.1) reports speaker independence for the MUCS corpus but never specifies whether the 10 synthetic TTS voices are partitioned across folds; if the same TTS voices appear in training and test, the F1 may be further inflated by voice memorization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Dhvani, a weakly-supervised CAPT system for Hindi built from an RCNN-ARNN phoneme recognizer adapted from WEAKLY-S, a synthetic data-generation pipeline that creates mispronounced Hindi speech with suno/bark-small, and a rule-based articulatory feedback module. The authors report an 82% F1-score for phonemic error detection on a 5-fold cross-validated dataset, and a human study with 22 non-native speakers in which self-rated pronunciation improved after using the system (Wilcoxon p = 0.0022). The central scientific claim is that Dhvani is the first effective pronunciation-training system for Hindi, substantially outperforming prior work.","tokens_in":7318,"tokens_out":4672,"duration_ms":44037,"significance":"If the detection claim were validated on real non-native Hindi speech and the feedback claim on objective pronunciation measures, this would be a noteworthy contribution to CAPT for Indic languages. The synthetic mispronunciation-generation pipeline and the detailed articulatory feedback design are practical assets, and the decision to release the code and model artifacts is commendable. However, the current evidence base consists of in-distribution synthetic evaluation and self-reported ratings; no real L2 Hindi corpus is used anywhere in the paper. The significance of the reported results therefore depends entirely on unvalidated assumptions about the representativeness of synthetic errors and of self-assessed pronunciation, so the contribution as presented is not yet established.","major_comments":[{"comment":"The reported 82% F1-score is computed on a test set generated by the same synthetic pipeline used for training: Section 3.2 describes 1,000 correct and 1,000 mispronounced sentence pairs created by uniformly perturbing phonemes (addition, deletion, modification) with probability 0.05 per phoneme and synthesizing them with bark-small using ten voices. This is an in-distribution evaluation on the exact distribution the model was trained on. It cannot support the claim in Table 1 of 'setting a new benchmark for Hindi pronunciation training' because real non-native errors are structured by L1 transfer (e.g., dental/retroflex substitutions, aspiration errors) rather than uniform random corruption. The manuscript provides no external L2 corpus, no expert-annotated non-native recordings, and no analysis showing that the synthetic error distribution matches real learner errors. This is a load-bearing validity issue; the detection claim rests entirely on it.","section":"Section 4.1 / Table 1"},{"comment":"The human evaluation uses only 5-point Likert self-ratings before and after using the feedback system. It does not measure actual acoustic or phonetic accuracy, and it does not involve the Dhvani detector at all, so it cannot validate the detection claim or demonstrate genuine improvement in pronunciation. The statistically significant p-value of 0.0022 shows that participants rated themselves higher after the intervention, not that their pronunciation improved. Without an objective pronunciation measure (e.g., expert annotations or acoustic features) or a control group, the conclusion in Section 4.2 of 'a meaningful improvement in pronunciation scores' is not supported by the data.","section":"Section 4.2"},{"comment":"The description of the 5-fold split is incomplete. The paper says speaker independence is preserved with 59 speakers in training and 19 in test, but it never specifies how the 10 synthetic TTS voices are partitioned across folds, nor whether the same synthetic utterances or voices appear in both training and test sets. If the same TTS voices or the same generated utterances are shared across folds, the F1-score can be inflated by voice memorization rather than genuine generalization. The fold construction over the synthetic portion of the dataset must be documented explicitly.","section":"Section 4.1"},{"comment":"The manuscript states that architectural changes—removing the shared decoder, replacing location-sensitive attention with multi-head attention, and adding prosodic augmentations—were 'validated through rigorous ablation studies,' but no ablation results are reported anywhere. The claim that these modifications are responsible for the reported performance is therefore unverifiable. The authors should include an ablation table with F1/PER for each architectural variant, or clearly state that such experiments were not run.","section":"Section 2 and Section 4.1"},{"comment":"The comparison in Table 1 is not apples-to-apples. The prior systems are evaluated on different tasks (phoneme recognition vs. mispronunciation detection), different languages (English, German, Polish, Kannada, Bangla), different datasets (TIMIT, L2-ARCTIC, Isle, etc.), and different metrics (accuracy, PER, F1). Directly concluding 'substantial improvement and setting a new benchmark' from this table is unsupported. The authors should restrict the comparison to systems measured on the same task and metric, or explicitly discuss the methodological caveats of cross-dataset comparisons.","section":"Table 1"}],"minor_comments":[{"comment":"There is a typo in Section 1: 'enables the provision of providing comprehensive personalized feedback' should read 'enables the provision of comprehensive personalized feedback.'","section":"Introduction"},{"comment":"The error-injection process is described only by a per-phoneme probability of 0.05. The authors should report the average number of phonemes per sentence and the resulting expected number of errors per utterance, so that the difficulty of the synthetic task is interpretable.","section":"Section 3.2"},{"comment":"The statement that the standard deviation is 'less than 2% for all metrics' is vague. Please report the exact mean and standard deviation for each metric (F1, precision, recall, PER) and state whether F1 is computed at the phoneme level or the word level.","section":"Section 4.1"},{"comment":"The feedback mechanism is described as 'personalized,' but the personalization logic is not operationalized. Clarify how the feedback adapts to an individual learner's prior errors rather than simply providing the same static articulatory explanation for each mispronounced phoneme.","section":"Section 3.4"},{"comment":"The GitHub repository link is mentioned, but the paper should specify the license, the exact contents (model weights, data-generation scripts, evaluation scripts), and whether the synthetic dataset itself can be downloaded. This is important for reproducibility of the reported F1-score.","section":"Section 3.4 / Reproducibility"}],"recommendation":"reject","confidential_remarks":"The paper reads more like a system demonstration than a validated research contribution. The core detection result is circular because training and evaluation share the same synthetic data-generation process, and the user study measures self-perception rather than pronunciation accuracy. These are not presentation issues; they concern the validity of the central claims. If the authors later collect and evaluate on real non-native Hindi speech with expert annotations, the work could become a meaningful contribution to CAPT for Indic languages."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real engineering effort with a useful synthetic-data pipeline for Hindi, but the headline 82% F1 does not yet tell us anything about performance on real non-native speakers. The detection evaluation is entirely in-domain synthetic data.\n\nWhat's new: it adapts the WEAKLY-S architecture to Hindi with three explicit changes (separate decoders, multi-head attention, prosodic augmentation), builds a 1,000-pair synthetic mispronunciation corpus using suno/bark-small, and designs a detailed articulatory feedback module covering 64 Hindi phonemes. The user study, while small (n=22), shows self-rated improvement after feedback. The GitHub link suggests code will be released. Citation pattern is fine.\n\nSoft spots: (1) The 82% F1 is measured on synthetic test data generated by the same perturbation and TTS pipeline used for training. Random per-phoneme corruption (p=0.05) is not a model of L1-transfer errors like dental/retroflex substitution or aspiration loss. The paper gives no argument for representativeness. (2) \"Rigorous ablation studies\" are claimed but no ablation results are reported anywhere; that's a verifiable omission. (3) Table 1 compares against systems on different corpora, languages, and tasks; the 82% is not a like-for-like benchmark. (4) The cross-validation description does not say whether the ten synthetic voices are split across folds. If they're shared, the F1 could be inflated by voice memorization; a sentence would fix this. (5) The human study uses self-ratings, not objective phonetic measures, so it validates the feedback experience, not the detector.\n\nThis paper deserves peer review because the problem is important and the system is thoughtfully built. But acceptance should be conditional on showing real L2 evaluation, reporting the numbers behind the \"ablation\" claims, and doing honest baselines. If the real-data evaluation comes in well, this could be a useful contribution to CAPT for low-resource languages. As it stands, the central benchmark is not credible.\n\nI would not cite it in its current form. I might bring it to a reading group as an instructive example of synthetic-data overclaiming. Recommendation: send it to review, but with the clear expectation of major revision.","headline":"Real engineering, but the 82% F1 is synthetic-only and doesn't support the benchmark claim.","tokens_in":7832,"tokens_out":4252,"would_cite":false,"duration_ms":37913,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dhvani, a new Hindi pronunciation trainer, reports 82% F1 error detection and significant learner gains.","keywords":["Hindi pronunciation training","mispronunciation detection","computer-assisted pronunciation training","weakly supervised learning","speech synthesis","phoneme recognition","personalized feedback"],"falsifier":"Record a corpus of Hindi mispronunciations from genuine non-native learners, for instance speakers of Dravidian or other first languages, run Dhvani on those utterances, and compare its detection accuracy with the 82% F1 score reported on synthetic errors. If the F1 score on real learner errors is substantially lower, the system's reported performance does not transfer to its intended users.","tokens_in":6894,"feed_emoji":"🗣️","tokens_out":8474,"duration_ms":81527,"temperature":0.7,"pith_summary":"Dhvani is a computer-assisted pronunciation training system for Hindi, a language with about 600 million speakers but almost no dedicated pronunciation technology. The paper claims the system detects mispronounced phonemes with an F1 score of 82%, trained on real Hindi speech plus over three hours of synthetic mispronunciations generated by perturbing phoneme sequences and reading them with a text-to-speech model. When a mismatch is found, Dhvani returns personalized articulatory feedback dealing with tongue position, lip shape, airflow, and diagrams for the specific phoneme. In a study with 22 non-native learners, self-rated pronunciation improved significantly after using the feedback ($p = 0.0022$), with 10 of 13 difficult phonemes showing significant gains. If these results transfer to real learners, Dhvani would be the first scalable CAPT system for Hindi and a template for other Indian languages.","feed_headline":"Hindi pronunciation system detects errors at an 82% F1 score","feed_subtitle":"Dhvani pairs phoneme-level detection with articulatory feedback, and learners self-report significant improvements.","key_machinery":"The system is a weakly-supervised encoder-decoder: a recurrent-convolutional neural network (RCNN) encodes mel-spectrograms, and an attention-based recurrent neural network (ARNN) decodes phoneme sequences through a 67-unit classification head (64 Hindi phonemes plus end-of-word, end-of-sentence, and padding tokens). The paper's design decisions are to use separate decoders instead of a shared decoder across tasks, to replace location-sensitive attention with multi-head attention, and to preserve prosodic variation through energy and speed augmentation, which the authors found necessary because prosody distinguishes phonetically similar Hindi word pairs. The same decoded-phoneme sequence feeds the feedback generator, which maps each mispronounced phoneme to articulatory instructions and tongue diagrams.","core_discovery":"The paper's central claim is that Hindi's near one-to-one grapheme-to-phoneme mapping makes mispronunciation detection solvable by a single phoneme-recognition network: the system decodes an utterance into a phoneme sequence and compares it with the canonical sequence from the written Devanagari text, using classifier confidence to score error severity. To overcome the absence of real mispronounced Hindi speech, the authors synthesize training pairs by randomly adding, deleting, or modifying phonemes with probability 0.05 per phoneme and generate speech from the perturbed text with a text-to-speech system, keeping speaker embeddings matched between correct and error versions. On a held-out test set drawn from a public Hindi corpus plus synthetic errors, the model achieves an 82% F1 score under 5-fold cross-validation. The feedback study reports that learners rate the articulatory guidance highly and show statistically significant self-rated improvement.","pith_inferences":["Editorial inference: the strongest test of the system is whether its F1 score survives on genuinely non-native errors; the synthetic data generation is the core uncertainty, not the architecture.","Editorial inference: because the human evaluation relied on self-ratings, the significant $p$-value demonstrates perceived improvement; an independent measure, such as expert listening or acoustic comparison, is needed to confirm actual pronunciation gains.","Editorial inference: the same phoneme-decode-and-compare approach may transfer more readily to other transparent orthographies than to English, whose many-to-many grapheme-phoneme mapping would require a dedicated error-detection head."],"forward_implications":["Hindi learners could receive immediate, personalized pronunciation correction without a human teacher.","The same synthetic-error training recipe could be applied to other Indian languages with phonetic orthographies.","Weak supervision lowers the annotation burden: the system needs correct transcriptions plus synthetic error pairs, not expert-labeled mispronunciations.","If the self-reported gains reflect real pronunciation change, Dhvani-style feedback could be embedded in language-learning apps and digital tutors.","The reported 82% F1 score sets a benchmark against which future Hindi CAPT systems would be compared."],"supporting_citations":[{"why":"provides the weakly-supervised encoder-decoder architecture that Dhvani adapts and ablates for Hindi.","marker":"[4]"},{"why":"supplies the speech-synthesis-based method for producing synthetic mispronunciation training data.","marker":"[5]"},{"why":"provides the L1 Hindi speech corpus and transcriptions used as the real-speech training and evaluation base.","marker":"[17]"},{"why":"motivates the phoneme-perturbation strategy used to generate error-bearing transcripts for synthesis.","marker":"[19]"},{"why":"is the text-to-speech model that renders the perturbed phoneme sequences as synthetic Hindi speech.","marker":"[24]"},{"why":"supplies textbook sentences that are paired into correct and mispronounced training examples.","marker":"[25]"}],"fun_headline_variants":["Dhvani: phonemic error detection hits 82% F1 for Hindi","Hindi mispronunciation spotter reaches 82% F1 using phonemes","Weakly-supervised system targets Hindi phoneme errors, 82% F1","Phoneme-based feedback helps Hindi learners, 82% F1","Dhvani teaches Hindi pronunciation via phonemic analysis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that mispronunciations synthesized by randomly perturbing phonemes and reading them aloud with a text-to-speech model are representative of the errors real non-native Hindi learners make; the system's 82% F1 score and its usefulness depend on that resemblance.","fun_headline_variants_meta":{"raw":{"variants":["Dhvani: phonemic error detection hits 82% F1 for Hindi","Hindi mispronunciation spotter reaches 82% F1 using phonemes","Weakly-supervised system targets Hindi phoneme errors, 82% F1","Phoneme-based feedback helps Hindi learners, 82% F1","Dhvani teaches Hindi pronunciation via phonemic analysis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000306,"raw_usage":{"total_tokens":1719,"prompt_tokens":879,"completion_tokens":840,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":743}},"tokens_in":495,"tokens_out":840,"duration_ms":9109,"temperature":1.0,"reasoning_tokens":743,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:28:34.307125+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record a corpus of Hindi mispronunciations from genuine non-native learners, for instance speakers of Dravidian or other first languages, run Dhvani on those utterances, and compare its detection accuracy with the 82% F1 score reported on synthetic errors. If the F1 score on real learner errors is substantially lower, the system's reported performance does not transfer to its intended users.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the weakly-supervised encoder-decoder architecture that Dhvani adapts and ablates for Hindi."},{"cited_title":"[20, 21]","cited_arxiv_id":null,"evidence_quote":"supplies the speech-synthesis-based method for producing synthetic mispronunciation training data."},{"cited_title":"Computer-assisted pronunciation training—speech synthesis is almost all you need,","cited_arxiv_id":null,"evidence_quote":"provides the L1 Hindi speech corpus and transcriptions used as the real-speech training and evaluation base."},{"cited_title":"Detection of phonemic aspiration for spoken hindi pronunciation evaluation,","cited_arxiv_id":null,"evidence_quote":"motivates the phoneme-perturbation strategy used to generate error-bearing transcripts for synthesis."},{"cited_title":"Bangla phoneme recognition using hybrid features,","cited_arxiv_id":null,"evidence_quote":"is the text-to-speech model that renders the perturbed phoneme sequences as synthetic Hindi speech."},{"cited_title":"Read—a bangla phoneme recognition system,","cited_arxiv_id":null,"evidence_quote":"supplies textbook sentences that are paired into correct and mispronounced training examples."}],"review_version":1}