{"id":"a77f7fb6-aae5-44f9-9653-3c7f0e0af0d4","arxiv_id":"2501.11065","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A funnel-shaped TDNN plus 1x1 layers is reported to reach 97% language identification accuracy on ten Common Voice languages, but the evaluation is insufficient to support the claim.","lead":"This paper reports a modified neural network for spoken language recognition, reaching about 97% accuracy on a ten-language subset of Mozilla Common Voice. It matters because language identification is useful for speech tools, but the result is weakened by missing code, unclear data splits, and signs of overfitting on the test set.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 97% figure rests on validation accuracy only; the paper's own test-set confusion matrix is described as showing overfitting, and no numeric test accuracy or speaker-disjoint split is reported.","rationale":"I reviewed the paper in good faith. The strongest claim is a single accuracy figure, so the load-bearing question is whether that figure measures language recognition on unseen data. It does not, for two concrete reasons. First, the only reported number is validation accuracy; the test-set confusion matrix is acknowledged to show overfitting and no test accuracy is provided. Second, the split protocol is absent, and Common Voice's multi-utterance-per-speaker structure makes speaker leakage a real risk; the paper itself raises speaker memorization as a concern. These are not stylistic weaknesses: they determine whether the central claim is true. The reader's weakest assumption (speaker-disjoint split) is one half of the problem; the validation-only reporting is the other, so my agreement is partial. The suggested check is a single experimental rerun that would settle both issues. I do not see a need to change the reader's rejection: the paper's own evidence does not support the advertised 97% generalization claim. I am not raising any objection to the authors' integrity or to the architectural ideas; the issue is purely that the evaluation reported does not establish the headline number.","tokens_in":12168,"tokens_out":3347,"duration_ms":31941,"concrete_test":"Rerun the final model with a speaker-disjoint split: use Common Voice speaker_id metadata to assign all segments of each speaker entirely to one of train, validation, or test; train with the same hyperparameters and augmentation; report numeric top-1 accuracy and a confusion matrix on the held-out test partition. If the speaker-disjoint test accuracy is materially below 0.969 (or the validation accuracy drops when same-speaker segments are excluded), the headline claim is inflated by speaker leakage and/or by reporting validation accuracy instead of test accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('achieving a 97% accuracy rate') is supported only by Table 7's validation accuracy of 0.969 for the Final Model. Section 5 explicitly says the test-set confusion matrix 'indicat[es] potential overfitting compared to validation results,' yet no numeric test accuracy is given anywhere in the paper. Because the advertised number is a validation-set number, it does not by itself establish generalization to unseen data. In addition, Sections 3.5 and 3.6 describe only mp3-to-wav conversion, dead-segment removal, and class balancing; they never state that the train/validation/test partitions are speaker-disjoint. Common Voice contains multiple utterances per speaker, so if any speaker appears in both training and validation, an x-vector model can solve the task in part by recognizing the speaker rather than the language. The authors themselves flag that the model might 'memorize data or inadvertently focus on identifying speakers.' Together these gaps mean the 97% claim conflates validation-set fit (and possibly speaker identity) with language-recognition accuracy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an enhanced spoken language recognition (SLR) system built on an x-vector/time-delay neural network (TDNN) architecture, trained on ten languages from the Common Voice corpus. The claimed contributions are three architectural modifications: tuning TDNN context size and dilation via grid search, inserting 1x1 TDNN intermediate layers, and restructuring the network into a funnel shape. A final model with data augmentation is reported to achieve 97% accuracy. The experimental section compares incremental modifications on a validation set, with the final model reaching 0.969, and includes confusion matrices for validation and test sets. The paper discusses limitations such as linguistic ambiguity, code-switching, and data scarcity, and proposes future work on unverified data and speaker diarization.","tokens_in":12365,"tokens_out":5246,"duration_ms":53303,"significance":"If the reported 97% accuracy were a robust, externally validated result, the paper would offer a modest but useful contribution to the SLR literature, particularly for low-resource languages. The architectural description is clear, and the incremental ablation (Table 7) is a reasonable way to isolate the effect of each modification. However, the central claim is currently supported only by a validation-set number, the test-set result is acknowledged in the paper to show potential overfitting, and no speaker-disjoint split protocol is described. These issues place the entire reported performance in question. The paper does not ship code, data, or a comparison with existing SLR systems under a common protocol, so its practical value is limited at this stage.","major_comments":[{"comment":"The abstract and Section 7 claim a 97% accuracy rate, but the only number supporting this is the validation accuracy of 0.969 in Table 7. The text in Section 5 explicitly states that the test confusion matrix (Figure 4) indicates 'potential overfitting compared to validation results,' yet no numeric test accuracy or test loss is reported anywhere. Because the model's hyperparameters were selected using validation data, the 0.969 figure is a fitted value rather than an estimate of generalization to unseen data. Report the test-set accuracy, provide error bars or repeated-run statistics, and clearly state which number supports the headline claim.","section":"Section 5, Table 7, Figures 3-4"},{"comment":"No speaker-disjoint partitioning protocol is described. The dataset is drawn from Common Voice, which contains multiple utterances per speaker, and an x-vector model can potentially discriminate languages by exploiting speaker identity if the same speaker appears in both training and validation/test partitions. The paper itself acknowledges the risk of the model focusing on speakers in Section 5, but the data-preprocessing sections only mention mp3-to-wav conversion, dead-segment removal, and class balancing. Specify exactly how the train/validation/test splits were constructed, including whether any speaker appears in more than one partition, and report the degree of speaker overlap.","section":"Sections 3.5 and 3.6"},{"comment":"The grid search over TDNN context size and dilation is performed separately for each of the first three layers, with a limited number of epochs per layer, and the text concedes that the identified values 'may not be the absolute optimal.' This greedy, per-layer selection is not a joint optimization, and it is performed on the same validation set that is later used to report the final accuracy. The improvements attributed to 'grid search' in Table 7 are therefore not independent of the evaluation procedure. Provide a joint search budget, or at minimum, report the validation accuracy on a held-out split that was not used for hyperparameter selection.","section":"Section 4.2.1"}],"minor_comments":[{"comment":"The accuracy claims are inconsistent: the abstract and Section 7 say 97%, Section 4.3 says 95%, and Table 7 reports 0.969. Please reconcile these numbers and ensure the same value is used throughout.","section":"Multiple sections"},{"comment":"The phrase 'Vector-X technique' appears to be a misnomer; the paper otherwise discusses 'x-vector' embeddings. Please correct the terminology for consistency.","section":"Section 1.2"},{"comment":"The abbreviation table lists 'DNN' twice, and the second entry is spelled 'Depp Neural Network.' This should be fixed.","section":"Table 1"},{"comment":"The text states 'Table 1 provides a comparison' of i-vector, x-vector, and d-vector, but the relevant table is Table 2.","section":"Section 2.1"},{"comment":"References [26] and [34] appear to be the same work ('Stacked Long-Term TDNN for Spoken Language Recognition'), with inconsistent author attribution. Please merge or correct.","section":"References"},{"comment":"The text refers to 'Temporal Convolutional Neural Network (TDNN)' but TDNN is an abbreviation for 'Time Delay Neural Network.'","section":"Section 4.2.1"},{"comment":"There is a typo in the opening sentence: 'n our research' should be 'In our research.'","section":"Section 3.2"},{"comment":"Several dataset statistics appear implausible: French shows 12 recorded hours but 15 validated hours, Farsi 2 vs 4, and Chinese 5 vs 95. If these are not typos, please clarify how validated hours can exceed recorded hours; if they are typos, correct them.","section":"Tables 3-5"}],"recommendation":"reject","confidential_remarks":"The paper has a clearly described architecture and a reasonable ablation structure, but the experimental validation is not sufficient to support the central claim. The missing test accuracy, absent speaker-disjoint split, and greedy hyperparameter search are fundamental flaws that cannot be addressed by minor edits; substantial new experiments and a complete rewrite of the evaluation sections would be required. I do not see a path to acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is an engineering exploration, not a research advance. The authors take the standard x-vector/TDNN recipe (Snyder et al.) and modify it with three known tricks: a funnel/bottleneck layer structure, 1x1 TDNN intermediate layers, and grid-searched context/dilation settings. All three appear in the cited prior work, so what is new here is the specific combination and the numbers on a self-selected ten-language Common Voice subset. Credit where it is due: the ablation table is easy to follow, the authors are candid about possible overfitting and speaker memorization, and the related work covers the right x-vector/TDNN papers.\n\nThe soft spots are serious, though. The advertised 97% accuracy comes from Table 7, which reports validation accuracy. The paper's own Figure 4 shows the test-set confusion matrix and says it indicates potential overfitting, but no numeric test accuracy is given anywhere. That alone undercuts the headline claim. On top of that, the train/validation/test split protocol is not described: Common Voice contains multiple utterances per speaker, and if speakers are not disjoint across splits, an x-vector model can partly solve the task by recognizing speakers rather than languages. The authors themselves flag this concern, but they do not rule it out. There are also no error bars or repeated runs, the grid search is greedy per layer rather than joint, and there is no comparison to state-of-the-art language identification systems on standard benchmarks like NIST LRE or VoxLingua107. No code or data are released.\n\nNone of this makes the paper incoherent; it is a coherent and honest report of a limited engineering exercise. The citation pattern is heavy on the authors' own prior work, but the core references to Snyder, Garcia-Romero, and Tishby are appropriate. The problem is that the central quantitative claim is not supported by the evidence as presented.\n\nWho gets value from this? A speech engineer looking for a quick description of how to combine funnel layers, 1x1 TDNNs, and grid search on Common Voice might skim it. It is not a paper that should be cited for a result or a method. I would not send this version to peer review; it needs a major revision with released code and data, speaker-disjoint splits, a reported test-set number, and a comparison to existing systems before it would deserve referee time.","headline":"A readable but incremental TDNN/x-vector engineering study whose headline 97% accuracy is a validation-set number, with no test-set accuracy, speaker-disjoint splits, or SOTA comparison to back it up.","tokens_in":12904,"tokens_out":1631,"would_cite":false,"duration_ms":18321,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that restructuring a Time Delay Neural Network (TDNN) into a funnel shape, adding two 1x1 intermediate layers, and grid-searching temporal context and dilation lifts spoken language recognition to 97% accuracy across ten…","keywords":["Language Identification","Spoken Language Recognition","X-vectors","Multilingual Datasets","TDNN","Time Delay Neural Network","Common Voice","Data Augmentation"],"falsifier":"Re-run the final model on a speaker-disjoint partition of the same ten Common Voice languages, ensuring no speaker appears in more than one split; if accuracy drops far below 97%, the original number reflects speaker identity leakage rather than language recognition.","tokens_in":11968,"feed_emoji":"🗣️","tokens_out":5006,"duration_ms":45441,"temperature":0.7,"pith_summary":"This paper claims that a spoken language recognition system based on x-vector embeddings can reach 97% accuracy on ten languages from the Common Voice corpus by restructuring the Time Delay Neural Network (TDNN). The key changes are adding two 1x1 TDNN intermediate layers, reshaping the TDNN layers into a progressively narrowing \"funnel\" that reduces dimensionality, and grid-searching the context size and dilation per layer. These modifications lift validation accuracy from 0.54 in the baseline architecture to 0.969 in the final model, with data augmentation adding robustness. If the result holds, it suggests that architectural changes to embedding networks matter as much as more data for language identification.","feed_headline":"Modified TDNN lifts language ID to 97% accuracy","feed_subtitle":"Adding 1x1 layers and a funnel shape lifts validation accuracy from 0.54 to 0.97 on ten languages.","key_machinery":"The central object is the x-vector Time Delay Neural Network with a statistical pooling layer that aggregates frame-level features over the full utterance. The paper's modifications are (1) two 1x1 TDNN intermediate layers, which apply nonlinear transforms to the current frame only; (2) a funnel-shaped width schedule, 1280 down to 256 neurons per layer, which progressively reduces dimensionality and acts as an information bottleneck; and (3) a grid search over context size and dilation in the first three TDNN layers. The statistical pooling layer is what lets the network represent language-level characteristics rather than frame-level acoustics; the funnel and 1x1 layers are what the paper shows improve discrimination.","core_discovery":"On the paper's own terms, the discovery is that a standard x-vector TDNN can be converted into a high-accuracy language identifier by a specific set of architectural interventions: 1x1 TDNN layers inserted as intermediate nonlinear transforms, a funnel-shaped width schedule (1280 down to 256 neurons) that acts as an information bottleneck, and per-layer grid-searched temporal contexts and dilations. The final model, trained on balanced ten-language data from Common Voice with speed, pitch, and noise augmentation, achieves 0.969 validation accuracy, which the abstract reports as a 97% language recognition rate. The authors attribute the gains to better capture of long-term language characteristics through the temporal pooling layer combined with the funnel's hierarchical feature extraction and regularization.","pith_inferences":["If speaker-disjoint re-evaluation confirms the accuracy, the same funnel-plus-1x1 recipe could be transferred to other embedding tasks such as speaker verification or emotion recognition, where long-term temporal pooling already plays a role.","The paper does not report per-language accuracy breakdowns beyond a comment that Spanish outperformed Russian; a per-language analysis would reveal whether the 97% is evenly distributed or driven by a few easy languages.","A practical extension would test the model on unseen languages or code-switched speech, where the paper itself notes remaining challenges; the architecture would need to handle mixed-language segments."],"forward_implications":["The integrated architecture (grid search + 1x1 layers + funnel + augmentation) reaches 0.969 validation accuracy, up from 0.54 for the unmodified x-vector baseline.","Each modification contributes independently: grid search 0.85, intermediate layers 0.79, funnel structure 0.92, integrated model 0.95, and final augmented model 0.969.","The funnel structure alone provides the largest single improvement, suggesting that dimensionality reduction and bottleneck regularization are effective for language identification.","With less data than typical x-vector baselines, the modified network still outperforms the baseline, pointing to architecture rather than dataset size as the main driver."],"supporting_citations":[{"why":"supplies the baseline x-vector embedding architecture that the paper modifies","marker":"[11]"},{"why":"introduces the temporal pooling layer for capturing long-term language characteristics","marker":"[21]"},{"why":"provides the ten-language multilingual corpus from Common Voice used for training and validation","marker":"[28]"},{"why":"motivates the data augmentation techniques (speed, pitch, noise) applied to the final model","marker":"[37]"},{"why":"motivates the grid-search procedure for TDNN context size and dilation","marker":"[40]"},{"why":"provides the information-bottleneck rationale for the funnel-shaped layer widths","marker":"[48]"},{"why":"introduces the stacked TDNN approach for spoken language recognition that the paper extends","marker":"[26]"}],"fun_headline_variants":["Funnel-shaped TDNN hits 97% on language ID","From 54% to 97%: TDNN redesign for language ID","1x1 layers and funnel shape lift language ID to 97%","Multilingual TDNN achieves 97% language recognition","TDNN with funnel architecture reaches 97% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands on the assumption that the training, validation, and test segments do not share the same speakers; the paper never states that the Common Voice split is speaker-disjoint, so part of the 97% could come from recognizing voices rather than languages.","fun_headline_variants_meta":{"raw":{"variants":["Funnel-shaped TDNN hits 97% on language ID","From 54% to 97%: TDNN redesign for language ID","1x1 layers and funnel shape lift language ID to 97%","Multilingual TDNN achieves 97% language recognition","TDNN with funnel architecture reaches 97% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000321,"raw_usage":{"total_tokens":1769,"prompt_tokens":868,"completion_tokens":901,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":814}},"tokens_in":484,"tokens_out":901,"duration_ms":8398,"temperature":1.0,"reasoning_tokens":814,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:40:12.776839+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the final model on a speaker-disjoint partition of the same ten Common Voice languages, ensuring no speaker appears in more than one split; if accuracy drops far below 97%, the original number reflects speaker identity leakage rather than language recognition.","supporting_citations":[{"cited_title":"X-vectors: Robust dnn embeddings for speaker recognition","cited_arxiv_id":null,"evidence_quote":"supplies the baseline x-vector embedding architecture that the paper modifies"},{"cited_title":"Spo- ken language recognition using x-vectors","cited_arxiv_id":null,"evidence_quote":"introduces the temporal pooling layer for capturing long-term language characteristics"},{"cited_title":"Deep learning and the information bottleneck principle","cited_arxiv_id":null,"evidence_quote":"provides the information-bottleneck rationale for the funnel-shaped layer widths"},{"cited_title":"Stacked long-term tdnn for spoken language recognition","cited_arxiv_id":null,"evidence_quote":"introduces the stacked TDNN approach for spoken language recognition that the paper extends"}],"review_version":1}