{"id":"e3b50b2c-e794-47d9-939b-e08c9c753c5e","arxiv_id":"2506.04915","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A hybrid HMM system using self-supervised features and extra text/audio data beats fine-tuned Whisper by 32 percent relative WER on Scottish Gaelic broadcast speech.","lead":"The authors built a Scottish Gaelic speech recognition system by combining hybrid HMM methods with modern self-supervised audio features, language models, and semi-supervised training. It achieved word error rates 32 percent lower than their best fine-tuned Whisper model, a practical result for low-resource languages.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 32% relative improvement may be inflated by test-set contamination: the News test episodes (16–22 Oct 2023) are never stated to be excluded from the 184h An Là semi-supervised training pool.","rationale":"The manuscript is a careful, competent case study that assembles published components (SSL features, continued pre-training, BPE units, RNN-LM rescoring, noise augmentation, and semi-supervised training) into a coherent hybrid recipe, and it gives credit to prior work. The internal story is plausible: each step yields a monotone WER improvement in Table 3, and the use of a held-out PC test set shows awareness of the need for clean evaluation. However, the central claim—32% relative WER reduction over fine-tuned Whisper—rests on a clean comparison, and the paper never states that the News test episodes were excluded from the 184h An Là semi-supervised training pool. Because the untranscribed data is itself news audio, and because the News test set is drawn from the same programme, the overlap risk is concrete rather than speculative. The reader's weakest_assumption identifies exactly this point, and I agree that it is the most load-bearing concern. A secondary limitation is that the Whisper baseline uses only LoRA fine-tuning rather than full fine-tuning; the authors explicitly acknowledge this, so it is a caveat on the strength of the headline rather than a threat to internal validity. Pending disclosure of the date range of the An Là pool, conditional acceptance remains the correct verdict.","tokens_in":9221,"tokens_out":5290,"duration_ms":58843,"concrete_test":"Obtain from the authors the broadcast dates (or programme identifiers) of the An Là recordings used to build the 184h semi-supervised training pool and check for intersection with the seven News test episodes broadcast 16–22 October 2023. If the pool ends before 16 October 2023, the concern is resolved and the reported numbers stand. If any test episode is present in the pool, rerun the final hybrid recipe and the Whisper baseline after removing the overlapping recordings from continued pre-training, semi-supervised training, and Whisper fine-tuning, and report the corrected News WER and average WER.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 defines the News test set as seven episodes of a BBC Alba evening news programme broadcast from 16 to 22 October 2023. Section 4.6 then describes semi-supervised training on 300 hours of An Là data, which after filtering yields 184 hours; Table 1 labels exactly this resource as 'News (untranscribed)'. Unlike the Prògram Choinnich test set, for which the paper states 'we also held out 2 episodes', no equivalent holdout is stated for the News test set. If any of the seven test episodes fall within the 184h pool, then the reported News WER of 10.4%, the final average WER of 12.8%, and the headline 32% relative improvement over the best fine-tuned Whisper model are all invalid for that portion of the evaluation. Note that the Whisper baseline was also trained with the same '+184 hours of automatically transcribed news', so both systems may be contaminated, but not necessarily symmetrically; the comparison cannot simply be assumed to be fair. The absence of an explicit date-range statement for the untranscribed An Là data is a material gap in the experimental protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a practitioner-oriented recipe for building an ASR system for Scottish Gaelic, a low-resource language. The central empirical claim is that a carefully tuned hybrid HMM-DNN system using self-supervised features (XLS-R with continued pre-training), BPE acoustic units, subword n-gram and RNN language models, noise augmentation, active learning, and semi-supervised training on 184 hours of untranscribed news audio outperforms a fine-tuned Whisper-Turbo baseline by 32% relative WER and the previous Tarsgrìobhadair system by 54%. The paper provides a detailed ablation chain over feature extractors, acoustic units, language models, and data augmentation strategies, with results on four Gaelic test sets.","tokens_in":9507,"tokens_out":3860,"duration_ms":45594,"significance":"If the reported comparison is valid, the paper makes a valuable practical contribution: it demonstrates a concrete, reproducible recipe in which traditional hybrid ASR components combined with modern self-supervised models beat a widely used end-to-end fine-tuning approach in a low-resource setting. The strengths include the systematic ablations, the use of publicly available SSL models and toolkits, the comparison against an existing deployed system, and the use of real broadcast data with professional transcription. However, the headline claim rests on the integrity of the evaluation protocol, especially the separation between the semi-supervised training pool and the News test set. The paper also compares against a LoRA fine-tuned Whisper model rather than a fully fine-tuned one, and it reports a validation set as part of the test average. These issues need to be resolved before the main conclusion can be accepted.","major_comments":[{"comment":"The News test set is defined in Section 3 as seven episodes broadcast from 16 to 22 October 2023, but the paper never states that these episodes were excluded from the 184 hours of untranscribed An Là data used for semi-supervised training in Section 4.6. Table 1 labels this resource as 'News (untranscribed)', and Section 4.6 only says '300 hours of An Là data' with no date range. If any of the seven test episodes fall within the semi-supervised training pool, the reported News WER of 10.4%, the average WER of 12.8%, and the headline 32% relative improvement are invalid for that part of the evaluation. Please state explicitly the date range and programme identifiers of all An Là data used for training, and if overlap exists, rerun the evaluation on held-out episodes or otherwise demonstrate that the improvement survives uncontaminated evaluation.","section":"Section 4.6 / Table 1 / Section 3"},{"comment":"The Whisper baseline is only fine-tuned with LoRA on frozen 8-bit weights (ranks 64 to 512), not with full fine-tuning. The paper itself acknowledges in Section 5 that 'we might achieve even better results when fine-tuning the whole Whisper-Turbo model.' The abstract's phrase 'our best fine-tuned Whisper model' is therefore potentially misleading. Either run a full fine-tuning experiment to establish the true Whisper baseline, or scope the claim explicitly to 'our best LoRA fine-tuned Whisper-Turbo model' throughout the abstract, results, and conclusion. As written, the 32% relative improvement may understate the gap or overstate the advantage over the best possible fine-tuned Whisper.","section":"Section 4.1 / Section 5"},{"comment":"The CLTW test set (0.9 hours) is described in Section 3 as validation data: 'Due to its small size, we decided to use this dataset as a validation set during our experiments.' Nevertheless, Table 3 reports CLTW test as one of the four evaluation sets and includes it in the average WER used for the headline comparison. Using the same data for model selection and final reporting overstates generalization. Please remove CLTW from the reported test average or present it separately as a validation set, and recalculate the relative improvements without it. At minimum, the paper should clarify how CLTW was used during development and why it remains in the reported test suite.","section":"Section 5 / Table 3 / Section 3"},{"comment":"No confidence intervals, bootstrap resampling, or significance tests are reported for any WER differences. Several test sets are very small (CLTW 0.9h, News 2.6h, PC 1.7h), and WER estimates on such sets can have wide variance. The 32% relative improvement is large, but the paper should accompany the key comparisons with uncertainty estimates (e.g., bootstrap confidence intervals) to confirm that the advantage over the Whisper baseline is not within noise. This is particularly important for the per-test-set numbers such as the News WER, which is central to the contamination concern.","section":"Sections 3 and 5"}],"minor_comments":[{"comment":"The word 'broadcasted' should be 'broadcast' in the description of the News test set, and the text has a LaTeX spacing artifact 'An L `a' that should render as 'An Là'.","section":"Section 3"},{"comment":"The BPE inventory sizes are listed as {500,1000,2000,5000}, and the choice of 1000 is justified later; consider stating the selection criterion (lowest average WER) in the same paragraph to avoid a brief ambiguity.","section":"Section 4.4"},{"comment":"The sentence 'Furthermore, we make additional use of all the untranscribed data, we decided to continue pre-training...' contains a comma splice; split it into two sentences for clarity.","section":"Section 5"},{"comment":"References [2] and [3] have missing spaces in author names ('B. Shiet al.' and 'V . Sanhet al.') due to LaTeX formatting; these should be corrected.","section":"References"},{"comment":"The text in Section 4.6 refers to 'An Là data' while Table 1 labels the same resource as 'News (untranscribed)'; please use consistent terminology, especially because the News test set is also called 'News'.","section":"Section 4.6 / Table 1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the potential overlap between the 184h semi-supervised An Là training data and the News test episodes. If the authors cannot confirm a clean separation, the headline result is invalid. Given the otherwise careful ablation and the practical value of the recipe, I believe this is fixable within revision, so I recommend major revision rather than rejection. Please also ask for the Whisper baseline to be either fully fine-tuned or explicitly scoped as LoRA-only, and for the CLTW set to be removed from or clearly separated in the reported test average."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: if you want to see how far a carefully tuned hybrid HMM+SSL pipeline goes on a low-resource language, this is one of the better write-ups I've read. The 12.8% average WER on four Gaelic test sets, with a 32% relative gain over their best Whisper-Turbo LoRA baseline, is a plausible result—but it's not fully established until the authors confirm that the seven News test episodes were not part of the 184 hours of An Là data used for semi-supervised training. That's the main thing to check.\n\nWhat's genuinely useful: the paper isolates each ingredient with ablations. BPE acoustic units for code-switched speech is a nice practical finding; continued pre-training of XLS-R on domain audio is a clear win; the paper is transparent that Whisper was only LoRA fine-tuned and explicitly says full fine-tuning might do better. It doesn't claim a new theory—it's a practitioner's guide, and it earns that title. The writing is clear and the recipe is reproducible enough that a competent Kaldi user could follow it.\n\nNow the soft spots, in proportion. The contamination issue is real. Section 3 defines the News test set as seven episodes broadcast from 16 to 22 October 2023. Section 4.6 describes semi-supervised training on 300 hours of An Là data, filtered to 184 hours, and Table 1 labels exactly that resource as 'News (untranscribed)'. The paper explicitly says the PC test episodes were held out, but never says the same for News. If the test episodes are in the 184-hour pool, the reported News WER and the headline comparison are compromised. The Whisper baseline also trains on that same automatically transcribed data, so both systems are contaminated, but there's no reason to assume the contamination is symmetric. This is not a theoretical nit; it changes whether the result is a real improvement or an artifact.\n\nOther issues are minor by comparison. CLTW test doubles as a validation set and is still reported in Table 3; that's a tuned number, not a held-out number. No confidence intervals or significance tests are given, and the test sets are small (0.9–2.6 hours). The 'best fine-tuned Whisper' claim is slightly overstated because it's LoRA-only, though the authors admit this. None of these are fatal; they just need to be stated clearly.\n\nWho this is for: practitioners building ASR for under-resourced languages, especially with broadcast data and code-switching. It deserves a serious referee, and the referee should ask for a clear statement of data provenance and, ideally, retraining without the test episodes. I'd engage with it, but I wouldn't cite the headline number until the overlap question is resolved.","headline":"A useful hybrid-ASR recipe for Gaelic with a clean ablation story, but the headline 32% gain rests on a data-leakage ambiguity the authors need to resolve.","tokens_in":10027,"tokens_out":2487,"would_cite":false,"duration_ms":30594,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hybrid HMM recognizer beats fine-tuned end-to-end models by 32% on Scottish Gaelic.","keywords":["automatic speech recognition","low-resource languages","Scottish Gaelic","hybrid HMM","self-supervised speech models","semi-supervised training","byte-pair encoding acoustic units","language model rescoring"],"falsifier":"Check the broadcast dates in the 184 hours of untranscribed news audio used for semi-supervised training and test whether any of the seven News test episodes (16-22 October 2023) appear; if any do, remove that overlap and recompute the News WER and the overall average.","tokens_in":9024,"feed_emoji":"🎙️","tokens_out":12509,"duration_ms":126196,"temperature":0.7,"pith_summary":"This paper tries to overturn a common default in low-resource speech recognition: fine-tuning a large public multilingual end-to-end model is not necessarily the best path. Working on Scottish Gaelic, the authors build a hybrid HMM recognizer whose neural acoustic model runs on features from a self-supervised speech model, uses byte-pair-encoding subword units instead of graphemes, and is rescored by language models trained on all available Gaelic text. This system reaches a 12.8% average word error rate, 32% relative better than their best fine-tuned end-to-end baseline and 54% relative better than the previous Gaelic system. The result matters because many low-resource languages have more untranscribed audio and raw text than transcribed speech, and this recipe is designed to use exactly those resources.","feed_headline":"Hybrid speech recognizer beats fine-tuned end-to-end model by 32%","feed_subtitle":"A hybrid model exploiting untranscribed audio and text beats fine-tuned end-to-end models for low-resource speech.","key_machinery":"The load-bearing mechanism is the hybrid HMM pipeline: a neural acoustic model trained with sequence-level maximum mutual information on top of features extracted from a large self-supervised speech model (one trained on unlabeled audio by predicting masked time steps) that was continued-pretrained on target-language audio; byte-pair-encoding subword units as acoustic outputs, which handle Gaelic-English code-switching better than graphemes; subword n-gram and RNN language models trained on all available text and used for lattice rescoring; and two ways of consuming untranscribed audio, continued self-supervised pre-training and pseudo-labeled semi-supervised training, with noise augmentation throughout. The gain is not from any single component but from combining all of them.","core_discovery":"The paper's central claim is that a carefully assembled hybrid model can outperform fine-tuned end-to-end models under limited transcribed data. The authors show that the hybrid's advantage comes from being able to exploit two resources end-to-end fine-tuning does not easily reach: all available text, through subword n-gram and RNN language models, and all available audio, through continued self-supervised pre-training and semi-supervised pseudo-labeling of untranscribed news broadcasts. On Scottish Gaelic the complete recipe gives a 12.8% average WER, a 32% relative improvement over the strongest fine-tuned end-to-end baseline (19.0% WER) and 54% relative over the prior best Gaelic model (28.0% WER achieved by the deployed system). The authors read these numbers as evidence for their broader claim: for low-resource languages with similar amounts of speech and text, the hybrid recipe is a better default than fine-tuning a public multilingual end-to-end model.","pith_inferences":["Beyond the paper: the same combination of continued pre-training, BPE units, language-model rescoring, and pseudo-labeling should transfer to other low-resource languages with comparable broadcast archives, although the compute for continued pre-training and the quality of the seed model are practical preconditions the paper does not quantify.","Beyond the paper: the final hybrid system could be run over much larger archives to produce pseudo-labels, and those labels could then be used to train a stronger end-to-end model, a direction the paper's future-work section points to but leaves untested.","Beyond the paper: the high WER on the BBC test set (17.7%) is concentrated in children's programmes with music and overlapping voices, so the recipe's next bottleneck is acoustically difficult conditions rather than language modelling or data quantity."],"forward_implications":["For low-resource languages with similar amounts of transcribed speech, untranscribed audio, and raw text, the tuned hybrid recipe is a better starting point than fine-tuning a large public end-to-end model.","Continued self-supervised pre-training on target-language audio improves self-supervised features even when the base model saw almost none of the target language.","Replacing grapheme acoustic units with BPE subword units reduces word error on code-switched speech.","Semi-supervised pseudo-labeling of untranscribed broadcast audio, plus noise augmentation, improves every test set and can be repeated as better seed models become available.","The reported gains suggest that just adding more untranscribed audio to the same pipeline would lower WER further."],"supporting_citations":[{"why":"Defines the multilingual end-to-end model that is fine-tuned as the main baseline to beat.","marker":"[1]"},{"why":"Supplies the hybrid HMM acoustic model recipe and training pipeline used for all proposed models.","marker":"[4]"},{"why":"Provides prior evidence that hybrid models can beat end-to-end models, which this paper extends to low-resource conditions.","marker":"[5]"},{"why":"Provides RNN language model rescoring, which yields a further relative WER gain.","marker":"[8]"},{"why":"Supplies the pre-trained self-supervised speech encoder whose features replace standard acoustic features.","marker":"[10]"},{"why":"Establishes continued self-supervised pre-training on target-language audio, which the paper shows is decisive.","marker":"[13]"},{"why":"Provides the semi-supervised training method used to exploit 184 hours of untranscribed news audio.","marker":"[15]"},{"why":"Introduces BPE subword units, used as both the language-model units and the acoustic units.","marker":"[21]"},{"why":"Defines the prior Scottish Gaelic ASR system and the CLTW train/test resources used as the starting point and a benchmark.","marker":"[24]"}],"fun_headline_variants":["Hybrid ASR beats fine-tuned Whisper by 32% on Gaelic","Low-resource ASR: Hybrid model beats end-to-end fine-tuning","Self-supervised hybrid trumps fine-tuned E2E for Gaelic","For low-resource speech, hybrid beats fine-tuning","32% WER cut: Hybrid ASR over fine-tuned Whisper"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison is valid only if the 184 hours of untranscribed news audio used for semi-supervised training do not include the seven News test episodes broadcast from 16 to 22 October 2023; the paper explicitly holds out one test set but never says this for News.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid ASR beats fine-tuned Whisper by 32% on Gaelic","Low-resource ASR: Hybrid model beats end-to-end fine-tuning","Self-supervised hybrid trumps fine-tuned E2E for Gaelic","For low-resource speech, hybrid beats fine-tuning","32% WER cut: Hybrid ASR over fine-tuned Whisper"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1192,"prompt_tokens":912,"completion_tokens":280,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":185}},"tokens_in":528,"tokens_out":280,"duration_ms":3185,"temperature":1.0,"reasoning_tokens":185,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:31:16.082208+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the broadcast dates in the 184 hours of untranscribed news audio used for semi-supervised training and test whether any of the seven News test episodes (16-22 October 2023) appear; if any do, remove that overlap and recompute the News WER and the overall average.","supporting_citations":[{"cited_title":"Whisper Fine-Tuning We fine-tuned Whisper [1] with the HuggingFace Transformers library [3] as a baseline approach for training Gaelic ASR","cited_arxiv_id":null,"evidence_quote":"Supplies the hybrid HMM acoustic model recipe and training pipeline used for all proposed models."},{"cited_title":"The Tar-sgr`ıobhadair API achieved an average WER of 28.0%","cited_arxiv_id":null,"evidence_quote":"Provides prior evidence that hybrid models can beat end-to-end models, which this paper extends to low-resource conditions."},{"cited_title":"Transformers: State-of-the-art natural language processing,","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained self-supervised speech encoder whose features replace standard acoustic features."},{"cited_title":"Madlad-400: A multilin- gual and document-level large audited dataset,","cited_arxiv_id":null,"evidence_quote":"Establishes continued self-supervised pre-training on target-language audio, which the paper shows is decisive."},{"cited_title":"Recurrent neural network based language model","cited_arxiv_id":null,"evidence_quote":"Provides the semi-supervised training method used to exploit 184 hours of untranscribed news audio."},{"cited_title":"Lightly supervised and unsupervised acoustic model training,","cited_arxiv_id":null,"evidence_quote":"Introduces BPE subword units, used as both the language-model units and the acoustic units."},{"cited_title":"Self-training and pre- training are complementary for speech recognition,","cited_arxiv_id":null,"evidence_quote":"Defines the prior Scottish Gaelic ASR system and the CLTW train/test resources used as the starting point and a benchmark."}],"review_version":1}