{"id":"a773bc30-edb5-4465-8c41-371c9db72150","arxiv_id":"2411.12587","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuning Whisper on a curated Nepali corpus improves word error rates relative to the original model on the paper's own evaluation split.","lead":"This paper fine-tunes OpenAI's Whisper speech recognition models on a newly assembled 42.9 hour Nepali dataset, reporting lower word error rates than OpenAI's baseline. It is a case study in adapting large speech models to a low-resource language, though its evaluation protocol makes the headline comparison unreliable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 6’s claimed WER gains compare Whisper on Fleurs against models evaluated on a different, non-public split; the 36.2/23.8 reductions are not established.","rationale":"The reader's weakest assumption is exactly the load-bearing flaw: the Whisper baselines are from the original paper's Fleurs test set, while the fine-tuned models are assessed on the authors' own 20% split of a mixed corpus. Every headline number depends on Table 6 being an apples-to-apples comparison, and the text gives no evidence that it is. The internal mismatch between absolute WERs and percentage reductions further obscures the result, and the lack of released data or code makes independent verification impossible. The qualitative Devanagari examples in Table 7 show the fine-tuned models produce more readable script output than Whisper's Latin transliterations, but they do not quantify WER and do not repair the invalid baseline comparison. I therefore see no reason to change the reader's REJECT verdict; the central claim remains unsupported as stated.","tokens_in":12249,"tokens_out":5750,"duration_ms":55416,"concrete_test":"Request the fine-tuned small and medium checkpoints (or the training/evaluation split metadata) from the authors, then evaluate both Whisper zero-shot and the fine-tuned models on the public Fleurs Nepali test split using identical text normalization. If the fine-tuned small/medium WERs on Fleurs-test are close to 36.2/23.8, the central claim is supported; if they resemble the Table 4 Fleurs-row values near 80.7/73.9, Table 6 used a different evaluation set and the headline reductions are invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that fine-tuning reduces WER from 69.5 to 36.2 (small) and 54.4 to 23.8 (medium) rests entirely on Table 6. The Whisper column is copied from Radford et al. and was computed on the official Fleurs test set, while Section 2.2 defines the evaluation set as a 20% holdout from the paper's own combined training corpus (Fleurs + Common Voice + SLR + custom), with no exact split definition or normalization details. The 'Ours (Fleurs)' column heading does not establish that those WERs were measured on Fleurs; the values 68.5/70.2/36.2/23.8 appear in the abstract and intro as percentage improvements but are used in Table 6 as absolute WERs, so the reported comparison is internally inconsistent. Table 4 shows fine-tuning on Fleurs alone yields small/medium WER 80.7/73.9 at 1500 epochs, whereas Table 6 reports 36.2/23.8 for the same condition with no explanation. Without releasing checkpoints or a script that evaluates Whisper and fine-tuned models on identical Fleurs test clips, the claimed reductions are confounded by test-set mismatch and cannot be verified. This is the load-bearing assumption: the two columns in Table 6 must be WERs on the same evaluation distribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes fine-tuning OpenAI's Whisper models (tiny, base, small, medium) for Nepali ASR, using a corpus that combines Google FLEURS, Mozilla Common Voice, OpenSLR SLR43/SLR143, and a self-recorded custom dataset (33.97 hours raw; 42.9 hours after preprocessing and 8000 Hz white-noise augmentation). The authors report that fine-tuning reduces WER for Nepali relative to Whisper's published FLEURS numbers, with headline absolute WERs of 36.2 (small) and 23.8 (medium). The paper also presents WER comparisons across dataset compositions, augmentation results, and qualitative prediction examples.","tokens_in":12520,"tokens_out":13629,"duration_ms":116660,"significance":"If established, the result would be a practically valuable demonstration that a small, carefully curated corpus can substantially improve Whisper on a low-resource language, and the metadata-rich custom dataset could serve as a resource for further Nepali ASR work. The paper deserves credit for compiling and manually curating a multi-source corpus, reporting explicit WER tables rather than only learning curves, and testing a simple augmentation. However, the central empirical claim is not verifiable from the paper as written: the two columns of Table 6 are not shown to be WERs on the same evaluation set, the 'Ours' column does not match any described training condition, and no code, exact split, checkpoints, or error bars are provided. The comparison with the published Whisper baseline is therefore not a sound basis for the abstract's central claim.","major_comments":[{"comment":"The headline comparison is not apples-to-apples. The Whisper column of Table 6 (101.8, 102.4, 69.5, 54.4) is taken from Radford et al., where these WERs are computed on the official FLEURS Nepali test set. The second column is, according to §2.2, evaluated on a 20% holdout drawn from the paper's own combined corpus (Fleurs + Common Voice + SLR43 + SLR143 + custom), not on the official FLEURS test set. The paper does not report the exact split, the random seed, or the normalization used for WER, and it does not report Whisper's WER on the paper's own holdout. Because the two columns are not shown to be WERs on the same evaluation distribution, the claimed reductions (69.5 to 36.2 for small, 54.4 to 23.8 for medium) are not established. This is the central claim of the paper and it is load-bearing for every downstream conclusion.","section":"§2.2, §4.3 (Table 6)"},{"comment":"The 'Ours' column of Table 6 does not correspond to any experimental condition described in the paper. Table 4 reports fine-tuning on Fleurs alone for 1500 epochs with small=80.7 and medium=73.9, whereas Table 6 reports small=36.2 and medium=23.8 for the same evaluation domain. §4.2 states that augmentation results are restricted to the small model on all_combined, yet §4.3 says 'we use augmented datasets for training the models' for all four sizes, without specifying the training set composition or epoch count that produces the Table 6 numbers. In addition, the abstract and introduction describe 68.5, 70.2, 36.2, and 23.8 as WER values, while the abstract simultaneously claims 'WER reductions of up to 36.2% on the small and 23.8% on medium models'; relative to baselines 69.5 and 54.4, the reductions would be 47.9% and 56.3%, so the abstract's percentage claim is inconsistent with the numbers in Table 6.","section":"§4.1 (Table 4) vs §4.3 (Table 6)"},{"comment":"The pipeline section states that the authors 'employ the Whisper architecture by fully training its base model rather than using a pre-trained model.' This directly contradicts the title, the abstract, and §2.2, all of which describe the contribution as fine-tuning Whisper's pre-trained models. If training starts from random initialization, the setup is not fine-tuning and the comparison in Table 6 would need to be interpreted differently; if this is a wording error, the actual initialization and the number of trainable parameters must be stated explicitly. As written, this contradiction affects the interpretation of every experimental result in the paper.","section":"§3, model initialization"},{"comment":"The evaluation protocol is underspecified and self-referential. The paper first states an 80/20 train/evaluation split, then states that 'we use evaluation data as 30% the size of the shuffled training dataset for individual corpus,' which is a different ratio. No seed, no utterance-level versus speaker-level split, and no statement about speaker overlap between partitions are provided; shuffling after the split does not remove correlation between the partitions, as the paper appears to assume. Because the evaluation set is drawn from the same corpus used for fine-tuning and is not released, the reported absolute WERs cannot be reproduced or checked for overlap with the training data. This undermines the abstract's claims of generalization to unseen Nepali speech.","section":"§2.2, evaluation protocol"}],"minor_comments":[{"comment":"The paper repeatedly describes Whisper's numbers as coming from 'Whisper's baseline models trained on Fleur's dataset'; this is inaccurate because FLEURS is an evaluation benchmark for Whisper, not part of its training data. The baseline comparison should be described as zero-shot Whisper evaluation on FLEURS.","section":"Abstract and §1"},{"comment":"The claim that 'all_combined achieved a significant WER reduction of approximately 56%' does not define the baseline of the reduction (relative to the Fleurs-only fine-tuned model, or relative to the published Whisper numbers), so the reader cannot verify it.","section":"§4.1"},{"comment":"The statement that 'the observed WER reduction is modest ( 4%)' is not supported by any numeric WER values in the text or figures; Figure 6 shows curves but no tabulated all_combined versus augmented WERs, so the 4% figure cannot be checked.","section":"§4.2"},{"comment":"The caption 'comparison between OpenAI's whisper [1] small and our fine-tuned models' is inconsistent with the table, which lists tiny, base, small, and medium rows; the second column is unlabeled and should match the 'Ours (Fleurs)' header used in Table 7.","section":"§4.3, Table 6 caption"},{"comment":"The claimed diversity by speaker age is only partially supported by Table 2: the age range is 25-60 and gender includes an 'Unknown' category, but the paper does not report the distribution of speakers across these categories, so the diversity claims are not quantified.","section":"§2.1 (Table 2) and Abstract"},{"comment":"The paper does not state how WER is computed (script normalization, punctuation handling, and whether Devanagari and Latin outputs are compared directly), which matters given the reported WERs above 100 for Latin-script outputs; additionally, the 'RTX 3090 @ 33MHz' specification contains an obviously wrong clock speed and should be corrected.","section":"§4 and hardware details"}],"recommendation":"reject","confidential_remarks":"The reader's assessment (REJECT, moderate confidence) matches my own reading. This is an empirical paper whose central claim rests on one comparison table, and the two columns of that table are not established to be WERs on the same evaluation distribution; the headline numbers also do not match any fully specified training condition in the text. The paper contains a genuinely useful dataset-construction effort, but the current evaluation protocol makes the central claim unverifiable. If the authors were to re-evaluate their checkpoints on the official FLEURS test set, report Whisper baselines on their own holdout, and reconcile Tables 4 and 6, a resubmission could be considered; in its current form the manuscript does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a routine fine-tuning paper with an evaluation that doesn't hold up. The headline WER reductions (36.2 on small, 23.8 on medium) compare Whisper's published Fleurs numbers from Radford et al. against models evaluated on the authors' own, non-public 20% split from a combined corpus. Those are different test sets, so the improvement is not established.\n\nWhat is genuinely useful: the authors assembled a 42.9-hour Nepali corpus by combining Fleurs, Common Voice, SLR43, SLR143, and their own curated recordings, applied noise augmentation, and fine-tuned Whisper at four sizes. They report overfitting with small datasets and show that the combined corpus helps. The qualitative examples make the well-known script problem concrete—Whisper's tiny and base models transliterate Nepali into Latin script, while their fine-tuned models stay in Devanagari. That is a real, practical result.\n\nThe soft spots are load-bearing. Table 6's Whisper column is from a different benchmark (official Fleurs), while their column is defined in Section 2.2 as a 20% holdout of the combined corpus. No seed, no exact split definition, no normalization details. Table 4 shows Fleurs-only fine-tuning at 1500 epochs giving small WER 80.7 and medium 73.9; Table 6 reports 36.2 and 23.8 for what looks like the same condition, with no explanation. The abstract calls these numbers 'WER reductions' while Table 6 lists them as absolute WERs. The two readings are incompatible. Without released checkpoints, the split, or an evaluation script that runs both baselines and fine-tuned models on identical clips, the central claim cannot be verified.\n\nThere are smaller issues: Section 3 says they trained the Whisper base model from scratch rather than using a pre-trained model; if that is literally true it is not fine-tuning, and if it is a typo it should be fixed. Hyperparameters (learning rate, batch size, optimizer) and error bars are missing. The 'Ours (Fleurs)' column in Table 6 is ambiguous about which training set produced it, especially since Section 4.3 says they used augmented datasets for training.\n\nWho is this for? Practitioners working on low-resource ASR who want a concrete example of dataset curation and augmentation. As a research claim, it is not ready. I would send it to a knowledgeable referee to check the baseline protocol and the table consistency, with major revision required. I would not desk-reject it because the problem is important and the flaws are fixable.","headline":"A well-intended dataset effort undermined by an apples-to-oranges baseline and internal number conflicts; the headline WER gains are not established as written.","tokens_in":13062,"tokens_out":4485,"would_cite":false,"duration_ms":41635,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning Whisper on a curated Nepali speech corpus cuts the word error rate from 69.5 to 36.2 on the small model and from 54.4 to 23.8 on the medium.","keywords":["Automatic speech recognition","Nepali","Whisper","Fine-tuning","Word error rate","Low-resource languages","Data augmentation","Speech corpus curation"],"falsifier":"Run the fine-tuned small and medium models on the official FLEURS Nepali test set, or on any external Nepali corpus untouched by training, and compute WER; if the scores sit near Whisper's published 69.5 and 54.4 instead of 36.2 and 23.8, the claimed gain is an artifact of the training and evaluation split.","tokens_in":12034,"feed_emoji":"🎙️","tokens_out":13441,"duration_ms":112290,"temperature":0.7,"pith_summary":"The paper claims that fine-tuning Whisper, a large multilingual speech-to-text model, on a purpose-built and augmented Nepali corpus substantially lowers the word error rate (WER) for Nepali at every model size tested. On the small model the fine-tuned system reaches WER 36.2, against Whisper's published 69.5 on the FLEURS multilingual benchmark; on the medium model it reaches 23.8, against 54.4. The authors attribute the gain to dataset design: longer and denser 15-30 second clips that match Whisper's input window, wider speaker and dialect diversity, and manual transcript curation, with simple white-noise augmentation adding a smaller further improvement. If the result holds, it strengthens the case that careful data curation and fine-tuning can bring large multilingual models to under-resourced languages without adding model scale.","feed_headline":"Fine-tuned Whisper cuts Nepali word error from 69.5 to 36.2","feed_subtitle":"A curated 42.9-hour corpus with longer clips and noise augmentation improves every model size.","key_machinery":"The load-bearing object is the curated corpus: 33.97 raw hours of Nepali audio drawn from four public sources plus 18.24 hours of self-recorded read and lecture speech, cleaned to 13.58 hours, then expanded to 27.17 hours with white-noise augmentation and combined into a 42.9-hour training set. The pipeline resamples audio to 16 kHz, segments it into 15-30 second clips, filters silence and corrupted audio, manually corrects transcriptions, and fully fine-tunes Whisper. Whisper is a large pre-trained transformer for speech-to-text; the fine-tuning stage adapts each checkpoint to the target language, and the augmentation stage is what the paper credits for the final few WER points and for letting training continue without overfitting.","core_discovery":"Fine-tuning Whisper's tiny, base, small, and medium checkpoints on a combined 42.9-hour Nepali corpus lowers WER at every size. The reported numbers are WER 68.5 for tiny, 70.2 for base, 36.2 for small, and 23.8 for medium, compared with Whisper's published values of 101.8, 102.4, 69.5, and 54.4. The paper also reports that the self-recorded custom corpus alone outperforms each public corpus individually and is roughly comparable to all public corpora combined, and that adding white-noise augmentation improves results by a few more WER points. The authors read these results as evidence that the limiting factor for low-resource Nepali ASR is not model capacity alone but the match between training audio and Whisper's input assumptions.","pith_inferences":["Because the fine-tuned WERs are measured on a held-out split of the combined training corpus, while Whisper's published numbers come from a different benchmark test set, a head-to-head run on a single shared Nepali test set is still needed before the absolute numbers can be treated as a benchmark claim; the ordering within the paper's own pipeline is more secure than the cross-paper comparison.","The pattern of gains, largest at tiny and base and smaller at medium once the script problem is fixed, suggests that after data is made Whisper-compatible the next bottleneck becomes model capacity; fine-tuning large or large-v2 under the same recipe would test that directly.","A testable extension is to apply the same 15-30 second segmentation and manual transcript correction to other low-resource FLEURS languages and check whether the >100 Latin-script baseline failures appear as often as they do for Nepali, which would indicate a general transcript-script problem in Whisper rather than a Nepali-specific one."],"forward_implications":["Small and medium Whisper models fine-tuned on the curated corpus reach WER 36.2 and 23.8, down from 69.5 and 54.4, which would make them practical for Nepali transcription without moving to larger checkpoints.","The custom corpus alone beats each public source and roughly matches all public sources combined, implying that curation, segment length, and speaker diversity can matter as much as raw hours for fine-tuning Whisper on a low-resource language.","Adding white noise to the combined corpus cuts WER by roughly 4 points, so a simple augmentation recipe can stretch scarce audio data and reduce overfitting.","Fine-tuned models transcribe in native Devanagari script while Whisper's tiny and base baselines often output Latin-script transliteration, which explains the >100 baseline WERs and shows fine-tuning fixes a script-level failure.","The same pipeline of combining public corpora, adding a curated set, segmenting to Whisper-compatible lengths, and augmenting can be transferred to other under-resourced languages."],"supporting_citations":[{"why":"Supplies the Whisper architecture, the pre-trained checkpoints being fine-tuned, and the published Nepali baseline error rates.","marker":"[1]"},{"why":"Provides the multilingual benchmark whose Nepali portion is used for comparison and contributes one public speech corpus to the training data.","marker":"[12]"},{"why":"Contributes a public multilingual speech corpus that supplies additional Nepali training audio.","marker":"[13]"},{"why":"Contributes an open Nepali speech corpus to the combined training data.","marker":"[14]"},{"why":"Contributes an open Nepali speech corpus to the combined training data.","marker":"[15]"},{"why":"Supplies the audio-processing tool used to apply white-noise augmentation to the corpus.","marker":"[16]"}],"fun_headline_variants":["Fine-tuned Whisper cuts Nepali WER from 69.5 to 36.2","Whisper learns Nepali: WER drops to 36.2 on small model","Curated Nepali corpus helps Whisper hit 36.2 WER","Fine-tuning Whisper on Nepali drops WER to 36.2","Nepali data variety cuts Whisper WER: 36.2 small, 23.8 medium"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison of fine-tuned and original Whisper scores assumes the two were measured on equivalent Nepali audio, but the fine-tuned numbers come from this paper's own held-out split of its combined training corpus while the published Whisper numbers come from a different benchmark test set.","fun_headline_variants_meta":{"raw":{"variants":["Fine-tuned Whisper cuts Nepali WER from 69.5 to 36.2","Whisper learns Nepali: WER drops to 36.2 on small model","Curated Nepali corpus helps Whisper hit 36.2 WER","Fine-tuning Whisper on Nepali drops WER to 36.2","Nepali data variety cuts Whisper WER: 36.2 small, 23.8 medium"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001482,"raw_usage":{"total_tokens":5963,"prompt_tokens":965,"completion_tokens":4998,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":4882}},"tokens_in":581,"tokens_out":4998,"duration_ms":33558,"temperature":1.0,"reasoning_tokens":4882,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:21:17.932556+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the fine-tuned small and medium models on the official FLEURS Nepali test set, or on any external Nepali corpus untouched by training, and compute WER; if the scores sit near Whisper's published 69.5 and 54.4 instead of 36.2 and 23.8, the claimed gain is an artifact of the training and evaluation split.","supporting_citations":[{"cited_title":"& Sutskever, I","cited_arxiv_id":null,"evidence_quote":"Supplies the Whisper architecture, the pre-trained checkpoints being fine-tuned, and the published Nepali baseline error rates."},{"cited_title":"& Bapna, A","cited_arxiv_id":null,"evidence_quote":"Provides the multilingual benchmark whose Nepali portion is used for comparison and contributes one public speech corpus to the training data."},{"cited_title":"& Weber, G","cited_arxiv_id":null,"evidence_quote":"Contributes a public multilingual speech corpus that supplies additional Nepali training audio."},{"cited_title":"& Joshi, B","cited_arxiv_id":null,"evidence_quote":"Contributes an open Nepali speech corpus to the combined training data."},{"cited_title":"& Others Torchaudio: Building blocks for audio and speech processing","cited_arxiv_id":null,"evidence_quote":"Supplies the audio-processing tool used to apply white-noise augmentation to the corpus."}],"review_version":1}