REVIEW 4 major objections 6 minor 2 cited by
Whisper Finetuning on Nepali Language
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Fine-tuning Whisper on a curated Nepali speech corpus cuts the word error rate from 69.5 to 36.2 on the small model and from 54.4 to 23.8 on the medium.
desk verdict A well-intended dataset effort undermined by an apples-to-oranges baseline and internal number conflicts; the headline WER gains are not established as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the curated corpus: 33.97 raw hours of Nepali audio drawn from four public sources plus 18.24 hours of self-recorded read and lecture speech, cleaned to 13.58 hours, then expanded to 27.17 hours with white-noise augmentation and combined into a 42.9-hour training set. The pipeline resamples audio to 16 kHz, segments it into 15-30 second clips, filters silence and corrupted audio, manually corrects transcriptions, and fully fine-tunes Whisper. Whisper is a large pre-trained transformer for speech-to-text; the fine-tuning stage adapts each checkpoint to the target language, and the augmentation stage is what the paper credits for the final few WER points and for letting training continue without overfitting.
What would settle it
Run the fine-tuned small and medium models on the official FLEURS Nepali test set, or on any external Nepali corpus untouched by training, and compute WER; if the scores sit near Whisper's published 69.5 and 54.4 instead of 36.2 and 23.8, the claimed gain is an artifact of the training and evaluation split.
Extended reading notes
Core claim
Fine-tuning Whisper's tiny, base, small, and medium checkpoints on a combined 42.9-hour Nepali corpus lowers WER at every size. The reported numbers are WER 68.5 for tiny, 70.2 for base, 36.2 for small, and 23.8 for medium, compared with Whisper's published values of 101.8, 102.4, 69.5, and 54.4. The paper also reports that the self-recorded custom corpus alone outperforms each public corpus individually and is roughly comparable to all public corpora combined, and that adding white-noise augmentation improves results by a few more WER points. The authors read these results as evidence that the limiting factor for low-resource Nepali ASR is not model capacity alone but the match between training audio and Whisper's input assumptions.
Load-bearing premise
The comparison of fine-tuned and original Whisper scores assumes the two were measured on equivalent Nepali audio, but the fine-tuned numbers come from this paper's own held-out split of its combined training corpus while the published Whisper numbers come from a different benchmark test set.
Editorial extensions
If this is right
- Small and medium Whisper models fine-tuned on the curated corpus reach WER 36.2 and 23.8, down from 69.5 and 54.4, which would make them practical for Nepali transcription without moving to larger checkpoints.
- The custom corpus alone beats each public source and roughly matches all public sources combined, implying that curation, segment length, and speaker diversity can matter as much as raw hours for fine-tuning Whisper on a low-resource language.
- Adding white noise to the combined corpus cuts WER by roughly 4 points, so a simple augmentation recipe can stretch scarce audio data and reduce overfitting.
- Fine-tuned models transcribe in native Devanagari script while Whisper's tiny and base baselines often output Latin-script transliteration, which explains the >100 baseline WERs and shows fine-tuning fixes a script-level failure.
- The same pipeline of combining public corpora, adding a curated set, segmenting to Whisper-compatible lengths, and augmenting can be transferred to other under-resourced languages.
Reading between the lines
- Because the fine-tuned WERs are measured on a held-out split of the combined training corpus, while Whisper's published numbers come from a different benchmark test set, a head-to-head run on a single shared Nepali test set is still needed before the absolute numbers can be treated as a benchmark claim; the ordering within the paper's own pipeline is more secure than the cross-paper comparison.
- The pattern of gains, largest at tiny and base and smaller at medium once the script problem is fixed, suggests that after data is made Whisper-compatible the next bottleneck becomes model capacity; fine-tuning large or large-v2 under the same recipe would test that directly.
- A testable extension is to apply the same 15-30 second segmentation and manual transcript correction to other low-resource FLEURS languages and check whether the >100 Latin-script baseline failures appear as often as they do for Nepali, which would indicate a general transcript-script problem in Whisper rather than a Nepali-specific one.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes fine-tuning OpenAI's Whisper models (tiny, base, small, medium) for Nepali ASR, using a corpus that combines Google FLEURS, Mozilla Common Voice, OpenSLR SLR43/SLR143, and a self-recorded custom dataset (33.97 hours raw; 42.9 hours after preprocessing and 8000 Hz white-noise augmentation). The authors report that fine-tuning reduces WER for Nepali relative to Whisper's published FLEURS numbers, with headline absolute WERs of 36.2 (small) and 23.8 (medium). The paper also presents WER comparisons across dataset compositions, augmentation results, and qualitative prediction examples.
Significance. If established, the result would be a practically valuable demonstration that a small, carefully curated corpus can substantially improve Whisper on a low-resource language, and the metadata-rich custom dataset could serve as a resource for further Nepali ASR work. The paper deserves credit for compiling and manually curating a multi-source corpus, reporting explicit WER tables rather than only learning curves, and testing a simple augmentation. However, the central empirical claim is not verifiable from the paper as written: the two columns of Table 6 are not shown to be WERs on the same evaluation set, the 'Ours' column does not match any described training condition, and no code, exact split, checkpoints, or error bars are provided. The comparison with the published Whisper baseline is therefore not a sound basis for the abstract's central claim.
major comments (4)
- [§2.2, §4.3 (Table 6)] The headline comparison is not apples-to-apples. The Whisper column of Table 6 (101.8, 102.4, 69.5, 54.4) is taken from Radford et al., where these WERs are computed on the official FLEURS Nepali test set. The second column is, according to §2.2, evaluated on a 20% holdout drawn from the paper's own combined corpus (Fleurs + Common Voice + SLR43 + SLR143 + custom), not on the official FLEURS test set. The paper does not report the exact split, the random seed, or the normalization used for WER, and it does not report Whisper's WER on the paper's own holdout. Because the two columns are not shown to be WERs on the same evaluation distribution, the claimed reductions (69.5 to 36.2 for small, 54.4 to 23.8 for medium) are not established. This is the central claim of the paper and it is load-bearing for every downstream conclusion.
- [§4.1 (Table 4) vs §4.3 (Table 6)] The 'Ours' column of Table 6 does not correspond to any experimental condition described in the paper. Table 4 reports fine-tuning on Fleurs alone for 1500 epochs with small=80.7 and medium=73.9, whereas Table 6 reports small=36.2 and medium=23.8 for the same evaluation domain. §4.2 states that augmentation results are restricted to the small model on all_combined, yet §4.3 says 'we use augmented datasets for training the models' for all four sizes, without specifying the training set composition or epoch count that produces the Table 6 numbers. In addition, the abstract and introduction describe 68.5, 70.2, 36.2, and 23.8 as WER values, while the abstract simultaneously claims 'WER reductions of up to 36.2% on the small and 23.8% on medium models'; relative to baselines 69.5 and 54.4, the reductions would be 47.9% and 56.3%, so the abstract's percentage claim is inconsistent with the numbers in Table 6.
- [§3, model initialization] The pipeline section states that the authors 'employ the Whisper architecture by fully training its base model rather than using a pre-trained model.' This directly contradicts the title, the abstract, and §2.2, all of which describe the contribution as fine-tuning Whisper's pre-trained models. If training starts from random initialization, the setup is not fine-tuning and the comparison in Table 6 would need to be interpreted differently; if this is a wording error, the actual initialization and the number of trainable parameters must be stated explicitly. As written, this contradiction affects the interpretation of every experimental result in the paper.
- [§2.2, evaluation protocol] The evaluation protocol is underspecified and self-referential. The paper first states an 80/20 train/evaluation split, then states that 'we use evaluation data as 30% the size of the shuffled training dataset for individual corpus,' which is a different ratio. No seed, no utterance-level versus speaker-level split, and no statement about speaker overlap between partitions are provided; shuffling after the split does not remove correlation between the partitions, as the paper appears to assume. Because the evaluation set is drawn from the same corpus used for fine-tuning and is not released, the reported absolute WERs cannot be reproduced or checked for overlap with the training data. This undermines the abstract's claims of generalization to unseen Nepali speech.
minor comments (6)
- [Abstract and §1] The paper repeatedly describes Whisper's numbers as coming from 'Whisper's baseline models trained on Fleur's dataset'; this is inaccurate because FLEURS is an evaluation benchmark for Whisper, not part of its training data. The baseline comparison should be described as zero-shot Whisper evaluation on FLEURS.
- [§4.1] The claim that 'all_combined achieved a significant WER reduction of approximately 56%' does not define the baseline of the reduction (relative to the Fleurs-only fine-tuned model, or relative to the published Whisper numbers), so the reader cannot verify it.
- [§4.2] The statement that 'the observed WER reduction is modest ( 4%)' is not supported by any numeric WER values in the text or figures; Figure 6 shows curves but no tabulated all_combined versus augmented WERs, so the 4% figure cannot be checked.
- [§4.3, Table 6 caption] The caption 'comparison between OpenAI's whisper [1] small and our fine-tuned models' is inconsistent with the table, which lists tiny, base, small, and medium rows; the second column is unlabeled and should match the 'Ours (Fleurs)' header used in Table 7.
- [§2.1 (Table 2) and Abstract] The claimed diversity by speaker age is only partially supported by Table 2: the age range is 25-60 and gender includes an 'Unknown' category, but the paper does not report the distribution of speakers across these categories, so the diversity claims are not quantified.
- [§4 and hardware details] The paper does not state how WER is computed (script normalization, punctuation handling, and whether Devanagari and Latin outputs are compared directly), which matters given the reported WERs above 100 for Latin-script outputs; additionally, the 'RTX 3090 @ 33MHz' specification contains an obviously wrong clock speed and should be corrected.
Circularity Check
No circular derivation: WERs are measured outputs of an independent fine-tuning experiment, not quantities defined in terms of fitted parameters or self-citations.
full rationale
The paper reports measured Word Error Rates from fine-tuned Whisper models on Nepali speech data and compares them with baseline WERs from Radford et al. The central numbers (tiny 68.5, base 70.2, small 36.2, medium 23.8) are experimental measurements, not quantities that reduce by definition to the training inputs or to fitted parameters. There is no equation or construction in which the claimed 'prediction' is equivalent to its input: the WER is computed by decoding held-out audio and comparing with reference text, and the training objective is standard cross-entropy on Whisper's architecture. The paper also contains no load-bearing self-citation: all cited works are external (Whisper, Fleurs, Common Voice, OpenSLR, torchaudio, etc.), and no uniqueness theorem or ansatz is imported from the authors' prior work. The main weakness is evaluative rather than circular: the comparison in Table 6 appears to place Whisper's published Fleurs WER against WERs measured on the paper's own 80/20 split, and the text shifts between treating 36.2/23.8 as absolute WERs and as percentage reductions. That is a benchmark-comparability and reporting-consistency concern, not a circularity concern. No derivation chain in the paper reduces to its own premises, so the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Training epochs =
1500, 4000, 7000 depending on experiment
- Augmentation noise frequency =
8000 Hz white noise
- Silence removal threshold =
1 second
assumptions (3)
- domain assumption The Whisper architecture and its feature extractor/tokenizer are suitable for Nepali speech.
- domain assumption Manual transcription corrections produce accurate ground truth labels.
- ad hoc to paper A random 80/20 split of the combined corpus yields an evaluation set representative of general Nepali speech.
Cite this review
Pith. "Pith review of Whisper Finetuning on Nepali Language." pith.science (2026). https://pith.science/paper/7GUOXLWU
@misc{pith2026241112587,
author = {Pith},
title = {Pith review of: Whisper Finetuning on Nepali Language},
year = {2026},
howpublished = {\url{https://pith.science/paper/7GUOXLWU}},
note = {Machine review of arXiv:2411.12587}
}
read the original abstract
Despite the growing advancements in Automatic Speech Recognition (ASR) models, the development of robust models for underrepresented languages, such as Nepali, remains a challenge. This research focuses on making an exhaustive and generalized dataset followed by fine-tuning OpenAI's Whisper models of different sizes to improve transcription (speech-to-text) accuracy for the Nepali language. We leverage publicly available ASR datasets and self-recorded custom datasets with a diverse range of accents, dialects, and speaking styles further enriched through augmentation. Our experimental results demonstrate that fine-tuning Whisper models on our curated custom dataset substantially reduces the Word Error Rate (WER) across all model sizes attributed to larger data variations in terms of speaker's age, gender, and sentiment, acoustic environment, dialect, denser audio segments (15-30 seconds) that are more compatible with Whisper's input, and manual curation of audios and transcriptions. Notably, our approach outperforms Whisper's baseline models trained on Fleur's dataset, achieving WER reductions of up to 36.2% on the small and 23.8% on medium models. Furthermore, we show that data augmentation plays a significant role in enhancing model robustness. Our approach underlines the importance of dataset quality, variation, and augmentation in the adaptation of state-of-the-art models to underrepresented languages for developing accurate ASR systems.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 2 Pith papers
-
Breaking the Transcription Bottleneck: Fine-tuning ASR Models for Extremely Low-Resource Fieldwork Languages
Fine-tuned MMS outperforms XLS-R on fieldwork ASR with less than one hour of training data, while XLS-R reaches parity beyond one hour.
-
A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data
Fine-tuning Whisper-large-v2 on 10,000 hours of synthesized Mandarin plus small real English/code-switching sets yields Twister, cutting mixed error rate by up to 56% on code-switching and 19% on Taiwanese Mandarin.
Reference graph
Works this paper leans on
-
[1]
Radford, A., Kim, J., Xu, T., Brockman, G., McLeavey, C. & Sutskever, I. Robust speech recognition via large-scale weak supervision. International Conference On Ma- chine Learning. pp. 28492-28518 (2023)
work page 2023
-
[2]
Narayanan, A., Misra, A., Sim, K., Pundak, G., Tripathi, A., Elfeky, M., Haghani, P., Strohman, T. & Bacchiani, M. Toward domain-invariant speech recognition via large scale training. 2018 IEEE Spoken Language Technology Workshop (SLT) . pp. 441-447 (2018)
work page 2018
-
[3]
& Others Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training
Hsu, W., Sriram, A., Baevski, A., Likhomanenko, T., Xu, Q., Pratap, V., Kahn, J., Lee, A., Collobert, R., Synnaeve, G. & Others Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training. ArXiv Preprint ArXiv:2104.01027 . (2021)
arXiv 2021
-
[4]
& Others Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio
Chen, G., Chai, S., Wang, G., Du, J., Zhang, W., Weng, C., Su, D., Povey, D., Trmal, J., Zhang, J. & Others Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. ArXiv Preprint ArXiv:2106.06909 . (2021)
arXiv 2021
-
[5]
Bhanushali, A., Bridgman, G., Deekshitha, G., Ghosh, P., Kumar, P., Kumar, S., Kolladath, A., Ravi, N., Seth, A., Singh, A. & Others Gram Vaani ASR Challenge on spontaneous telephone speech recordings in regional variations of Hindi. Proceedings Of The Annual Conference Of The International Speech Communication Association, INTERSPEECH. 2022 pp. 3548-3552 (2022)
work page 2022
-
[6]
Sailor, H., Patil, A. & Patil, H. Advances in Low Resource ASR: A Deep Learning Perspective.. SLTU. pp. 15-19 (2018)
work page 2018
-
[7]
Parikh, A., Bosch, L., Heuvel, H. & Tejedor-Garcı́a, C. Comparing Modular and End- To-End Approaches in ASR for Well-Resourced and Low-Resourced Languages. Proceed- ings Of The 6th International Conference On Natural Language And Speech Processing (ICNLSP 2023) . pp. 266-273 (2023)
work page 2023
-
[8]
Bhogale, K., Sundaresan, S., Raman, A., Javed, T., Khapra, M. & Kumar, P. Vis- taar: Diverse Benchmarks and Training Sets for Indian Language ASR. ArXiv Preprint ArXiv:2305.15386. (2023)
arXiv 2023
Show all 17 references
-
[9]
Fine-Tuning ASR Models for Very Low-Resource Languages: A Study on Mvskoke
Mainzinger, J. Fine-Tuning ASR Models for Very Low-Resource Languages: A Study on Mvskoke. (University of Washington,2024)
2024
-
[10]
& Auli, M
Baevski, A., Zhou, Y., Mohamed, A. & Auli, M. wav2vec 2.0: A framework for self- supervised learning of speech representations. Advances In Neural Information Process- ing Systems . 33 pp. 12449-12460 (2020)
2020
-
[11]
& Scharenborg, O
Patel, T. & Scharenborg, O. Using cross-model learnings for the Gram Vaani ASR Challenge 2022.. INTERSPEECH. pp. 4880-4884 (2022)
2022
-
[12]
& Bapna, A
Conneau, A., Ma, M., Khanuja, S., Zhang, Y., Axelrod, V., Dalmia, S., Riesa, J., Rivera, C. & Bapna, A. FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech. (2022)
2022
-
[13]
& Weber, G
Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. & Weber, G. Common Voice: A Massively-Multilingual Speech Corpus. (2020) 13
2020
-
[14]
& Sarin, S
Sodimana, K., Pipatsrisawat, K., Ha, L., Jansche, M., Kjartansson, O., Silva, P. & Sarin, S. A Step-by-Step Process for Building TTS Voices Using Open Source Data and Framework for Bangla, Javanese, Khmer, Nepali, Sinhala, and Sundanese. Proc. The 6th Intl. Workshop On Spoken ...
2018 doi
-
[15]
& Joshi, B
Khadka, S., G.C., R., Paudel, P., Shah, R. & Joshi, B. Nepali Text-to-Speech Synthesis using Tacotron2 for Melspectrogram Generation. SIGUL 2023, 2nd Annual Meeting Of The Special Interest Group On Under-resourced Languages: A Satellite Workshop Of Interspeech 2023. (2023)
2023
-
[16]
& Others Torchaudio: Building blocks for audio and speech processing
Yang, Y., Hira, M., Ni, Z., Astafurov, A., Chen, C., Puhrsch, C., Pollack, D., Genzel, D., Greenberg, D., Yang, E. & Others Torchaudio: Building blocks for audio and speech processing. ICASSP 2022-2022 IEEE International Conference On Acoustics, Speech And Signal Processing (I...
2022
-
[17]
(GitHub Repository,2019), https://github.com/audacity/audacity 14
Audacity audacity. (GitHub Repository,2019), https://github.com/audacity/audacity 14
2019
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.