Which one Performs Better? Wav2Vec or Whisper? Applying both in Badini Kurdish Speech to Text (BKSTT)

Hossein Hassani; Renas Adnan

REVIEW 4 major objections 4 minor 19 references

Which one Performs Better? Wav2Vec or Whisper? Applying both in Badini Kurdish Speech to Text (BKSTT)

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that Wav2Vec2-Large-XLSR-53, fine-tuned on roughly 15 hours of Badini Kurdish read speech, transcribes the dialect with 82.67% accuracy and 90.38% readability, clearly outperforming Whisper-small at 53.17% and 65.45%.

desk verdict Useful new Badini Kurdish corpus; the model comparison is under-specified and possibly confounded. read the letter →

arxiv 2508.09957 v1 pith:Z3HKUE6V submitted 2025-08-13 cs.CL

Renas Adnan , Hossein Hassani This is my paper · ORCID

classification cs.CL

keywords BadiniKurdishspeech-to-textWav2Vec2Whisperlow-resourcelanguagecorpusconstructiondialectASRfine-tuning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that Wav2Vec2-Large-XLSR-53, a self-supervised speech model, outperforms Whisper-small when both are fine-tuned for Badini Kurdish speech-to-text, a dialect that previously had no working STT system. To make the test possible, the authors built a new corpus from 78 children's stories in eight books, narrated by six speakers, yielding nearly 15 hours of audio split into 19,193 segments. On this corpus, Wav2Vec2 reaches 82.67% accuracy and 90.38% readability, while Whisper-small reaches 53.17% and 65.45%. If these numbers hold, the paper provides a first usable baseline for Badini STT and identifies which pretrained model is the better starting point for a low-resource dialect.

What carries the argument

The load-bearing machinery is the pair of pretrained models and the new corpus. Wav2Vec2-Large-XLSR-53 is a self-supervised speech encoder pre-trained on 53 languages, producing representations that are then fine-tuned on the Badini audio; Whisper-small is a transformer-based speech-recognition model trained on large-scale multilingual data with weak supervision. The corpus — 78 Badini children's stories, spoken by six narrators, cleaned and segmented into 19,193 segments containing 25,221 words — is what makes the comparison possible, because it gives both models the same target domain and evaluation data. The comparison itself is the argument: same data, same task, two models, and a report

What would settle it

Take the two fine-tuned models and evaluate them on a new held-out set of spontaneous Badini conversation recorded from adult speakers, none of whose voices appear in the 15-hour corpus. If Whisper-small matches or exceeds Wav2Vec2 on accuracy or readability there, the paper's conclusion that Wav2Vec2 'performs better' is reversed. A simpler check: release the exact train/validation/test splits and re-run the evaluation with test speakers completely excluded; the reported 82.67%-vs-53.17% gap should persist.

Watch

Extended reading notes

Core claim

The central claim is that Wav2Vec2-Large-XLSR-53 is the better model for Badini Kurdish speech recognition: fine-tuned on roughly 15 hours of read children's stories, it produces transcriptions that are both more accurate and more readable than Whisper-small under the same conditions. Specifically, the paper reports 82.67% accuracy and 90.38% readability for Wav2Vec2, versus 53.17% accuracy and 65.45% readability for Whisper-small. The authors interpret the gap as evidence that self-supervised speech representations can adapt to an under-resourced dialect more effectively than a large multitask model, and they frame the corpus itself as a contribution that closes a gap for Badini speakers, w

Load-bearing premise

The load-bearing premise is that roughly 15 hours of read children's stories from six narrators is representative enough of Badini speech to generalize the model ranking, and that the two models were fine-tuned and evaluated under genuinely equivalent conditions — including the same train/test split, hyperparameters, and tuning effort.

Editorial extensions

If this is right

If the ranking holds, Badini speakers get a functional speech-to-text baseline, opening the way for dictation, subtitles, and voice-controlled applications in the dialect.
The result suggests that for very low-resource dialects, a self-supervised speech encoder fine-tuned on a few dozen hours may beat a much larger multitask model trained for general speech recognition.
The corpus construction recipe — written children's stories, multiple narrators, segmentation into short clips — can be reused for other under-resourced dialects such as Hawrami, which the paper names as still lacking STT.
The reported readability gap (90.38% vs 65.45%) is larger than the accuracy gap, implying Whisper's errors are not just more frequent but also harder for a human to understand, which affects real-world usability.
Future work can test whether larger Whisper variants or newer models close the gap, and whether conversational Badini speech behaves like the read-story corpus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The comparison probably favors Wav2Vec2 more than a deployment would: read children's stories have clear articulation and limited vocabulary, so the gap on spontaneous adult conversation could be smaller or even reversed, especially for Whisper, which was trained on diverse audio.
The paper does not report whether the six narrators overlap between training and test sets; if the test set includes narrators already heard in training, the reported accuracy may overstate speaker-independent performance.
Readability is the more striking difference, but without a precise definition of the readability metric it is hard to know whether it measures human judgment, word-order preservation, or some automatic proxy.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's real asset is the corpus: 15 hours of Badini Kurdish read speech from six narrators, aligned to story transcripts. That is a genuinely useful resource for a dialect with essentially no STT support, and the authors are clear that they are filling a gap. The comparison of Wav2Vec2-Large-XLSR-53 and Whisper-small is a reasonable practical question. But the abstract is all we have, and it lacks the details that would make the comparison interpretable: no train/test split, no hyperparameters, no fine-tuning protocol, no definition of 'readability,' no error bars. More concerning, the 82.67% vs 53.17% gap could simply mean Whisper-small was evaluated differently—maybe zero-shot, maybe with less fine-tuning. That is not a minor caveat; the central claim is a superiority claim that only holds under matched training conditions. The stress-test note is right to flag it. I cannot confirm it from the garbled full text, but the burden is on the authors to show the two models were treated fairly. In a field where pretrained model comparisons are routine, this matters. The corpus itself is also limited to read children's stories, so generalizing to conversational Badini is an assumption. The authors say they chose stories for grammatical confidence and ready transcriptions, which is defensible, but the paper should say it outright. I would send this to review, not desk-reject it: the dataset is worth refereeing, and a patient reviewer can ask for the missing controls and a release of the corpus. If the code and data ship, this becomes a citable benchmark for a low-resource dialect. Recommended: accept with major revision after honest experimental details and a matched fine-tuning protocol.

Referee Report

4 major / 4 minor

Summary. This paper introduces a new Badini Kurdish speech corpus constructed from 78 children's stories from eight books, read by six narrators, with approximately 17 hours of audio reduced to nearly 15 hours after cleaning and segmentation (19,193 segments, 25,221 words). Using this corpus, the authors compare Wav2Vec2-Large-XLSR-53 and Whisper-small for speech-to-text. The abstract reports that Wav2Vec2 outperforms Whisper-small on readability (90.38% vs 65.45%) and accuracy (82.67% vs 53.17%), concluding that Wav2Vec2 performs better for Badini Kurdish STT. The paper's central claim is empirical, but the submitted text provides almost no experimental detail, and the body text is largely unreadable due to encoding corruption.

Significance. If the corpus is released and the comparison is properly controlled, this would be a valuable contribution to low-resource Kurdish dialect technology and a useful apples-to-apples comparison of two widely used pretrained models. The resource-creation effort (15 hours, 78 stories, six narrators) is a concrete strength. However, the significance cannot be assessed from the submitted version: the final performance numbers are not backed by any experimental protocol, and the manuscript body is garbled. The claim of Wav2Vec2 superiority depends entirely on a controlled comparison that the paper does not document.

major comments (4)

[Abstract / Experiments] The central comparative claim rests on two numbers, but the paper reports no training or evaluation protocol. Specify the data split (train/validation/test), fine-tuning hyperparameters (learning rate, batch size, epochs, optimizer, warmup), the exact fine-tuning data for each model, and whether both models were trained on the same Badini segments. If Whisper-small was evaluated zero-shot or with a different fine-tuning budget, the reported gap is not evidence of model superiority. This is load-bearing because the paper's conclusion is exactly that Wav2Vec2 performs better.
[Metric definitions] The term 'readability' is not defined. The abstract reports 90.38% and 65.45% readability and 82.67% and 53.17% accuracy, but no formula, annotation protocol, or scoring procedure is given. Clearly define both metrics (e.g., word/character error rate complement, token-level accuracy, human-rated readability) and state how each was computed.
[Statistical reliability] No error bars, confidence intervals, or significance tests are reported. With a single corpus and a single split, the reader cannot judge whether the 82.67% vs 53.17% gap is stable or an artifact of one split. Provide repeated runs, bootstrap intervals, or at least the test-set size and a statistical significance test.
[Full text / readability of submission] The body of the manuscript as provided is visually corrupted (mojibake/encoding errors), so the methodology, table, and results sections are not readable. A clean version is required for review. This is not a scientific flaw in the work itself, but it prevents verification of any claim beyond the abstract.

minor comments (4)

[Abstract] 'Bandin' appears to be a typo for 'Badini'; also 'SST' and 'STT' are used inconsistently.
[Data statistics] Numbers such as '19193 segments' and '25221 words' should be formatted with thousands separators for readability.
[References] Ensure full citations for Wav2Vec2-Large-XLSR-53 and Whisper-small are present; the reference list is not visible in the submitted version due to the encoding corruption.
[Scope / generalization] The corpus consists entirely of read children's stories; state explicitly that the reported performance may not generalize to conversational or spontaneous speech, which the introduction seems to motivate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports an empirical model comparison on a new dataset, with no derivation that reduces to its inputs.

full rationale

The paper's central claim is a direct empirical comparison of two externally pretrained speech-to-text models (Wav2Vec2-Large-XLSR-53 and Whisper-small) on a newly collected Badini Kurdish corpus. The reported readability and accuracy figures are measurements against reference transcriptions of the recorded stories, not quantities derived from the model definitions. There is no equation in which the output is defined in terms of the input, no parameter fitted to a subset and then renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem. The skeptical concerns—possible unequal fine-tuning conditions or absence of a held-out test split—are experimental-validity issues, not circularity. The full text is partially garbled, but the abstract's claim is an empirical result rather than a derivation. Under the rule that circularity requires a specific quotable reduction, none is present, so the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim is an empirical comparison, so no free parameters appear in the abstract. The axioms listed are the implicit domain assumptions about data representativeness, metric validity, and preprocessing fairness. There are no invented theoretical entities.

assumptions (3)

domain assumption The 78 children's stories and the speech of six narrators are representative of Badini Kurdish conversational speech.
The abstract describes using kids' stories for conversational coverage, but it does not justify that read stories from six narrators generalize to broader spoken Badini.
domain assumption The 'accuracy' and 'readability' metrics are well-defined and correctly computed.
The abstract reports numerical values for both metrics but does not define how they are calculated, so the reader cannot verify the measurement.
domain assumption The data cleaning, segmentation, and tokenization pipeline did not introduce systematic errors that favor one model.
The abstract describes preprocessing steps but gives no details on the rules or quality checks, leaving open the possibility of biases in the segments.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Which one Performs Better? Wav2Vec or Whisper? Applying both in Badini Kurdish Speech to Text (BKSTT)." pith.science (2026). https://pith.science/paper/Z3HKUE6V

@misc{pith2026250809957,
  author       = {Pith},
  title        = {Pith review of: Which one Performs Better? Wav2Vec or Whisper? Applying both in Badini Kurdish Speech to Text (BKSTT)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z3HKUE6V}},
  note         = {Machine review of arXiv:2508.09957}
}

read the original abstract

Speech-to-text (STT) systems have a wide range of applications. They are available in many languages, albeit at different quality levels. Although Kurdish is considered a less-resourced language from a processing perspective, SST is available for some of the Kurdish dialects, for instance, Sorani (Central Kurdish). However, that is not applied to other Kurdish dialects, Badini and Hawrami, for example. This research is an attempt to address this gap. Bandin, approximately, has two million speakers, and STT systems can help their community use mobile and computer-based technologies while giving their dialect more global visibility. We aim to create a language model based on Badini's speech and evaluate its performance. To cover a conversational aspect, have a proper confidence level of grammatical accuracy, and ready transcriptions, we chose Badini kids' stories, eight books including 78 stories, as the textual input. Six narrators narrated the books, which resulted in approximately 17 hours of recording. We cleaned, segmented, and tokenized the input. The preprocessing produced nearly 15 hours of speech, including 19193 segments and 25221 words. We used Wav2Vec2-Large-XLSR-53 and Whisper-small to develop the language models. The experiments indicate that the transcriptions process based on the Wav2Vec2-Large-XLSR-53 model provides a significantly more accurate and readable output than the Whisper-small model, with 90.38% and 65.45% readability, and 82.67% and 53.17% accuracy, respectively.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 18 canonical work pages

[1]

Abdalla, M. I. and Ali, H. S. (2010). Wavelet-based mel-frequency cepstral coefficients for speaker identification using hidden markov models. Journal of Telecommunications , 1(2):16--21

work page 2010
[2]

A., Veisi, H., and Rashid, T

Abdullah, A. A., Veisi, H., and Rashid, T. (2023). Breaking walls: Pioneering automatic speech recognition for central kurdish: End \- to \- end transformer paradigm. ArXiv preprint

work page 2023
[3]

Alharbi, S., Alrazgan, M., Alrashed, A., Alnomasi, T., Almojel, R., Alharbi, R., Alharbi, S., Alturki, S., Alshehri, F., and Almojil, M. (2021). Your paper title. Your Journal

work page 2021
[4]

N., and Sim, K

Das, D., Kumar, S., Li, B., Sainath, T. N., and Sim, K. C. (2021). Multi \- dialect speech recognition in english using attention on ensemble of experts. In Proc. Interspeech , pages 3061--3065

work page 2021
[6]

et al., B. (2024). Exploring adaptation techniques of large speech foundation models for low-resource languages. In Interspeech 2024 , pages 2524--2530

work page 2024
[7]

Ferraz, T. P. (2023). Efficient compression of multitask multilingual speech models. Master's thesis, University of São Paulo, October. Master's Thesis

work page 2023
[8]

Getman, Y., Moisio, A., Porjazovski, D., Rouhe, A., Virkkunen, A., AlGhezi, R., Lennes, M., Grósz, T., Lindén, K., and Kurimo, M. (2024). Your paper title. Your Journal

work page 2024
[9]

Gyulyustan, H., Hristov, H., Stavrev, S., and Enkov, S. (2024). Measuring and analysis of speech-to-text accuracy of some automatic speech recognition services in dynamic environment conditions. In AIP Conference Proceedings , volume 3063, page 030001. AIP Publishing

work page 2024

Show all 19 references

[10]

and Mohankumar, P

Haswani, V. and Mohankumar, P. (2022). Methods to optimize wav2vec with language model for automatic speech recognition in resource constrained environment. In Proceedings of the 19th International Conference on Natural Language Processing (ICON) , pages 149--153. Association ...

2022
[11]

Jain, R., Barcovschi, A., Yiwere, M., Corcoran, P., and Cucu, H. (2023). Adaptation of whisper models to child speech recognition. arXiv preprint arXiv:2307.09378

2023 arXiv
[12]

Li, Z., Graave, T., Liu, J., Lohrenz, T., Kunzmann, S., and Fingscheidt, T. (2024). Your paper title. Your Journal

2024
[13]

Lu, L., Liu, C., Li, J., and Gong, Y. (2020). Exploring transformers for large-scale speech recognition. arXiv preprint arXiv:2005.09684 . Microsoft

2020 arXiv
[14]

Ma, R., Qian, M., Gales, M. J. F., and Knill, K. M. (2023). Adapting an asr foundation model for spoken language assessment. INTERSPEECH 2023 Proceedings

2023
[15]

Orken, M., Dina, O., Keylan, A., Tolganay, T., and Mohamed, O. (2022). A study of transformer-based end-to-end speech recognition system for kazakh language. Scientific Reports , 12(8337):1--11

2022
[16]

Othman, M. (2022). Kurdistan region of iraq - population analysis report. Technical report, Ministry of Planning, Kurdistan Regional Government

2022
[17]

W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I

Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356

2022 arXiv
[18]

K., and Vats, N

Singh, G., Bhardwaj, A., Garg, B., Singh, M. K., and Vats, N. (2023). A comparative study of ai techniques for speech-to-text conversion. Journal of Emerging Technologies and Innovative Research (JETIR) , 10(11):e44--e48

2023
[19]

M., Tuytelaars, T., and Tolias, A

van de Ven, G. M., Tuytelaars, T., and Tolias, A. S. (2022). Three types of incremental learning. Nature Machine Intelligence , 4:1185--1197

2022
[20]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Abdalla, M. I. and Ali, H. S. (2010). Wavelet-based mel-frequency cepstral coefficients for speaker identification using hidden markov models. Journal of Telecommunications , 1(2):16--21

work page 2010

[2] [2]

A., Veisi, H., and Rashid, T

Abdullah, A. A., Veisi, H., and Rashid, T. (2023). Breaking walls: Pioneering automatic speech recognition for central kurdish: End \- to \- end transformer paradigm. ArXiv preprint

work page 2023

[3] [3]

Alharbi, S., Alrazgan, M., Alrashed, A., Alnomasi, T., Almojel, R., Alharbi, R., Alharbi, S., Alturki, S., Alshehri, F., and Almojil, M. (2021). Your paper title. Your Journal

work page 2021

[4] [4]

N., and Sim, K

Das, D., Kumar, S., Li, B., Sainath, T. N., and Sim, K. C. (2021). Multi \- dialect speech recognition in english using attention on ensemble of experts. In Proc. Interspeech , pages 3061--3065

work page 2021

[5] [6]

et al., B. (2024). Exploring adaptation techniques of large speech foundation models for low-resource languages. In Interspeech 2024 , pages 2524--2530

work page 2024

[6] [7]

Ferraz, T. P. (2023). Efficient compression of multitask multilingual speech models. Master's thesis, University of São Paulo, October. Master's Thesis

work page 2023

[7] [8]

Getman, Y., Moisio, A., Porjazovski, D., Rouhe, A., Virkkunen, A., AlGhezi, R., Lennes, M., Grósz, T., Lindén, K., and Kurimo, M. (2024). Your paper title. Your Journal

work page 2024

[8] [9]

Gyulyustan, H., Hristov, H., Stavrev, S., and Enkov, S. (2024). Measuring and analysis of speech-to-text accuracy of some automatic speech recognition services in dynamic environment conditions. In AIP Conference Proceedings , volume 3063, page 030001. AIP Publishing

work page 2024