Methods to Increase the Amount of Data for Speech Recognition for Low Resource Languages

Alexan Ayrapetyan; Ara Yeroyan; Boris Ginsburg; Mher Yerznkanyan; Nikolay Karpov; Nune Tadevosyan; Sofia Kostandian; Vitaly Lavrukhin

REVIEW 4 major objections 4 minor 15 references

Methods to Increase the Amount of Data for Speech Recognition for Low Resource Languages

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Paid crowd-sourcing beats volunteers for low-resource ASR data

desk verdict Useful open resources for Armenian and Georgian ASR, but the central ablation table is too inconsistent to support the paper's headline cost-quality ranking as printed. read the letter →

arxiv 2501.14788 v2 pith:7Q4LM7ZE submitted 2025-01-08 cs.SD cs.CLeess.AS

Alexan Ayrapetyan , Sofia Kostandian , Ara Yeroyan , Mher Yerznkanyan , Nikolay Karpov , Nune Tadevosyan , Vitaly Lavrukhin , Boris Ginsburg This is my paper · ORCID

classification cs.SDcs.CLeess.AS

keywords low-resourcespeechrecognitiondataextensioncrowdsourcingpseudo-labelingArmenianGeorgianFastConformerworderrorrate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how to grow speech recognition training data for low-resource languages, using Armenian and Georgian as test cases. It compares four data sources—offline Common Voice events, paid crowd-sourcing, audiobooks, and pseudo-labeled YouTube audio—by cost and by how much each lowers word error rate. The central finding is that paid crowd-sourcing gives the best cost-quality balance, and that pseudo-labeled data helps more when the labeled seed set is larger. With all sources combined, the authors report 5.73% WER for Georgian and 9.9% for Armenian on a relatively small FastConformer model, outperforming much larger Whisper and Seamless baselines. If the result holds, the paper offers a practical and reproducible recipe for scaling other low-resource languages.

What carries the argument

The central mechanism is an additive ablation over data sources: the same FastConformer Hybrid Large model is trained on the base set and then on that base plus each data source alone and in combination, with WER, punctuation error rate, and probability-of-improvement (POI) statistics computed for every comparison. A cost table, expressed in engineer or scientist months and extra dollars, anchors each quality gain to a cost-quality ranking.

What would settle it

Record or collect a held-out Armenian and Georgian test set of spontaneous conversational speech from a domain not represented in Common Voice, FLEURS, the crowd corpus, or the YouTube selection, then compare the expanded model against the base-only model. If the WER gap shrinks to near zero or the data-source ordering changes on that set, the cost-quality ranking is an artifact of domain overlap rather than a general property.

Watch

Extended reading notes

Core claim

The paper's core claim is that for Armenian and Georgian, different data-extension methods can be ranked by cost and by their effect on ASR quality, and that paid crowd-sourcing is the best buy: it contributed roughly 70 hours of verified speech and produced the largest relative WER drops in the ablation, about 19% on the Common Voice test and 24% on the FLEURS test. Audiobooks are a valid source only when the training set is already diverse enough; pseudo-labeled YouTube audio becomes more valuable as the labeled seed grows. Combining all three additions with the Common Voice and FLEURS base, a FastConformer hybrid model reaches 5.73% WER for Georgian and 9.9% for Armenian, outperforming the much larger Whisper and Seamless baselines on the same test sets.

Load-bearing premise

The cost-quality ranking rests on WER measurements taken from the same domains used to filter and label the extra data; if the filtering model only selects audio it already transcribes well, the measured gains are inflated and the ranking may not transfer to new conditions.

Editorial extensions

If this is right

Adding roughly 70 hours of paid crowd-sourced speech gives the largest relative WER improvement of any single data source: about 19% on the Common Voice test and 24% on the FLEURS test.
Audiobooks hurt when added to a small base but improve WER once the labeled set is large enough, so they are a valid source when combined with enough other data.
Pseudo-labeled YouTube audio becomes more effective as the labeled seed grows: relative WER improvement on Common Voice rises from 4% to 26% as labeled data is added.
The published small FastConformer models surpass Whisper and Seamless baselines on the same test sets, reaching 5.73% WER for Georgian and 9.9% for Armenian.
The additive cost-quality comparison provides a transferable recipe for choosing data-extension strategies in other low-resource languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the authors leave implicit: the cost-quality ranking assumes WER gains measured on Common Voice and FLEURS generalize to out-of-domain audio; a held-out conversational test set would settle that.
Because the Armenian labeled set was expanded to roughly Georgian's size, the results suggest data-quantity parity, not just data quality, can close much of the quality gap between two low-resource languages.
A testable extension is to apply the same four-source pipeline to a third low-resource language and compare per-dollar WER improvement, to see whether paid crowdsourcing remains optimal at different data volumes.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The useful part is real. This paper opens datasets and models for Armenian and Georgian that didn't exist in this form before: a ten-fold expansion of Armenian Common Voice, a new audiobook corpus on OpenSLR, a crowd-sourced speech set from a modern text corpus, and 145 hours of pseudo-labeled YouTube audio. The trained FastConformer models reach 9.9% WER on Armenian and 5.73% on Georgian, beating the Whisper and Seamless baselines they compare against. The cost table is a genuinely practical addition. The experiments use standard WER metrics, remove train/test overlap, and include bootstrap POI significance tests.

The soft spot is the central ablation. Table III and the prose around it are internally inconsistent, and the paper's key takeaway—that paid crowdsourcing is the best cost/quality choice—is not recoverable from the printed evidence. In Section V-C, the 'supervised vs semi-supervised audiobook' comparison points to row 10, which is Whisper Large v3, not a semi-supervised audiobook run; no such row exists. The text then says adding audiobooks to 'row 7: MCV FLEURS Crowd + YouTube' gives 21% and 17% gains, but row 7 is actually MCV FLEURS Crowd + Audiobooks, and the actual numbers for the audiobook addition (rows 6 to 8) are 6.8% and 9.9% relative improvements. So the specific claims and the table don't line up, and the ranking of data sources is not actually demonstrated. The paid-vs-volunteer claim is also not a controlled comparison, because the volunteer MCV data is the baseline and the paid crowd data is an addition, not an equivalent-size replacement.

The selection-bias issue the reader raised is real but secondary. Using a model trained on MCV and FLEURS to validate crowd data and pseudo-label YouTube audio could inflate gains on those same test sets. It doesn't invalidate the released assets, but it does mean the measured WERs might not transfer to a truly new domain. An external test set would help.

Bottom line: this is a useful applied paper with solid released resources, but the central ablation needs to be fixed before the headline claim can be trusted. It deserves peer review with major revision. For anyone doing low-resource ASR, the datasets and models are worth citing; the empirical cost-quality ordering should be read cautiously until the table is corrected.

Referee Report

4 major / 4 minor

Summary. The paper reports on methods for expanding ASR training data for two low-resource languages, Armenian and Georgian. The authors collect data from Common Voice offline events, paid crowdsourcing (Toloka), audiobooks, and pseudo-labeled YouTube audio, and then train FastConformer hybrid models on the expanded corpora. They report strong results (5.73% WER for Georgian, 9.9% for Armenian) and open-source the models and several datasets. The abstract and conclusion assert that paid crowdsourcing offers the best cost-quality balance, outperforming volunteer crowdsourcing, audiobooks, and unlabeled data. The paper also presents an ablation study in Table III intended to support this ordering.

Significance. If the central claims hold, the paper makes a useful practical contribution: it demonstrates how to scale low-resource ASR data using a mix of inexpensive sources, and it releases open models and datasets (including an audiobook corpus, a text corpus, and crowd speech data) that provide strong baselines for Armenian and Georgian. The use of standard WER metrics, explicit train/test overlap removal, and bootstrap POI significance testing are good practices. The reported WERs (5.73% for Georgian, 9.9% for Armenian) with a relatively small architecture are genuinely valuable for the community. However, the headline cost-quality ranking is not currently backed by the ablation as printed, so the significance of the paper's central takeaway is not yet established.

major comments (4)

[§V-C and Table III] The ablation table and the prose in Section V-C are internally inconsistent, which makes the key quantitative claims unverifiable. Specifically: (a) the text says 'Using a supervised version of Audiobooks (III row 9)', but row 9 of Table III is labelled 'MCV FLEURS Crowd YouTube' and contains no audiobooks; (b) the text says 'Table III row 7: MCV FLEURS Crowd + YouTube', but row 7 is labelled 'MCV FLEURS Crowd Audiobooks'; (c) the prose reports a '21% decrease from 10.62 to 9.9' and a '17% decrease from 13.67 to 12.32', but the relative decreases between the corresponding rows (row 6 to row 8) are 6.8% and 9.9%, respectively. These errors prevent the reader from determining which condition actually produced the best result. The table and the surrounding text must be corrected and reconciled before the ablation can support the stated conclusions.
[§V-C] The claimed comparison between supervised and semi-supervised audiobook training is not present in Table III. The text says 'Using a supervised version of Audiobooks (III row 9) we get superior results ... than with semi-supervised (III row 10) setup', but row 10 is 'Whisper Large v3', not a semi-supervised audiobook condition, and no semi-supervised audiobook row exists anywhere in the table. This is a load-bearing comparison for the claim that audiobooks are a valid source and that supervised labels matter. The missing condition must be added to the table, or the passage must be rewritten to describe what was actually compared.
[§V and Conclusion] The central claim that paid crowdsourcing outperforms volunteer crowdsourcing, open-source audiobooks, and unlabeled data is not directly supported by the ablation design. The experiments only add each source incrementally to an MCV+FLEURS baseline (or to combinations of other sources), without matched-size or matched-cost controls. There is no isolated condition adding only volunteer crowdsourcing beyond the MCV baseline, and no isolated condition adding only audiobooks to the baseline. The audiobook effect is only measured on top of Crowd+YouTube (row 6 vs row 8), and the crowd effect is measured on top of MCV+FLEURS (row 3 vs row 5). These comparisons do not establish an ordering among data sources. Either new ablations that isolate each source should be run, or the abstract and conclusion should be revised to state the more modest incremental-improvement findings.
[§III-B, §III-D, and §V] The validation of crowd data and the pseudo-labeling of YouTube audio both rely on an ASR model trained on MCV and FLEURS, which are the same domains used for the final evaluation test sets. If this base model preferentially accepts or transcribes audio that resembles MCV/FLEURS, the reported WER gains from crowd data and pseudo-labeled YouTube data could be inflated, and the cost-quality ranking may not transfer to genuinely new domains. This is a legitimate correctness-risk concern. Please either add an evaluation on an out-of-domain test set (e.g., a held-out set from a different source or an existing unrelated benchmark for these languages) or discuss explicitly why the filtering does not bias the measured improvements.

minor comments (4)

[Abstract] The abstract contains a typo: 'Gergian' should be 'Georgian'.
[§III-B] The sentence 'The entire process took two months cost, and around $150, and one month of engineering work is required one month of engineering work' is ungrammatical and repeats 'one month of engineering work'; please rewrite for clarity. Also, Table I lists '1 month' for the Crowd corpus while the text says 'two months'; these should be reconciled.
[Table III] The table layout is confusing: there is a stray 'Audiobooks' label between rows 8 and 9, and rows 6, 7, and 9 have similar labels that are easy to conflate. Please reformat the table so that each row is uniquely and clearly labeled with its full training-data composition.
[§V] The POI significance testing is described in the text but no confidence intervals or POI values are reported in the table or elsewhere, except for one value in the introduction to Section V. Please include the bootstrap confidence intervals or a supplementary table of POI values for the key pairwise comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's headline claims are empirical comparisons on external test sets, not derivations from the data-extension inputs.

full rationale

The central claims—that paid crowdsourcing gives the best cost-quality balance and that the expanded datasets improve WER on MCV and FLEURS—are empirical results obtained by training FastConformer models on expanded corpora and measuring held-out test sets. No equation or construction in the paper makes the reported WER equal to the input data or to a fitted parameter. The crowd-data validation and YouTube pseudo-labeling use ASR models trained on the same MCV/FLEURS domains used for final evaluation, which is a legitimate external-validity and selection-bias concern, but it does not make the reported numbers equivalent to the filtering model's outputs by construction; adding such data can still hurt or help, and the WERs are not predetermined by the filtering decisions. Self-citations ([10] for the vac_aligner audiobook tool, [11] for the Golos-style crowd pipeline, [14] for the PER metric) are used as tooling or evaluation references, not as a uniqueness theorem or as the sole justification for the cost-quality ranking; vac_aligner is an open-source package that is independently runnable. The internal inconsistencies in Table III and its prose (e.g., row labels not matching the described conditions, and the reported 21% and 17% relative gains not matching the printed WERs) are correctness and reproducibility problems, not circularity: a mistaken ablation table does not make the argument self-referential. Therefore the derivation chain is not circular.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities; its contributions are datasets and empirical comparisons. The listed assumptions are domain assumptions about benchmark validity, pseudo-label accuracy, and the harmlessness of unvalidated data, none of which are derived results.

free parameters (3)

initial learning rate = 0.005-0.006
Training hyperparameter chosen for the FastConformer models (Section IV); not derived from data, standard cosine annealing schedule.
number of training epochs = 150-200
Training duration for the final models (Section IV); selected empirically, not derived.
global batch size = 1024
Batch size used across 8 GPUs (Section IV); configuration choice, not a fitted constant.

assumptions (4)

domain assumption MCV and FLEURS test sets provide unbiased estimates of ASR quality for Armenian and Georgian
All comparisons in Table II and Table III use these test sets as ground truth (Section V).
domain assumption Auto-validation with an ASR model trained on MCV and FLEURS yields acceptable quality labels for crowd data
Crowd-sourced data was automatically validated by a model trained on MCV and Fleurs (Section III-B), and the same benchmark domains are used for final evaluation.
domain assumption Pseudo-labeling of YouTube data with a base ASR model is a valid semi-supervised method that improves rather than reinforces errors
The IPL method from [13] is applied to 145 hours of unlabeled audio (Section III-D); no oracle check on pseudo-label accuracy is reported.
domain assumption Using unvalidated Common Voice (Other) data in training does not harm Georgian model quality
The Georgian training pool included 63.4 hours of non-validated data with no separate ablation (Section V).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Methods to Increase the Amount of Data for Speech Recognition for Low Resource Languages." pith.science (2026). https://pith.science/paper/7Q4LM7ZE

@misc{pith2026250114788,
  author       = {Pith},
  title        = {Pith review of: Methods to Increase the Amount of Data for Speech Recognition for Low Resource Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7Q4LM7ZE}},
  note         = {Machine review of arXiv:2501.14788}
}

read the original abstract

This study explores methods to increase data volume for low-resource languages using techniques such as crowdsourcing, pseudo-labeling, advanced data preprocessing and various permissive data sources such as audiobooks, Common Voice, YouTube. While these methods are well-explored for highresource languages, their application for low-resource languages remains underexplored. Using Armenian and Georgian as case studies, we demonstrate how linguistic and resource-specific characteristics influence the success of these methods. This work provides practical guidance for researchers to choose cost-effective and quality-driven dataset extension strategies for low-resource languages. The key takeaway from various data extension approaches is that paid crowd-sourcing offers the best balance between cost and quality, outperforming volunteer crowd-sourcing, open-source audiobooks, and unlabeled data usage. Ablation study shows that models trained on the expanded datasets outperform existing baselines and achieve 5.73% for Gergian and 9.9% for Armenian ASR word error rate using a relatively small FastConformer architecture. We open-sourced both the Armenian and Georgian models to allow further research and practical applications.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 12 canonical work pages

[1]

Fleurs: Few-shot learnin g evaluation of universal representations of speech,

A. Conneau, M. Ma, S. Khanuja, Y . Zhang, V . Axelrod, S. Dal mia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learnin g evaluation of universal representations of speech,” 2022 IEEE Spoken Language Technology W orkshop (SLT), pp. 798–805, 2022

work page 2022
[2]

Com- mon voice: A massively-multilingual speech corpus,

R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J . Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Com- mon voice: A massively-multilingual speech corpus,” arXiv preprint arXiv:1912.06670, 2019

arXiv 1912
[3]

Making more of little data: Improving low-resource automa tic speech recognition using data augmentation,

M. Bartelds, N. San, B. McDonnell, D. Jurafsky, and M. Wie ling, “Making more of little data: Improving low-resource automa tic speech recognition using data augmentation,” arXiv preprint arXiv:2305.10951 , 2023

arXiv 2023
[4]

Empowering Low-Resource Language ASR via Large-Scale Pseudo Labeling

K. S. Bhogale, D. Mehendale, N. Parasa, T. Javed, P . Kumar , M. M. Khapra et al. , “Empowering low-resource language asr via large-scale pseudo labeling,” arXiv preprint arXiv:2408.14026 , 2024

work page Pith review arXiv 2024
[5]

Dynamic acoustic unit augmentation with bp e-dropout for low-resource end-to-end speech recognition,

A. Laptev, A. Andrusenko, I. Podluzhny, A. Mitrofanov, I . Medennikov, and Y . Matveev, “Dynamic acoustic unit augmentation with bp e-dropout for low-resource end-to-end speech recognition,” Sensors, vol. 21, no. 9, p. 3063, 2021

work page 2021
[6]

Robust speech recognition for low-resou rce languages,

A. Romanenko, “Robust speech recognition for low-resou rce languages,” Ph.D. dissertation, Universität Ulm, 2022

work page 2022
[7]

Georgian speechrecognizer in famous searc hing systems and management of software package by voice commands in geor gian language,

I. Kardava, “Georgian speechrecognizer in famous searc hing systems and management of software package by voice commands in geor gian language,” in Conference Proceedings–Third International Conference on Advances in Computing, Electronics and Communication , vol. 10, 2016, pp. 978–1

work page 2016
[8]

Noise-robust speech recognition system for armenian language,

A. V ardanyan, “Noise-robust speech recognition system for armenian language,” Ph.D. dissertation, Master’s thesis, American University of Armenia, 2016

work page 2016

Show all 15 references

[9]

Armspeech: Armenian spoken langua ge corpus,

V . H. Baghdasaryan, “Armspeech: Armenian spoken langua ge corpus,” International Journal of Scientiﬁc Advances , vol. 3, no. 3, pp. 454–459, 2022. [Online]. Available: https://www.ijscia.com/wp-content/uploads/2022/06/Volume3-Issue3-May-Jun-No.283-454-459.pdf

2022
[10]

Enabling asr for low-resourc e lan- guages: A comprehensive dataset creation approach,

A. Y eroyan and N. Karpov, “Enabling asr for low-resourc e lan- guages: A comprehensive dataset creation approach,” arXiv preprint arXiv:2406.01446, 2024

2024 arXiv
[11]

Golos: Russian Dataset for Speech Research,

N. Karpov, A. Denisenko, and F. Minkin, “Golos: Russian Dataset for Speech Research,” in Proc. Interspeech 2021 , 2021, pp. 1419–1423

2021
[12]

Literary freedom: Project gutenberg,

B. Stroube, “Literary freedom: Project gutenberg,” XRDS: Crossroads, The ACM Magazine for Students , vol. 10, no. 1, pp. 3–3, 2003

2003
[13]

slimipl: Language-model-free iterative pseudo-labelin g,

T. Likhomanenko, Q. Xu, J. Kahn, G. Synnaeve, and R. Coll obert, “slimipl: Language-model-free iterative pseudo-labelin g,” arXiv preprint arXiv:2010.11524, 2020

2010 arXiv
[14]

Librispeech-pc: Benchmark for evaluation of punctuation and capitalization capabilities of end-to-end asr models,

A. Meister, M. Novikov, N. Karpov, E. Bakhturina, V . Lav rukhin, and B. Ginsburg, “Librispeech-pc: Benchmark for evaluation of punctuation and capitalization capabilities of end-to-end asr models, ” in 2023 IEEE Automatic Speech Recognition and Understanding W orkshop ( ASRU)...

2023
[15]

Bootstrap estimates for conﬁdenc e intervals in asr performance evaluation,

M. Bisani and H. Ney, “Bootstrap estimates for conﬁdenc e intervals in asr performance evaluation,” in 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing , vol. 1. IEEE, 2004, pp. I–409

2004

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Fleurs: Few-shot learnin g evaluation of universal representations of speech,

A. Conneau, M. Ma, S. Khanuja, Y . Zhang, V . Axelrod, S. Dal mia, J. Riesa, C. Rivera, and A. Bapna, “Fleurs: Few-shot learnin g evaluation of universal representations of speech,” 2022 IEEE Spoken Language Technology W orkshop (SLT), pp. 798–805, 2022

work page 2022

[2] [2]

Com- mon voice: A massively-multilingual speech corpus,

R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J . Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, “Com- mon voice: A massively-multilingual speech corpus,” arXiv preprint arXiv:1912.06670, 2019

arXiv 1912

[3] [3]

Making more of little data: Improving low-resource automa tic speech recognition using data augmentation,

M. Bartelds, N. San, B. McDonnell, D. Jurafsky, and M. Wie ling, “Making more of little data: Improving low-resource automa tic speech recognition using data augmentation,” arXiv preprint arXiv:2305.10951 , 2023

arXiv 2023

[4] [4]

Empowering Low-Resource Language ASR via Large-Scale Pseudo Labeling

K. S. Bhogale, D. Mehendale, N. Parasa, T. Javed, P . Kumar , M. M. Khapra et al. , “Empowering low-resource language asr via large-scale pseudo labeling,” arXiv preprint arXiv:2408.14026 , 2024

work page Pith review arXiv 2024

[5] [5]

Dynamic acoustic unit augmentation with bp e-dropout for low-resource end-to-end speech recognition,

A. Laptev, A. Andrusenko, I. Podluzhny, A. Mitrofanov, I . Medennikov, and Y . Matveev, “Dynamic acoustic unit augmentation with bp e-dropout for low-resource end-to-end speech recognition,” Sensors, vol. 21, no. 9, p. 3063, 2021

work page 2021

[6] [6]

Robust speech recognition for low-resou rce languages,

A. Romanenko, “Robust speech recognition for low-resou rce languages,” Ph.D. dissertation, Universität Ulm, 2022

work page 2022

[7] [7]

Georgian speechrecognizer in famous searc hing systems and management of software package by voice commands in geor gian language,

I. Kardava, “Georgian speechrecognizer in famous searc hing systems and management of software package by voice commands in geor gian language,” in Conference Proceedings–Third International Conference on Advances in Computing, Electronics and Communication , vol. 10, 2016, pp. 978–1

work page 2016

[8] [8]

Noise-robust speech recognition system for armenian language,

A. V ardanyan, “Noise-robust speech recognition system for armenian language,” Ph.D. dissertation, Master’s thesis, American University of Armenia, 2016

work page 2016