{"id":"618f2c27-1c94-4e6d-9d90-34376cabcdde","arxiv_id":"2501.14788","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A systematic comparison of crowdsourcing, audiobooks, pseudo-labeling, and volunteer recording for Armenian and Georgian ASR, with open datasets and models achieving 9.9% and 5.73% WER respectively.","lead":"This paper tests four ways to add speech data for Armenian and Georgian, from paying crowd workers to mining YouTube audio, and reports which approach gives the best accuracy for the money. It also publishes new open datasets and ASR models that beat larger commercial systems on these two languages.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central ablation table is internally inconsistent: prose cites wrong rows, the semi-supervised audiobook condition is missing, and claimed 21%/17% gains do not match the printed WERs, so the cost-quality ranking is unverifiable.","rationale":"The central claim is a ranking of data-extension strategies. The reader's weakest assumption identifies a plausible selection-bias mechanism in the crowd-validation and pseudo-labeling pipelines, and I agree that an independent held-out test set would strengthen the paper. But the more immediate problem is internal: the printed ablation table cannot currently be used to verify the ranking because of mismatched row references, a missing condition, and arithmetic that does not reproduce the prose's claimed gains. This is not a matter of disagreeing with the authors' choice of benchmarks; it is an internal inconsistency in the evidence. The paper's open-sourced models, datasets, and pipelines are concrete contributions that should be preserved, which is why I would not escalate the verdict beyond conditional acceptance. The required revisions are: rebuild Table III with unique condition IDs and correct labels; add the supervised/semi-supervised audiobook comparison that the text references; report relative improvements computed from the printed rows; and, if possible, evaluate the final models on an independently transcribed, out-of-domain test set to address the reader's selection-bias concern.","tokens_in":7711,"tokens_out":14912,"duration_ms":137841,"concrete_test":"Request the training/evaluation logs and rebuild Table III with unique condition IDs, including the supervised and semi-supervised audiobook runs referenced in Section V-C. Then recompute the relative WER improvements from the printed baselines and verify whether any pair yields the claimed 21%/17%. If the semi-supervised audiobook condition is absent, or the claimed gains do not match the printed WERs, the cost-quality ranking is not supported by the reported experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's key takeaway—that paid crowdsourcing offers the best cost/quality balance—rests on Table III and the surrounding prose. As printed, that evidence is not internally coherent. Section V-C says a supervised audiobook model (III row 9) beats a semi-supervised one (III row 10), but row 10 is Whisper Large v3 and no semi-supervised audiobook row exists anywhere in the table. The same section says audiobooks were added to 'Table III row 7: MCV FLEURS Crowd + YouTube', yet row 7 is labelled MCV FLEURS Crowd Audiobooks. The prose then reports a '21% decrease from 10.62 to 9.9' on MCV and '17% from 13.67 to 12.32' on FLEURS; from the table, these pairs appear in rows 6 and 8, and the actual relative reductions are 6.8% and 9.9%, not 21% and 17%. Additionally, there is no isolated condition adding only volunteer crowd-sourcing or only audiobooks to the baseline, so the claimed superiority over those sources is not directly tested. The reader's selection-bias concern is real, but it is an external-validity issue; before that, the internal ablation itself must be recoverable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports on methods for expanding ASR training data for two low-resource languages, Armenian and Georgian. The authors collect data from Common Voice offline events, paid crowdsourcing (Toloka), audiobooks, and pseudo-labeled YouTube audio, and then train FastConformer hybrid models on the expanded corpora. They report strong results (5.73% WER for Georgian, 9.9% for Armenian) and open-source the models and several datasets. The abstract and conclusion assert that paid crowdsourcing offers the best cost-quality balance, outperforming volunteer crowdsourcing, audiobooks, and unlabeled data. The paper also presents an ablation study in Table III intended to support this ordering.","tokens_in":7919,"tokens_out":5423,"duration_ms":45566,"significance":"If the central claims hold, the paper makes a useful practical contribution: it demonstrates how to scale low-resource ASR data using a mix of inexpensive sources, and it releases open models and datasets (including an audiobook corpus, a text corpus, and crowd speech data) that provide strong baselines for Armenian and Georgian. The use of standard WER metrics, explicit train/test overlap removal, and bootstrap POI significance testing are good practices. The reported WERs (5.73% for Georgian, 9.9% for Armenian) with a relatively small architecture are genuinely valuable for the community. However, the headline cost-quality ranking is not currently backed by the ablation as printed, so the significance of the paper's central takeaway is not yet established.","major_comments":[{"comment":"The ablation table and the prose in Section V-C are internally inconsistent, which makes the key quantitative claims unverifiable. Specifically: (a) the text says 'Using a supervised version of Audiobooks (III row 9)', but row 9 of Table III is labelled 'MCV FLEURS Crowd YouTube' and contains no audiobooks; (b) the text says 'Table III row 7: MCV FLEURS Crowd + YouTube', but row 7 is labelled 'MCV FLEURS Crowd Audiobooks'; (c) the prose reports a '21% decrease from 10.62 to 9.9' and a '17% decrease from 13.67 to 12.32', but the relative decreases between the corresponding rows (row 6 to row 8) are 6.8% and 9.9%, respectively. These errors prevent the reader from determining which condition actually produced the best result. The table and the surrounding text must be corrected and reconciled before the ablation can support the stated conclusions.","section":"§V-C and Table III"},{"comment":"The claimed comparison between supervised and semi-supervised audiobook training is not present in Table III. The text says 'Using a supervised version of Audiobooks (III row 9) we get superior results ... than with semi-supervised (III row 10) setup', but row 10 is 'Whisper Large v3', not a semi-supervised audiobook condition, and no semi-supervised audiobook row exists anywhere in the table. This is a load-bearing comparison for the claim that audiobooks are a valid source and that supervised labels matter. The missing condition must be added to the table, or the passage must be rewritten to describe what was actually compared.","section":"§V-C"},{"comment":"The central claim that paid crowdsourcing outperforms volunteer crowdsourcing, open-source audiobooks, and unlabeled data is not directly supported by the ablation design. The experiments only add each source incrementally to an MCV+FLEURS baseline (or to combinations of other sources), without matched-size or matched-cost controls. There is no isolated condition adding only volunteer crowdsourcing beyond the MCV baseline, and no isolated condition adding only audiobooks to the baseline. The audiobook effect is only measured on top of Crowd+YouTube (row 6 vs row 8), and the crowd effect is measured on top of MCV+FLEURS (row 3 vs row 5). These comparisons do not establish an ordering among data sources. Either new ablations that isolate each source should be run, or the abstract and conclusion should be revised to state the more modest incremental-improvement findings.","section":"§V and Conclusion"},{"comment":"The validation of crowd data and the pseudo-labeling of YouTube audio both rely on an ASR model trained on MCV and FLEURS, which are the same domains used for the final evaluation test sets. If this base model preferentially accepts or transcribes audio that resembles MCV/FLEURS, the reported WER gains from crowd data and pseudo-labeled YouTube data could be inflated, and the cost-quality ranking may not transfer to genuinely new domains. This is a legitimate correctness-risk concern. Please either add an evaluation on an out-of-domain test set (e.g., a held-out set from a different source or an existing unrelated benchmark for these languages) or discuss explicitly why the filtering does not bias the measured improvements.","section":"§III-B, §III-D, and §V"}],"minor_comments":[{"comment":"The abstract contains a typo: 'Gergian' should be 'Georgian'.","section":"Abstract"},{"comment":"The sentence 'The entire process took two months cost, and around $150, and one month of engineering work is required one month of engineering work' is ungrammatical and repeats 'one month of engineering work'; please rewrite for clarity. Also, Table I lists '1 month' for the Crowd corpus while the text says 'two months'; these should be reconciled.","section":"§III-B"},{"comment":"The table layout is confusing: there is a stray 'Audiobooks' label between rows 8 and 9, and rows 6, 7, and 9 have similar labels that are easy to conflate. Please reformat the table so that each row is uniquely and clearly labeled with its full training-data composition.","section":"Table III"},{"comment":"The POI significance testing is described in the text but no confidence intervals or POI values are reported in the table or elsewhere, except for one value in the introduction to Section V. Please include the bootstrap confidence intervals or a supplementary table of POI values for the key pairwise comparisons.","section":"§V"}],"recommendation":"major_revision","confidential_remarks":"The paper's open resources and strong WER results for Armenian and Georgian are valuable and likely outweigh the internal-inconsistency issues if the authors can fix them. The biggest concern is not the typos but the missing conditions in the ablation: the headline cost-quality ordering cannot be recovered from the current table, and the mismatched row references suggest that the table may have been edited independently of the prose. A careful revision with corrected numbers and, ideally, additional isolated ablations would improve the paper substantially. The selection-bias concern regarding same-domain validation is also worth raising in the revision, as it affects the external validity of the cost-quality claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful part is real. This paper opens datasets and models for Armenian and Georgian that didn't exist in this form before: a ten-fold expansion of Armenian Common Voice, a new audiobook corpus on OpenSLR, a crowd-sourced speech set from a modern text corpus, and 145 hours of pseudo-labeled YouTube audio. The trained FastConformer models reach 9.9% WER on Armenian and 5.73% on Georgian, beating the Whisper and Seamless baselines they compare against. The cost table is a genuinely practical addition. The experiments use standard WER metrics, remove train/test overlap, and include bootstrap POI significance tests.\n\nThe soft spot is the central ablation. Table III and the prose around it are internally inconsistent, and the paper's key takeaway—that paid crowdsourcing is the best cost/quality choice—is not recoverable from the printed evidence. In Section V-C, the 'supervised vs semi-supervised audiobook' comparison points to row 10, which is Whisper Large v3, not a semi-supervised audiobook run; no such row exists. The text then says adding audiobooks to 'row 7: MCV FLEURS Crowd + YouTube' gives 21% and 17% gains, but row 7 is actually MCV FLEURS Crowd + Audiobooks, and the actual numbers for the audiobook addition (rows 6 to 8) are 6.8% and 9.9% relative improvements. So the specific claims and the table don't line up, and the ranking of data sources is not actually demonstrated. The paid-vs-volunteer claim is also not a controlled comparison, because the volunteer MCV data is the baseline and the paid crowd data is an addition, not an equivalent-size replacement.\n\nThe selection-bias issue the reader raised is real but secondary. Using a model trained on MCV and FLEURS to validate crowd data and pseudo-label YouTube audio could inflate gains on those same test sets. It doesn't invalidate the released assets, but it does mean the measured WERs might not transfer to a truly new domain. An external test set would help.\n\nBottom line: this is a useful applied paper with solid released resources, but the central ablation needs to be fixed before the headline claim can be trusted. It deserves peer review with major revision. For anyone doing low-resource ASR, the datasets and models are worth citing; the empirical cost-quality ordering should be read cautiously until the table is corrected.","headline":"Useful open resources for Armenian and Georgian ASR, but the central ablation table is too inconsistent to support the paper's headline cost-quality ranking as printed.","tokens_in":8532,"tokens_out":4426,"would_cite":true,"duration_ms":37057,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Paid crowd-sourcing beats volunteers for low-resource ASR data","keywords":["low-resource speech recognition","data extension","crowdsourcing","pseudo-labeling","Armenian","Georgian","FastConformer","word error rate"],"falsifier":"Record or collect a held-out Armenian and Georgian test set of spontaneous conversational speech from a domain not represented in Common Voice, FLEURS, the crowd corpus, or the YouTube selection, then compare the expanded model against the base-only model. If the WER gap shrinks to near zero or the data-source ordering changes on that set, the cost-quality ranking is an artifact of domain overlap rather than a general property.","tokens_in":7482,"feed_emoji":"🎙️","tokens_out":6983,"duration_ms":62867,"temperature":0.7,"pith_summary":"This paper asks how to grow speech recognition training data for low-resource languages, using Armenian and Georgian as test cases. It compares four data sources—offline Common Voice events, paid crowd-sourcing, audiobooks, and pseudo-labeled YouTube audio—by cost and by how much each lowers word error rate. The central finding is that paid crowd-sourcing gives the best cost-quality balance, and that pseudo-labeled data helps more when the labeled seed set is larger. With all sources combined, the authors report 5.73% WER for Georgian and 9.9% for Armenian on a relatively small FastConformer model, outperforming much larger Whisper and Seamless baselines. If the result holds, the paper offers a practical and reproducible recipe for scaling other low-resource languages.","feed_headline":"Paid crowd-sourcing beats volunteers for low-resource ASR data","feed_subtitle":"Expanded models reach 5.73% WER in Georgian and 9.9% in Armenian, the paper reports.","key_machinery":"The central mechanism is an additive ablation over data sources: the same FastConformer Hybrid Large model is trained on the base set and then on that base plus each data source alone and in combination, with WER, punctuation error rate, and probability-of-improvement (POI) statistics computed for every comparison. A cost table, expressed in engineer or scientist months and extra dollars, anchors each quality gain to a cost-quality ranking.","core_discovery":"The paper's core claim is that for Armenian and Georgian, different data-extension methods can be ranked by cost and by their effect on ASR quality, and that paid crowd-sourcing is the best buy: it contributed roughly 70 hours of verified speech and produced the largest relative WER drops in the ablation, about 19% on the Common Voice test and 24% on the FLEURS test. Audiobooks are a valid source only when the training set is already diverse enough; pseudo-labeled YouTube audio becomes more valuable as the labeled seed grows. Combining all three additions with the Common Voice and FLEURS base, a FastConformer hybrid model reaches 5.73% WER for Georgian and 9.9% for Armenian, outperforming the much larger Whisper and Seamless baselines on the same test sets.","pith_inferences":["An implication the authors leave implicit: the cost-quality ranking assumes WER gains measured on Common Voice and FLEURS generalize to out-of-domain audio; a held-out conversational test set would settle that.","Because the Armenian labeled set was expanded to roughly Georgian's size, the results suggest data-quantity parity, not just data quality, can close much of the quality gap between two low-resource languages.","A testable extension is to apply the same four-source pipeline to a third low-resource language and compare per-dollar WER improvement, to see whether paid crowdsourcing remains optimal at different data volumes."],"forward_implications":["Adding roughly 70 hours of paid crowd-sourced speech gives the largest relative WER improvement of any single data source: about 19% on the Common Voice test and 24% on the FLEURS test.","Audiobooks hurt when added to a small base but improve WER once the labeled set is large enough, so they are a valid source when combined with enough other data.","Pseudo-labeled YouTube audio becomes more effective as the labeled seed grows: relative WER improvement on Common Voice rises from 4% to 26% as labeled data is added.","The published small FastConformer models surpass Whisper and Seamless baselines on the same test sets, reaching 5.73% WER for Georgian and 9.9% for Armenian.","The additive cost-quality comparison provides a transferable recipe for choosing data-extension strategies in other low-resource languages."],"supporting_citations":[{"why":"Supplies the FLEURS evaluation and training data used in every experiment.","marker":"[1]"},{"why":"Supplies the Common Voice base corpus and its test split, the main evaluation benchmark.","marker":"[2]"},{"why":"Supplies the alignment pipeline used to segment long audiobook audio into ASR-ready clips.","marker":"[10]"},{"why":"Supplies the start-validate-download crowd-sourcing method adapted for the paid crowd data.","marker":"[11]"},{"why":"Supplies the iterative pseudo-labeling algorithm used to exploit unlabeled YouTube audio.","marker":"[13]"},{"why":"Supplies the WER and punctuation error rate evaluation convention used for all models.","marker":"[14]"},{"why":"Supplies the bootstrap probability-of-improvement statistic used to assert that WER differences are significant.","marker":"[15]"}],"fun_headline_variants":["Paid crowd-sourcing best for low-resource ASR data, study finds","Low-resource ASR: paid crowdsourcing outperforms volunteers","Paid crowdsourcing: best data boost for low-resource ASR","Armenian and Georgian ASR: paid crowdsourcing wins on cost and quality","For low-resource ASR, paid crowdsourcing trumps volunteer efforts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cost-quality ranking rests on WER measurements taken from the same domains used to filter and label the extra data; if the filtering model only selects audio it already transcribes well, the measured gains are inflated and the ranking may not transfer to new conditions.","fun_headline_variants_meta":{"raw":{"variants":["Paid crowd-sourcing best for low-resource ASR data, study finds","Low-resource ASR: paid crowdsourcing outperforms volunteers","Paid crowdsourcing: best data boost for low-resource ASR","Armenian and Georgian ASR: paid crowdsourcing wins on cost and quality","For low-resource ASR, paid crowdsourcing trumps volunteer efforts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000897,"raw_usage":{"total_tokens":3850,"prompt_tokens":914,"completion_tokens":2936,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":2838}},"tokens_in":530,"tokens_out":2936,"duration_ms":22195,"temperature":1.0,"reasoning_tokens":2838,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:29:26.278660+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record or collect a held-out Armenian and Georgian test set of spontaneous conversational speech from a domain not represented in Common Voice, FLEURS, the crowd corpus, or the YouTube selection, then compare the expanded model against the base-only model. If the WER gap shrinks to near zero or the data-source ordering changes on that set, the cost-quality ranking is an artifact of domain overlap rather than a general property.","supporting_citations":[{"cited_title":"Fleurs: Few-shot learnin g evaluation of universal representations of speech,","cited_arxiv_id":null,"evidence_quote":"Supplies the FLEURS evaluation and training data used in every experiment."},{"cited_title":"Enabling ASR for Low-Resource Languages: A Comprehensive Dataset Creation Approach","cited_arxiv_id":"2406.01446","evidence_quote":"Supplies the alignment pipeline used to segment long audiobook audio into ASR-ready clips."},{"cited_title":"Golos: Russian Dataset for Speech Research,","cited_arxiv_id":null,"evidence_quote":"Supplies the start-validate-download crowd-sourcing method adapted for the paid crowd data."},{"cited_title":"Librispeech-pc: Benchmark for evaluation of punctuation and capitalization capabilities of end-to-end asr models,","cited_arxiv_id":null,"evidence_quote":"Supplies the WER and punctuation error rate evaluation convention used for all models."},{"cited_title":"Bootstrap estimates for confidenc e intervals in asr performance evaluation,","cited_arxiv_id":null,"evidence_quote":"Supplies the bootstrap probability-of-improvement statistic used to assert that WER differences are significant."}],"review_version":1}