{"id":"4ed17c59-90d2-4a81-937f-edfc66d48734","arxiv_id":"2502.00421","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper releases Sagalee, a 100-hour, 283-speaker Oromo ASR dataset, and reports baseline WERs of 15.32% (Conformer AED), 18.74% (Conformer CTC), and 10.82% (Whisper Large-v3 fine-tuned).","lead":"Sagalee is a new 100-hour crowd-sourced speech dataset for Oromo, with recordings from 283 speakers and paired transcriptions. The authors report baseline ASR results, with fine-tuned Whisper reaching a 10.82% word error rate.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never states that train/dev/test splits are speaker-disjoint; with 283 speakers and roughly 192 utterances per speaker, random splits will likely overlap speakers, so the reported WERs may not measure speaker-independent ASR.","rationale":"The most load-bearing concern is the lack of an explicit speaker-disjoint data split. The central claim is not merely that a 100-hour Oromo dataset exists, but that it is suitable for ASR as demonstrated by the reported WERs. Those WERs are only meaningful as generalization estimates if the test set contains speakers, or at least utterances, not seen during training. The paper's Section V gives only hours per split, no speaker-level guarantee, and no description of how utterances were assigned. Since the average speaker contributes hundreds of utterances, an utterance-level random split would overlap speakers across splits. This is a concrete, checkable threat to correctness: it can make all three baselines look better than they are, and it is not resolved by the dataset being public because the user would need speaker IDs and the exact split logic to verify. The reader identified the same weakest assumption, and the conditional verdict remains appropriate. I considered transcription accuracy as an alternative concern, but even perfect transcriptions would not save a non-speaker-disjoint evaluation; the split issue is therefore the single most decisive gap. I give credit for the dataset's scale, speaker diversity, and public release, and for the use of standard toolkits and model configurations, which make the reproducibility check feasible. No formal verification or confidence intervals are present, but those are secondary to the split validity for the central claim.","tokens_in":7960,"tokens_out":3562,"duration_ms":38648,"concrete_test":"Download the public GitHub repository and inspect the split assignment and any speaker-ID metadata. Compute the overlap of speaker IDs among train, dev, and test. If any speaker appears in more than one split, or if speaker IDs are absent so the overlap cannot be checked, construct a speaker-disjoint split matched on dialect, gender, and duration, then retrain the Conformer AED and Whisper large-v3 systems with the paper's stated hyperparameters. If the test WER increases materially, the original baselines are optimistic and the paper must report the disjoint-split numbers; if the WER is essentially unchanged, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V defines the experiment split only by duration ('93.6hrs, 4.2hrs, and 2.4hrs') and never states that the train, dev, and test sets are speaker-disjoint. With 283 speakers and 53,573 total utterances (Table III), the average speaker contributes about 192 utterances, so an utterance-level random split would almost certainly place the same speaker in both training and test. If that is what was done, the Conformer AED WER of 15.32%, Conformer CTC WER of 18.74%, and Whisper large-v3 WER of 10.82% are not evaluations of speaker-independent generalization; they could be optimistically biased by the models memorizing speaker-specific acoustics, pronunciations, and reading styles. This directly affects the paper's central claim that Sagalee is a usable ASR dataset with credible baselines: even with perfectly accurate transcriptions, the reported numbers would not demonstrate how the models perform on new speakers. The paper also provides no stated transcription quality-control procedure, but the split issue is more load-bearing because it changes what the experiments establish regardless of transcription fidelity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Sagalee, a crowdsourced Oromo speech corpus of approximately 100 hours of read audio from 283 speakers, with 53,573 utterances and paired transcriptions, released publicly. It also reports baseline ASR experiments: a from-scratch Conformer with hybrid CTC-AED loss reaches 15.32% WER, a from-scratch Conformer with pure CTC reaches 18.74% WER, and fine-tuned Whisper large-v3 reaches 10.82% WER. The central claim is that Sagalee is a usable public ASR resource for the under-represented Oromo language and that these WERs establish credible baselines.","tokens_in":8199,"tokens_out":2494,"duration_ms":24627,"significance":"If the dataset is made available as stated and the evaluation protocol is sound, this is a valuable contribution to low-resource speech processing: Oromo has very few public speech resources, and a 100-hour, multi-speaker corpus with train/dev/test separation and baseline results would be directly useful to the community. The data-collection procedure is described concretely, the dataset is non-circular relative to the baselines, and the internal ranking of the three models (hybrid Conformer better than pure-CTC Conformer; fine-tuned Whisper best) is consistent with standard ASR experience. The main significance hinges on whether the reported numbers measure true speaker-independent generalization.","major_comments":[{"comment":"Section V states only that the dataset is \"split into train, dev and test sets with 93.6hrs, 4.2hrs, and 2.4hrs size respectively,\" with no statement that the split is speaker-disjoint. Given 283 speakers and an average of about 192 utterances per speaker (Table III), an utterance-level random split would very likely place the same speaker in both training and test sets. If that occurred, the reported WERs of 15.32%, 18.74%, and 10.82% would not measure speaker-independent ASR generalization; they could be optimistically biased by the models memorizing speaker-specific acoustics and reading styles. This directly affects the paper's central claim that Sagalee is usable for ASR with credible baselines. The authors should either explicitly state that the split is speaker-disjoint and specify how it was constructed, or, if it is not, re-split the data with no speaker overlap and rerun the experiments.","section":"V (Experiments), dataset split"},{"comment":"The paper does not describe any transcription quality-control procedure beyond allowing speakers to skip sentences with spelling errors or difficult text. Since the crowd-sourced transcriptions serve as ground truth for the reported WERs, the absence of an independent verification step, a post-hoc correction procedure, or an inter-annotator agreement measure is a load-bearing gap: inaccurate transcriptions would inflate WER even with a perfect model, and the paper's claim that the dataset is \"paired with transcriptions\" as a usable ASR resource depends on transcription fidelity. The authors should describe their quality-control pipeline or explicitly acknowledge this as a limitation with a concrete estimate of transcription error.","section":"IV.A.2 (Audio Recording) and V (Experiments)"}],"minor_comments":[{"comment":"The phrase \"digraphs knows as 'Qubee Dachaa'\" contains a typo; it should be \"known as.\"","section":"II.B.1 (Writing System)"},{"comment":"The vowel table uses the notation \"/5/\" for the open central vowel, which appears to be a placeholder or rendering error; it should be a proper IPA symbol such as /ɐ/ or /a/.","section":"Table I (Vowels)"},{"comment":"The description of fine-tuning Whisper says the convolutional layers at the front of the encoder were \"randomly initialized,\" but the reason for this choice is not explained; a sentence clarifying whether this was necessary for input feature compatibility or a deliberate regularization choice would improve reproducibility.","section":"V.B (Adaptation From Whisper)"},{"comment":"Table IV reports only WER percentages; reporting the number of utterances and speakers in the test set, as well as a confidence interval or significance test for the differences, would make the baselines more interpretable, especially since the test set appears to be only 2.4 hours.","section":"V.C (Result)"},{"comment":"The paper reports per-split durations but not the number of utterances or speakers in each split; these numbers should be added so readers can assess the evaluation's statistical power.","section":"IV.B (Data Statistics) and V (Experiments)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a dataset contribution with a clear and non-circular evaluation, but the speaker-disjoint split issue is central to whether the reported WERs support the claims. This is fixable within the manuscript's scope by either documenting the split method or re-running the experiments, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things. First, if the novelty claim holds up, this is the first public ASR corpus for Oromo, a language with tens of millions of speakers. That is a real contribution, and the dataset itself looks carefully built: 100 hours, 283 speakers, balanced gender, multiple dialect regions, and deliberately uncontrolled recording environments. Second, the reported WERs are probably somewhat optimistic, because the paper never says the data split is speaker-disjoint. It only gives durations (93.6/4.2/2.4 hours). With 283 speakers and about 192 utterances per speaker, a random utterance-level split would put the same speaker in train and test, and the models could memorize speaker characteristics. That would make the 15.32% / 18.74% / 10.82% numbers look better than they would on unseen speakers. This is the biggest flaw, and it is addressable: re-split by speaker, or at least report whether the current split is speaker-disjoint and give WERs under both conditions if needed.\n\nWhat the paper does well: the collection pipeline is described concretely (Android app, randomized sentences, skip button, compensation), the demographic and dialect distributions are shown, and the baseline experiments are internally consistent. BPE size tuning is reported honestly, and the finding that Whisper fine-tuning beats from-scratch training is unsurprising but useful. The dataset is public and reproducible, which is the core deliverable.\n\nThe softer spots: there is no stated transcription quality-control procedure. The skip button helps, but nothing verifies that the crowd-sourced transcriptions match the audio. Confidence intervals are absent, though for a dataset paper that is minor. Also, before publishing, the authors should check Common Voice and other open multilingual corpora for Oromo. If Oromo already appears there, the “first public ASR dataset” claim needs qualification. The citation pattern otherwise looks fine.\n\nFinal assessment: the dataset is the contribution, and it deserves to be reviewed. The split issue is load-bearing for the baseline numbers but not for the dataset's existence. A serious referee should ask for a speaker-disjoint split, more transparency on quality control, and a fuller novelty check. I would bring this to a reading group on low-resource speech, and I would cite it if I worked on Oromo or resources for under-served languages.","headline":"Sagalee is a genuinely useful public Oromo ASR corpus, but the baseline WERs may be optimistic because the train/dev/test split is never stated to be speaker-disjoint.","tokens_in":770,"tokens_out":873,"would_cite":true,"duration_ms":36092,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The authors introduce Sagalee, a 100-hour public speech corpus for Oromo built from 53,573 crowd-sourced utterances by 283 speakers, and report baseline word error rates of 15.32% (Conformer hybrid), 18.74% (Conformer CTC), and 10.82%…","keywords":["Sagalee","Oromo ASR","Afaan Oromoo","low-resource speech recognition","crowd-sourced speech corpus","Conformer","Whisper fine-tuning","word error rate"],"falsifier":"If a speaker-verification check of the released audio finds that the same voices appear in both the training and test portions of Sagalee, the reported 10.82% WER would not measure recognition of new speakers; likewise, a sample re-transcription showing high mismatch rates would undercut the claim that the transcriptions can serve as ground truth.","tokens_in":7784,"feed_emoji":"🎙️","tokens_out":5961,"duration_ms":50186,"temperature":0.7,"pith_summary":"The paper introduces Sagalee, a publicly released 100-hour speech corpus for Oromo (Afaan Oromoo), a language with over 45 million speakers that had no usable public ASR dataset. The authors collected 53,573 read utterances from 283 speakers through a crowd-sourcing mobile app, with transcriptions drawn from news, books, and the Oromia regional constitution. To show the corpus works for speech recognition, they trained a Conformer model from scratch and fine-tuned Whisper large-v3, reporting word error rates of 15.32%, 18.74%, and 10.82% respectively. The point is to give Oromo a public, real-world-condition dataset and baselines so that ASR research for the language can start from a shared foundation.","feed_headline":"100-hour Oromo speech dataset launches with 10.82% WER baseline","feed_subtitle":"Public Sagalee corpus gives Afaan Oromoo its first usable ASR benchmark, with Conformer and Whisper baselines.","key_machinery":"The load-bearing object is Sagalee itself: a crowd-sourced corpus built by (1) curating roughly 30,000 sentences from public texts, restricted to 4–25 words per sentence, (2) recording speakers through an Android app that includes a skip button for problematic sentences, and (3) logging speaker demographics such as gender, age, and dialect to capture phonetic and acoustic diversity. The baselines are produced with the WeNet toolkit, using a Conformer encoder-decoder with hybrid CTC-AED loss and a pure-CTC variant trained from scratch with a BPE unigram tokenizer, plus a fine-tuned Whisper large-v3 model.","core_discovery":"Sagalee is the first publicly available ASR dataset for Oromo that is suitable for training speech recognizers: it contains 100 hours of read speech from 283 speakers, recorded at 24 kHz in unconstrained acoustic environments, with 53,573 utterances and sentence-level transcriptions. The authors argue that previously released Oromo speech data—3 hours from 6 speakers and 17 hours from one male speaker, both intended for text-to-speech—cannot support ASR development. Their experiments establish that the dataset yields a reasonable baseline: a from-scratch Conformer with hybrid CTC-AED loss reaches 15.32% WER, a pure-CTC Conformer 18.74%, and fine-tuning Whisper large-v3 improves the WER to 10.82%.","pith_inferences":["One testable extension is to verify whether the train/test split is speaker-disjoint; the paper does not state that speakers are exclusive across the split, so the reported WERs could partly reflect speaker identity leakage if utterances from the same person appear in both sets.","Because transcriptions come from crowd workers reading prepared sentences, an accuracy check against the original texts or a manual re-transcription of a random sample would quantify transcription error and give users a quality bound, a measure the paper does not report.","The dialect metadata collected at registration could support dialect-conditioned ASR or dialect-identification studies, a direction the paper does not explore.","A future release reporting per-speaker and per-dialect WERs would help identify which speakers or dialect groups are hardest to recognize, which the current aggregate numbers do not reveal."],"forward_implications":["With Sagalee released on GitHub, Oromo ASR researchers can train and compare models on a common corpus instead of relying on private or TTS-only data.","The 10.82% WER from fine-tuned Whisper shows that large multilingual pre-training transfers to Oromo, giving a strong start for downstream applications like voice assistants and dictation.","The 15.32% WER from a from-scratch Conformer indicates that roughly 100 hours of real-world Oromo speech is enough to train a usable recognizer, though not yet production quality.","The reported baselines give future work concrete numbers to beat, enabling straightforward evaluation of new architectures, data augmentation, or transfer-learning methods."],"supporting_citations":[{"why":"Conformer architecture, the model trained from scratch for the two from-scratch baselines.","marker":"[3]"},{"why":"Whisper large-v3, the pre-trained multilingual model whose fine-tuning produces the 10.82% WER.","marker":"[4]"},{"why":"Prior Oromo acoustic modeling with only 64 command words, illustrating the gap Sagalee fills.","marker":"[21]"},{"why":"Earlier 22-hour Oromo read-speech corpus that is not publicly available, the closest existing resource before Sagalee.","marker":"[22]"},{"why":"WeNet toolkit, used for training the Conformer models and adapting Whisper.","marker":"[23]"},{"why":"BPE unigram subword tokenization, whose vocabulary size (nbpe=500) was tuned for the baselines.","marker":"[26]"}],"fun_headline_variants":["First open Oromo ASR dataset: 100 hours, 10.82% WER","Sagalee: 100-hour Oromo ASR corpus with 10.82% WER baseline","100 hours of Oromo speech: first public ASR dataset, 10.82% WER","Open-source Sagalee dataset gives Oromo ASR a 10.82% WER start","Sagalee: open 100-hr Oromo ASR set with 10.82% WER benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The crowd-sourced transcriptions are accurate enough to serve as ground truth, and the train/test split is speaker-disjoint, so the reported word error rates reflect generalization rather than memorization of individual speakers.","fun_headline_variants_meta":{"raw":{"variants":["First open Oromo ASR dataset: 100 hours, 10.82% WER","Sagalee: 100-hour Oromo ASR corpus with 10.82% WER baseline","100 hours of Oromo speech: first public ASR dataset, 10.82% WER","Open-source Sagalee dataset gives Oromo ASR a 10.82% WER start","Sagalee: open 100-hr Oromo ASR set with 10.82% WER benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00049,"raw_usage":{"total_tokens":2394,"prompt_tokens":915,"completion_tokens":1479,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":1348}},"tokens_in":531,"tokens_out":1479,"duration_ms":9856,"temperature":1.0,"reasoning_tokens":1348,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T19:04:41.049160+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a speaker-verification check of the released audio finds that the same voices appear in both the training and test portions of Sagalee, the reported 10.82% WER would not measure recognition of new speakers; likewise, a sample re-transcription showing high mismatch rates would undercut the claim that the transcriptions can serve as ground truth.","supporting_citations":[{"cited_title":"Afan oromo speech-based com- puter command and control: An evaluation with selected commands,","cited_arxiv_id":null,"evidence_quote":"Prior Oromo acoustic modeling with only 64 command words, illustrating the gap Sagalee fills."},{"cited_title":"Large vocabulary read speech corpora for four Ethiopian languages: Amharic, Tigrigna, Oromo, and Wolaytta,","cited_arxiv_id":null,"evidence_quote":"Earlier 22-hour Oromo read-speech corpus that is not publicly available, the closest existing resource before Sagalee."},{"cited_title":"Wenet: Production oriented streaming and non- streaming end-to-end speech recognition toolkit,","cited_arxiv_id":null,"evidence_quote":"WeNet toolkit, used for training the Conformer models and adapting Whisper."},{"cited_title":"Neural machine translation of rare words with subword units,","cited_arxiv_id":null,"evidence_quote":"BPE unigram subword tokenization, whose vocabulary size (nbpe=500) was tuned for the baselines."}],"review_version":1}