{"id":"c8a3aa57-3085-4b45-b58f-bcf5150bedde","arxiv_id":"2507.13977","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"New open-source FastConformer ASR models for Modern Standard Arabic and Classical Arabic claim state-of-the-art word error rates on MASC, MCV, and FLEURS, with diacritic-aware transcription for Quranic audio.","lead":"NVIDIA researchers built two Arabic speech recognition models, one for Modern Standard Arabic and one that also handles Classical Arabic, and report record-low error rates on several test sets. The models and the data processing pipeline are released openly, which could make accurate Arabic transcription widely available.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline MSA SOTA numbers are not reliable as stated: Tables III–IV select the model/configuration using the test sets, and the Section VI conclusion reports different WERs from Table V for the same model.","rationale":"I partially agree with the reader. Test-set selection is a genuine methodological flaw, but it is not automatically disqualifying for the qualitative SOTA claim because every pretrained candidate in Table III beats the cited prior SOTA; the most immediately disqualifying defect is that the paper's own conclusion and Table V give conflicting WERs for the MSA model, so the precise central claim is not internally consistent. The released checkpoints make it easy to settle both issues. I would not reject the paper on the evidence that the claim is false; I would make acceptance conditional on a corrected, validation-based evaluation and consistent reporting, with error bars or significance tests.","tokens_in":9443,"tokens_out":13326,"duration_ms":164819,"concrete_test":"Download the released MSA checkpoint and evaluate it on the official MASC, MCV, and FLEURS test splits using the paper's preprocessing; then repeat the Table III/IV selections using only held-out validation subsets and report final test WERs. If the validation-selected model's MCV or FLEURS WER exceeds 11.7% or 10.30%, the SOTA-over-MCV/FLEURS claim fails; if it does not, the paper should still be revised to reconcile the Section VI and Table V numbers before the claimed benchmark is treated as established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is a quantitative one: Table V reports 8.50% WER on MASC, 8.20% on MCV, and 5.01% on FLEURS for the best MSA model. For these numbers to be unbiased estimates of generalization, the test sets must be used only for final evaluation. That condition is violated: Section IV and Table III choose the starting checkpoint (Spanish) by comparing candidate checkpoints on the MASC, MCV, and FLEURS test sets, and Section V/Table IV compare data-processing configurations on the same test columns. The from-scratch control (17.53% on MCV, 12.38% on FLEURS) is worse than the cited SOTA (11.7%, 10.30%), so the SOTA claim depends on using a pretrained checkpoint. Because all three pretrained candidates beat SOTA, the qualitative claim may survive, but the exact margins are optimistically biased and cannot be verified from the paper. Independently, Section VI states the MSA model achieves 11.37% on MASC, 9.76% on MCV, and 7.73% on FLEURS, while Table V's 'Our MSA: PC & no Diacr' row gives 11.63%, 10.21%, and 8.18%; these are different numbers, so the paper does not contain a single authoritative statement of its headline result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents two Arabic automatic speech recognition models built on FastConformer: one for Modern Standard Arabic (MSA) and one unified model covering both MSA and Classical Arabic (CA) with punctuation and diacritics. The models are trained on open datasets (MASC, MCV17.0 Arabic, FLEURS Arabic, and EveryAyah), and the authors describe a data-preprocessing pipeline, report WERs on four evaluation sets, and claim state-of-the-art MSA performance and the first unified public MSA/CA diacritics model. The models and preprocessing code are released to support reproducibility.","tokens_in":9570,"tokens_out":4942,"duration_ms":55657,"significance":"If the reported results are valid, the contributions are practically significant: the paper provides open-source models with a commercial license, a documented preprocessing pipeline, and the first public unified MSA/CA model that outputs diacritics. The MSA model appears to offer large improvements over prior open Arabic ASR systems on MASC and competitive numbers on MCV and FLEURS. However, the central SOTA claim is currently not supported because model selection and data-processing decisions were made using the test sets, and the paper contains conflicting numbers for its headline result. The open-source release and the CA diacritics capability are valuable regardless of the exact SOTA margin.","major_comments":[{"comment":"The starting checkpoint is chosen by comparing WERs on the MASC, MCV, and FLEURS test sets, and Table V then reports the Spanish-checkpoint model as 'Our MSA: PC & no Diacr' with the same numbers. This means the headline WERs are the result of selection on the evaluation data, not unbiased estimates of generalization. Because the from-scratch model is substantially worse (17.53% on MCV and 12.38% on FLEURS, versus cited SOTA values of 11.7% and 10.30%), the SOTA claim depends on this selection. Please hold out a validation split for all configuration choices and report final test numbers once, or explicitly present Table III and Table IV as model-selection results and avoid calling the selected numbers 'SOTA'.","section":"Section IV, Table III"},{"comment":"The conclusion states that the MSA model achieves 11.37% WER on MASC, 9.76% on MCV, and 7.73% on FLEURS, but Table V lists 11.63%, 10.21%, and 8.18% for the same model. These are different numbers, so the paper does not contain a single authoritative statement of its headline result. Please correct the inconsistency and ensure that all sections report the same values.","section":"Section VI vs Table V"},{"comment":"The paper evaluates on 'MASC' sets extracted by the authors from raw data based on video IDs, whereas the cited SOTA of 21.8% [32] is reported on the official MASC test set. Unless the extracted sets exactly match the official MASC test split, the reported improvement on MASC is not directly comparable. Please clarify the relationship between the extracted sets and the official MASC dev/test splits, and if they differ, either provide comparable numbers on the official test set or refrain from a direct SOTA comparison.","section":"Section III.A, Table V"},{"comment":"The data-preprocessing configurations in Table IV are also compared on the MASC, MCV, and FLEURS test sets, and the final row selects the configuration that yields the best numbers. This is another instance of test-set-based selection, in addition to the checkpoint selection in Table III. The reported improvements from the 'O' preprocessing and the noisy-subset extraction are therefore optimistically biased and should not be treated as final evaluation results.","section":"Section V.A, Table IV"}],"minor_comments":[{"comment":"The abbreviation 'extr. nsy' in the last row of Table IV is not defined; please spell it out as 'extracted from noisy subset' or similar.","section":"Table IV"},{"comment":"The statement that 'Best results were obtained with learning rate lr = 5e-3' is given without an ablation or reference; since the learning rate is a free parameter, please provide supporting evidence or a citation.","section":"Section IV"},{"comment":"The data-filtering loop uses a model trained on clean MASC/MCV/FLEURS data to filter the noisy MASC subset; this is not circular with respect to test labels, but the paper should note that such filtering can propagate systematic transcription errors from the filtering model into the training data.","section":"Section V.A"},{"comment":"Reference [12] is cited as 'Ahmet et al.' in the text, but the reference itself lists the first author as 'A. Ahmed'; please correct the name for consistency.","section":"Section II.A"},{"comment":"The MediaSpeech column has a footnote about letter normalization applied in post-processing; please clarify whether the same normalization was applied to the prior systems for a fair comparison.","section":"Table V"}],"recommendation":"major_revision","confidential_remarks":"The test-set selection issue is serious: the headline SOTA numbers cannot be accepted as unbiased without a held-out re-evaluation or a clear separation between model selection and final evaluation. The internal inconsistency between Section VI and Table V also needs correction. The open-source release and the CA diacritics model are positive contributions that make the paper worth considering after these issues are addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hey,\n\nThe short version: this paper gives the community something genuinely useful—two open Arabic ASR checkpoints and a real preprocessing pipeline—but the headline SOTA numbers are not trustworthy because the test sets were used to pick the model configuration. That's not a minor quibble; it's the difference between 8.2% and 17.5% on MCV depending on whether you start from Spanish or from scratch.\n\nWhat's actually new: the first public unified MSA + Classical Arabic model with diacritic and punctuation output, a clean recipe for Arabic data cleaning, and a careful analysis of the available datasets. The CA result on EveryAyah—1.55% diacritic-aware WER after cleaning—looks like a real advance, not just a selection artifact, because it's compared against their own baseline trained without that data. The models are on HuggingFace, which makes the work immediately usable.\n\nWhere it falls down: Section IV and Table III compare starting checkpoints on the test sets and pick the winner. Table IV does the same for data-processing choices. That breaks the standard protocol for claiming SOTA. The from-scratch control is worse than the cited SOTA on MCV and FLEURS, so the claimed margins come from choosing a pretrained checkpoint after peeking. Also, the conclusion reports WERs (11.37/9.76/7.73) that don't match Table V's numbers (11.63/10.21/8.18) for the same model. The paper needs to explain that discrepancy. There are no error bars, which for differences of a few tenths of a percent matters.\n\nTo be fair, the qualitative conclusion—a Spanish-pretrained checkpoint beats English and multilingual options, and beats prior published results—probably survives, because all three pretrained options beat SOTA. But the exact numbers are optimistically biased and can't be verified from the paper.\n\nWho should read it: anyone working on Arabic ASR or looking for a strong open baseline. It's a useful resource, just not a reliable benchmark paper.\n\nMy recommendation: send it to peer review, but require a corrected evaluation protocol (validation-based checkpoint selection, locked test set, consistent numbers) before accepting. The work deserves referee time; the claims just need to be made honest.\n\nBest","headline":"The open Arabic ASR models and pipeline are a real resource, but the SOTA numbers are not trustworthy because the test sets were used to pick the model configuration.","tokens_in":10284,"tokens_out":2455,"would_cite":true,"duration_ms":27096,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One open data pipeline plus the FastConformer architecture yields an Arabic Modern Standard model with state-of-the-art word error rates and the first open unified model that transcribes Classical Arabic with diacritics.","keywords":["Arabic ASR","automatic speech recognition","Modern Standard Arabic","Classical Arabic","diacritics","punctuation","FastConformer","open-source models"],"falsifier":"Retrain the two models while holding out the MASC, MCV, FLEURS, and EveryAyah test splits until the very end, selecting the starting checkpoint and early stopping on a development split, then evaluate once on the held-out sets. If the final MCV WER does not beat the prior 11.7% or the FLEURS WER does not beat 10.30%, the claimed state-of-the-art generalization fails to reproduce; the paper's own from-scratch model (17.53% on MCV, 12.38% on FLEURS) shows how large this selection effect can be.","tokens_in":9079,"feed_emoji":"🎙️","tokens_out":13728,"duration_ms":129037,"temperature":0.7,"pith_summary":"The paper claims that a single universal pipeline for cleaning and normalizing Arabic audio-plus-text, paired with the FastConformer hybrid architecture, is enough to produce open Arabic speech-recognition models that beat prior published systems. Its Modern Standard Arabic model reports word error rates of 8.50% on MASC, 8.20% on Common Voice, and 5.01% on FLEURS, improving on the cited state of the art on all three benchmarks. Its unified MSA-and-Classical-Arabic model is, the authors say, the first open public model to output fully diacritized text for Classical Arabic, reaching a diacritic- and punctuation-aware WER of 6.65% on the overlap-free EveryAyah test split. Both models and the full training recipes are released openly. A reader should care because public Arabic ASR is scarce, Classical Arabic was essentially unserved by end-to-end models, and diacritics carry meaning that plain transcripts drop.","feed_headline":"One open model reads Classical Arabic with diacritics at 6.65% error","feed_subtitle":"One recipe sets MSA records on three benchmarks and, for the first time, covers Qur'anic audio with diacritics.","key_machinery":"The load-bearing mechanism is the pairing of the FastConformer hybrid RNN-T/CTC model, with a 1,024-token SentencePiece vocabulary over a 44-symbol diacritized Arabic alphabet, with the paper's data-processing pipeline. The pipeline filters by alphabet and duration, normalizes Eastern Arabic numerals and ligatures, decomposes positional letter forms, removes rare punctuation and Kasheeda, drops samples a pretrained model flags as high-error, and deduplicates texts that appear in both train and test. Its final step, self-filtering the noisy MASC subset to add 255 clean training hours, produces the largest single improvement in the results. The Spanish-pretrained starting checkpoint is a secondary mechanism: it converges faster than from-scratch training, with similar final accuracy.","core_discovery":"The central discovery is that a FastConformer hybrid RNN-T/CTC model, trained on four open corpora with a 44-symbol alphabet that includes six Hamza forms, six diacritics, and Arabic punctuation, can reach state-of-the-art word error rates on Modern Standard Arabic while also emitting fully diacritized text for Classical Arabic. The MSA model achieves 8.50% WER on MASC, 8.20% on MCV, and 5.01% on FLEURS, beating Whisper-Large-v3, SeamlessM4T, and the previously reported SOTA systems on these sets. Adding the EveryAyah corpus to the training mix yields a unified model whose diacritic- and punctuation-aware WER on the overlap-free EveryAyah test set is 6.65%, a result the paper attributes mainly to the cleaned data rather than to the architecture alone. The authors report that the data pipeline—filtering out-of-alphabet and noisy samples, normalizing numerals and ligatures, removing train/test overlap, and using a pretrained model to extract 255 additional clean hours from the noisy MASC subset—is what moves the combined-set WER from 9.95% to 8.50%.","pith_inferences":["The state-of-the-art numbers on MCV and FLEURS are likely optimistic: those test sets were used to pick the Spanish starting checkpoint, so the reported WERs contain selection signal; an honest held-out evaluation would probably land somewhere between the from-scratch numbers (17.53% on MCV, 12.38% on FLEURS) and the reported ones.","The noisy-subset extraction step is effectively a self-training loop, and Table IV shows every cleaning round improves all three benchmarks simultaneously; iterating this loop further is a natural extension that the paper does not explore.","The EveryAyah diacritic results depend on deduplicating train/test overlaps, and the large gap between the 1.55% full-set and 6.65% cleaned-set WERs suggests part of the gain is memorization; the 6.65% figure itself may not transfer to unseen reciters or non-Qur'anic Classical Arabic.","The Spanish-checkpoint warm start hints that the FastConformer family carries phonetically transferable features across languages; a systematic comparison of transfer versus from-scratch training for other low-resource languages would clarify when this recipe is safe to apply."],"forward_implications":["The MSA model outperforms Whisper-Large-v3 and SeamlessM4T on MASC, MCV, and FLEURS, showing that a well-tuned open-data recipe can beat far larger proprietary-trained systems on Arabic.","The unified model is the first open end-to-end ASR system that outputs diacritics for Classical Arabic, with a diacritic-aware WER of 6.65% on the overlap-free EveryAyah split, enabling Qur'anic and classical-text applications such as recitation aid and education.","Punctuation can be added to the output at nearly no cost to word accuracy (8.52% versus 8.50% on MASC), so readability improvements do not trade away recognition quality.","Adding diacritics to the training alphabet has a minor effect on MSA word accuracy, so a single model can serve both fully diacritized Classical Arabic and undiacritized Modern Standard Arabic.","The data-cleaning steps—alphabet filtering, normalization, overlap removal, and noise extraction—transfer to other languages and low-resource settings, since the gains come from data quality rather than architecture."],"supporting_citations":[{"why":"Supplies the FastConformer architecture used for both the MSA and unified models.","marker":"[38]"},{"why":"Provides the MASC corpus, the main training set and the benchmark whose prior SOTA was 21.8%.","marker":"[32]"},{"why":"Reports the previous best MCV WER (11.7%) that the MSA model claims to beat.","marker":"[17]"},{"why":"Reports the previous best FLEURS WER (10.30%) that the MSA model claims to beat.","marker":"[26]"},{"why":"Defines the Whisper-Large-v3 baseline compared in Table V.","marker":"[27]"},{"why":"Defines the SeamlessM4T baseline compared in Table V.","marker":"[46]"},{"why":"Implements the data preprocessing and filtering pipeline described in Section III.","marker":"[40]"},{"why":"Provides the EveryAyah Qur'anic dataset used for Classical Arabic training and the diacritic test split.","marker":"[39]"},{"why":"Supplies the Common Voice Arabic corpus used for training and the MCV test set.","marker":"[25]"},{"why":"Performs text normalization of numerals and orthography before tokenization.","marker":"[41]"}],"fun_headline_variants":["Open Arabic ASR: SOTA MSA, first Classical with diacritics","First open unified model for Modern and Classical Arabic","Arabic ASR model sets MSA records, adds Classical diacritics","FastConformer ASR: SOTA MSA and Classical, fully open"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported word error rates honestly measure generalization to unseen audio, even though the MASC, MCV, and FLEURS test sets were consulted during development to choose the starting checkpoint; if those sets had been reserved for final evaluation only, the new-benchmark claims would be weaker.","fun_headline_variants_meta":{"raw":{"variants":["Open Arabic ASR: SOTA MSA, first Classical with diacritics","First open unified model for Modern and Classical Arabic","Arabic ASR model sets MSA records, adds Classical diacritics","FastConformer ASR: SOTA MSA and Classical, fully open"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000288,"raw_usage":{"total_tokens":1704,"prompt_tokens":977,"completion_tokens":727,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":648}},"tokens_in":593,"tokens_out":727,"duration_ms":7912,"temperature":1.0,"reasoning_tokens":648,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:12:07.868948+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the two models while holding out the MASC, MCV, FLEURS, and EveryAyah test splits until the very end, selecting the starting checkpoint and early stopping on a development split, then evaluate once on the held-out sets. If the final MCV WER does not beat the prior 11.7% or the FLEURS WER does not beat 10.30%, the claimed state-of-the-art generalization fails to reproduce; the paper's own from-scratch model (17.53% on MCV, 12.38% on FLEURS) shows how large this selection effect can be.","supporting_citations":[{"cited_title":"Aswat: Arabic audio dataset for automatic speech recognition using speech- representation learning,","cited_arxiv_id":null,"evidence_quote":"Reports the previous best MCV WER (11.7%) that the MSA model claims to beat."},{"cited_title":"Masc: Massive arabic speech corpus,","cited_arxiv_id":null,"evidence_quote":"Provides the MASC corpus, the main training set and the benchmark whose prior SOTA was 21.8%."},{"cited_title":"Fast conformer with linearly scalable attention for efficient speech recognition,","cited_arxiv_id":null,"evidence_quote":"Supplies the FastConformer architecture used for both the MSA and unified models."},{"cited_title":"N-Shot Benchmarking of Whisper on Diverse Arabic Speech Recognition,","cited_arxiv_id":null,"evidence_quote":"Reports the previous best FLEURS WER (10.30%) that the MSA model claims to beat."},{"cited_title":"Seamlessm4t: Massively multilingual & multimodal machine transla- tion,","cited_arxiv_id":null,"evidence_quote":"Defines the SeamlessM4T baseline compared in Table V."},{"cited_title":"Speech data processor (sdp) toolkit,","cited_arxiv_id":null,"evidence_quote":"Implements the data preprocessing and filtering pipeline described in Section III."},{"cited_title":"Tarteel ai’s everyayah dataset,","cited_arxiv_id":null,"evidence_quote":"Provides the EveryAyah Qur'anic dataset used for Classical Arabic training and the diacritic test split."},{"cited_title":"Common voice: A massively- multilingual speech corpus,","cited_arxiv_id":null,"evidence_quote":"Supplies the Common Voice Arabic corpus used for training and the MCV test set."},{"cited_title":"NeMo (Inverse) Text Normalization: From Development to Production,","cited_arxiv_id":null,"evidence_quote":"Performs text normalization of numerals and orthography before tokenization."}],"review_version":1}