{"id":"3cfb779b-4b63-45ae-8c96-452a954d75cf","arxiv_id":"2412.08274","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"2M-BELEBELE is a new multilingual speech and ASL comprehension benchmark built from BELEBELE and FLEURS, with human recordings for 74 spoken languages and ASL video with glosses.","lead":"This paper releases 2M-BELEBELE, a dataset of human speech recordings for reading comprehension questions in 74 languages plus American Sign Language video. It reports that adding speech lowers comprehension accuracy by about 2 to 3 percent compared to reading the same text.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FLEURS–BELEBELE passage alignment is unspecified and unvalidated; a wrong or partial concatenation would break the speech–text equivalence that the headline 2–3% gap depends on.","rationale":"The reader's weakest assumption correctly identifies the FLEURS–BELEBELE alignment as the load-bearing step. The central contribution is a dataset that enables direct comparison of reading comprehension and speech comprehension; that comparison is only valid if the two modalities present the same passage content. The paper does not specify the alignment algorithm, does not report per-language match rates, and does not provide any human or automatic validation that concatenated FLEURS audio matches the BELEBELE text. Without this, the headline result could be an artifact of mismatched or incomplete passages rather than a genuine modality effect. This concern is not a mere disagreement with the field consensus; it is a concrete correctness risk in the construction pipeline. The other issues noted by the reader—the 2–3% figure computed on 39 rather than 74 languages, and the unsupported ASL statements—are real but secondary; they affect the interpretation of the experiments, whereas the alignment issue affects the validity of the dataset itself. The paper does have independent strength in releasing a large human-recorded resource and in providing a falsifiable construction pipeline, but the missing validation of the alignment is precisely the kind of gap that can be closed with a focused audit. I therefore agree with the reader's conditional verdict: the dataset should be accepted only after the alignment is documented and validated, or the claims should be scaled back to the verified subset.","tokens_in":13494,"tokens_out":3194,"duration_ms":35878,"concrete_test":"Independently reconstruct the alignment for a stratified random sample of at least 50 passages across 10 languages spanning high- and low-resource settings. For each passage, take the aligned FLEURS audio segments and concatenate them; have a bilingual annotator (or a strong ASR model followed by human verification) compare the transcript against the BELEBELE text passage, checking that every sentence appears, in order, with no extras and no audio from another passage. Compute the passage-level semantic equivalence rate. Separately, inspect the alignment script or data keys to confirm that the mapping from BELEBELE passage IDs to FLORES-200 sentence IDs is exact and does not rely on fuzzy matching. If the equivalence rate is not close to 100%, re-run the headline experiments after correcting the mismatched passages and report whether the 2–3% gap changes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The speech half of 2M-BELEBELE is built by aligning BELEBELE passages (multi-sentence groups drawn from FLORES-200) with FLEURS sentence-level audio. Section 2 says this is done 'just by computing the alignment' but gives no algorithm, no identity key, and no validation that the concatenated FLEURS audio segments correspond exactly to the BELEBELE text passage. Because FLEURS does not contain every sentence of every passage (the paper reports only an approximate 80% passage match rate), passages may be partially matched, have sentences in the wrong order, or include audio from a neighboring FLORES-200 passage. The downstream evaluation then feeds the ASR transcript of this concatenated audio, along with the original BELEBELE questions and answers, to an LLM. If the speech passage is missing a sentence or contains a sentence from another passage, the multiple-choice question may become unanswerable in a way that has nothing to do with the speech modality, artificially lowering speech comprehension accuracy. The 2–3% gap reported in the abstract and Table 2 is therefore only interpretable if every speech passage used in the 39-language evaluation is semantically equivalent to its text counterpart. No such verification is reported, and the alignment code is not described. This is the most load-bearing assumption in the central claim, since the dataset's core value proposition is cross-modal comparability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces 2M-BELEBELE, a multilingual reading-comprehension benchmark extended to speech and sign language. For 74 spoken languages shared between BELEBELE and FLEURS, the authors align BELEBELE passages with FLEURS speech segments and commission human recordings for the unmatched parts, questions, and answers; for American Sign Language, they record 2,000 sentences with glosses by two interpreters. The evaluation uses Whisper and Seamless M4T cascaded with Llama-3 (and SpiRitLM for English) on 39 languages in 5-shot and zero-shot settings, reporting that speech comprehension accuracy is about 2–3% lower than text. An ablation in Appendix D shows that TTS-synthetic FLEURS data changes the ranking of ASR systems, motivating human recordings.","tokens_in":13718,"tokens_out":3447,"duration_ms":39483,"significance":"If the FLEURS–BELEBELE alignment is correct, the dataset is a valuable resource: it provides the first highly multilingual speech-comprehension benchmark at the intersection of two widely used datasets, extends FLEURS coverage, and adds an ASL component with glosses, which is genuinely novel. The synthetic-data ablation is a useful negative result that supports the authors' decision to use human recordings. The work is open-sourced, and the dataset enables cross-modal comparisons that were previously impossible for most of the 74 languages. The reported experiments are basic, as the authors acknowledge, but they suffice as reference results for a dataset paper.","major_comments":[{"comment":"The alignment between FLEURS speech segments and BELEBELE passages is the load-bearing step for the entire speech half of the dataset, but it is not described beyond 'just by computing the alignment.' No algorithm, identity key, or validation is given. Because FLEURS is sentence-level while BELEBELE passages are multi-sentence, and about 20% of passages are not matched, the concatenated audio could contain missing, duplicated, or out-of-order sentences, or even sentences from neighboring FLORES-200 passages. The headline 2–3% gap and all cross-modal comparisons are only interpretable if every speech passage is semantically equivalent to its text counterpart. Please specify the alignment procedure and report validation statistics, such as the fraction of passages with exact sentence-level matches, an automatic consistency check (e.g., transcript-to-text alignment), and a manual audit with examples.","section":"2 (FLEURS and BELEBELE passage alignment)"},{"comment":"The abstract and the Results paragraph state that 'speech decreases performance in about 2-3% average across languages,' but Table 2 shows that this holds only for the Seamless M4T + Llama-3 cascade (2.2-point drop in zero-shot, 3.2 in 5-shot). For Whisper + Llama-3, the drop is 7.8 points (5-shot) and 7.9 points (zero-shot). The claim is therefore not representative of the reported systems and should be qualified per ASR backbone, or the abstract should report the range, e.g., 2–8% depending on the system. As written, the abstract overstates the consistency of the cross-modal gap.","section":"3 (Results; Table 2)"}],"minor_comments":[{"comment":"The abstract says the evaluation is 'across languages' without noting that only 39 of the 74 spoken languages were used in the experiments; this subset is stated in §3 but should appear in the abstract or prominently in the results section to avoid overgeneralization.","section":"Abstract"},{"comment":"The paper positions 2M-BELEBELE as the 'first highly multilingual speech and sign comprehension dataset,' but the ASL portion is not evaluated with any model, and the only statement about ASL performance is that an open model 'did not perform above chance' (§3, ASL paragraph). The dataset contribution is clear, but the wording 'comprehension dataset' with no ASL results should be softened or clarified as dataset-only.","section":"1 (Introduction)"},{"comment":"The name '2M-BELEBELE' suggests two million items, but the statistics describe 488 passages, 2,000 utterances per language, 900 questions, and 3,600 answers per language; it would help to state explicitly what '2M' refers to (e.g., two modalities) in the introduction or statistics section.","section":"2 (2M-BELEBELE Statistics)"},{"comment":"The quality check of human recordings is described as 'randomly selecting 270 recordings (30% of sample size)'; this is a small absolute number relative to the total number of recorded sentences (on the order of 100k+). The authors do acknowledge in the limitations that not all recordings were verified, but the reporting of 30% is misleading because it is 30% of an unspecified sample; specify what the sample is.","section":"2 (Speech recordings)"},{"comment":"The SpiRitLM result is given only for English and only as an accuracy number in Table 2; the text mentions that 5-shot examples are in text while the passage is in speech as a possible cause, but no analysis of SpiRitLM's zero-shot behavior is provided. A brief discussion of why the direct model underperforms would help interpret the reference results.","section":"3 (Systems)"},{"comment":"The synthetic-data ablation uses WER on the ASR task, which is appropriate, but the definition of 'inversion' is informal ('a swap between two models that appear in adjacent positions on the list'). Clarify whether adjacent inversions are counted with respect to the human-data ranking for each language and whether ties are handled; this would make the last column of Table 4 reproducible.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The two major comments point to issues that are fixable within the scope of a revision: the alignment procedure can be documented and validated, and the accuracy claim can be made system-specific. The dataset itself and the synthetic-data ablation are solid contributions. The institutional overlap (Meta authors, Meta models, Meta-created dataset) is not a correctness problem, but an independent check of the alignment, or at least a detailed manual audit, would increase confidence in the headline numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a useful dataset paper with a somewhat over-stated headline number. The resource itself is the contribution: human-recorded speech for BELEBELE passages in 74 languages, plus ASL video with glosses, all open-sourced. That fills a real gap—there is no other highly multilingual speech comprehension benchmark of this size, and the ASL extension is genuinely new. The annotation pipeline is described in sensible detail (native speakers, quality checks, recording guidelines), and the synthetic-data ablation in Appendix D is a reasonable caution against TTS-generated evaluation sets, even if its scope is narrower than the text suggests.\n\nThe soft spots are mostly around the empirical claim. The abstract says speech comprehension is ~2-3% lower than reading, but that number comes from a 39-language subset (intersection of Whisper, Seamless M4T, and the dataset) and is presented as if it were general. More importantly, the speech-to-text alignment is under-documented. The paper says the BELEBELE passages are aligned with FLEURS 'just by computing the alignment', but no algorithm, identity key, or validation is reported. If the concatenated FLEURS audio does not exactly match the BELEBELE passage text, the multiple-choice questions can become unanswerable for reasons unrelated to speech, which would bias the 2-3% gap. Given that cross-modal comparability is the whole point of the dataset, this needs to be addressed in the final version—either by describing the alignment in detail, releasing the alignment code, and reporting a sample-based verification, or by scaling back the claim.\n\nThe ASL part is mostly a data release; the experiments there are thin (one open model at chance), and the paper says as much. That is fine for a resource paper, but the abstract's 'evaluate' overstates it. The synthetic-data 'proof' is also stronger than the evidence: it's a 23-language ASR-only study using one TTS system. I'd call that an interesting data point, not a proof.\n\nMy overall read: the dataset deserves to exist and will be cited; the paper needs a moderate revision to clarify the alignment and temper the abstract. I'd send it to review, not desk-reject. The authors have done the hard, useful work of building the resource; they just need to be more careful about what they claim on top of it.","headline":"A genuinely useful multilingual speech and ASL comprehension dataset, but the headline 2-3% gap rests on an under-validated alignment and the abstract over-generalizes.","tokens_in":14308,"tokens_out":3732,"would_cite":true,"duration_ms":45619,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"2M-BELEBELE extends the BELEBELE reading-comprehension benchmark with human speech in 74 spoken languages and American Sign Language video with glosses, and reports that listening comprehension trails reading by about 2–3 percentage…","keywords":["2M-BELEBELE","speech comprehension","reading comprehension","multilingual benchmark","American Sign Language","ASL glosses","FLEURS","BELEBELE"],"falsifier":"Take a random sample of the released dataset's aligned passage pairs and have bilingual annotators judge whether each audio sentence corresponds to the matching text sentence and whether the quiz's answer-relevant content is present in the audio. If a nontrivial share of samples (for example, more than a few percent) contains mismatched or dropped content, the spoken and written versions are not comparable and the 2–3% gap would not be a stable result. A cheaper proxy is to run a strong ASR system on the audio and compare its transcript to the BELEBELE text at the token level.","tokens_in":13289,"feed_emoji":"🧏","tokens_out":10035,"duration_ms":93976,"temperature":0.7,"pith_summary":"2M-BELEBELE is a new benchmark that puts listening and signing on the same footing as reading: it extends the BELEBELE reading-comprehension test with human speech recordings in 74 spoken languages and American Sign Language video with glosses, reusing the same 488 passages, 900 questions, and four-answer multiple-choice options in every language. The paper's main measurement is that a cascade of a speech recognizer plus a large text language model scores about 2–3 percentage points lower on the spoken passages than the text model scores on the written passages, in both zero-shot and 5-shot settings, with low-resource languages showing wider gaps. This matters because no prior dataset let researchers compare comprehension across text, speech, and sign on identical content at this language scale, and because the paper argues synthetic speech cannot safely stand in for human recordings when building such a benchmark. A by-product, 2M-FLORES, extends the FLEURS speech corpus by about 20 percent and adds an ASL component.","feed_headline":"Speech and ASL join a 75-language comprehension benchmark","feed_subtitle":"Human audio for 74 spoken languages plus signed ASL let the same questions be tested by reading, listening, or watching.","key_machinery":"The load-bearing mechanism is the passage-level alignment between FLEURS and BELEBELE. Because both resources descend from FLORES-200 sentences, the speech segments and text passages can be spliced into a common test without fresh annotation; unclear or uncovered passages are then filled by paid native speakers, and all questions and answer choices are recorded as well. The paper's Appendix D ablation—synthesizing a copy of FLEURS with neural text-to-speech and counting ranking inversions among ASR models—is the other mechanism: it supports the decision to use human recordings for evaluation rather than synthetic audio.","core_discovery":"The central claim is that a human-recorded multimodal comprehension benchmark of this scale is feasible and useful. The construction is direct: since BELEBELE's paragraphs are built from FLORES-200 sentences and FLEURS already contains human audio for many of those sentences, the paper aligns the two resources, matches roughly 80% of the paragraphs without extra annotation, and then commissions native-speaker recordings for the missing passages and for all questions and answers. For ASL, interpreters translated the English sentences, wrote glosses, and recorded video at 60 frames per second against plain backgrounds. On the evaluation side, cascading Whisper or Seamless M4T into a 70-billion-parameter text model yields speech comprehension about 2–3 points below text comprehension on average across 39 languages; an end-to-end speech-text model performs far worse. The paper presents the dataset itself, not the model results, as the contribution—future work can use it to separate perception errors from comprehension errors and to study signed-language comprehension.","pith_inferences":["If the alignment is validated, a natural next step is to vary the ASR front-end on the same 74-language passage set and directly measure how much of the 2–3% gap is perception error versus comprehension error.","Because FLEURS-style recordings are clean read speech, the 2–3% gap is probably a lower bound for current cascades; noisier or conversational audio would likely widen it.","The sentence-by-sentence ASL recording protocol likely understates discourse-level signing phenomena such as spatial placement; a paragraph-context evaluation could reveal how much comprehension depends on those phenomena.","Adding further sign languages at the same paragraph level would turn the sign-vs-text comprehension gap into a cross-signed-language question rather than a single English-ASL data point."],"forward_implications":["The same passages, questions, and answers exist in text, audio, and ASL, so accuracy can be decomposed by language and modality on identical content.","The presence of recorded questions and answers, not only passages, allows cross-modal settings such as spoken questions with written answers, a comparison the paper leaves for future work.","2M-FLORES adds human speech segments and an ASL subset to FLEURS, expanding the material available for ASR and language-identification evaluation.","The 2–3% speech-vs-text gap supplies a reference result for judging whether future speech-aware language models are closing the listening gap.","Paragraph-level ASL passages with glosses make it possible to train and evaluate context-aware sign-language translation, which sentence-level signing does not support."],"supporting_citations":[{"why":"Supplies the BELEBELE reading-comprehension benchmark—the parallel passages, questions, and answer options that 2M-BELEBELE extends to speech and sign.","marker":"Bandarkar et al., 2023"},{"why":"Provides FLEURS, the human speech corpus over FLORES-200 sentences used to align audio segments to BELEBELE passages.","marker":"Conneau et al., 2022"},{"why":"Establishes FLEURS-ASL, the prior sign-language extension of FLEURS that motivates and contextualizes the ASL portion of the new dataset.","marker":"Tanzer, 2024"},{"why":"Supplies Whisper, one of the two automatic speech recognition front-ends in the cascaded comprehension evaluation.","marker":"Radford et al., 2022"},{"why":"Supplies Seamless M4T, the other speech front-end used in the cascade, as well as evidence on multilingual speech translation.","marker":"Seamless-Communication et al., 2023"},{"why":"Defines the closest existing spoken question-answering evaluation, which the paper extends from English to 75 languages.","marker":"Nachmani et al., 2023"},{"why":"Supplies the MMS text-to-speech system used in Appendix D to test whether synthetic audio gives reliable benchmark rankings.","marker":"Pratap et al., 2024"},{"why":"Provides the open sign-language translation model that, in the paper's test, did not exceed chance on the ASL comprehension task, supporting the need for the glossed dataset.","marker":"Rust et al., 2024"},{"why":"Supports the claim that paragraph-level context improves sign-language translation, which motivates the paragraph-level organization of the ASL portion.","marker":"Sincan et al., 2023"}],"fun_headline_variants":["Human audio and sign for a 75-language comprehension test","Speech lags text by 2-3 points across 39 languages","First benchmark to read, listen, and watch for comprehension","2M-BELEBELE: 75 languages, including ASL, in one dataset"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatic alignment between FLEURS audio and BELEBELE text preserves each passage's content, so the audio and written versions are truly the same comprehension question; the paper does not describe or validate that alignment. If sentences are matched incorrectly, the reported 2–3% gap and the cross-modal comparison would be an artifact rather than a measure of listening comprehension.","fun_headline_variants_meta":{"raw":{"variants":["Human audio and sign for a 75-language comprehension test","Speech lags text by 2-3 points across 39 languages","First benchmark to read, listen, and watch for comprehension","2M-BELEBELE: 75 languages, including ASL, in one dataset"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000956,"raw_usage":{"total_tokens":4013,"prompt_tokens":821,"completion_tokens":3192,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":437,"completion_tokens_details":{"reasoning_tokens":3114}},"tokens_in":437,"tokens_out":3192,"duration_ms":23085,"temperature":1.0,"reasoning_tokens":3114,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:59:32.180656+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the released dataset's aligned passage pairs and have bilingual annotators judge whether each audio sentence corresponds to the matching text sentence and whether the quiz's answer-relevant content is present in the audio. If a nontrivial share of samples (for example, more than a few percent) contains mismatched or dropped content, the spoken and written versions are not comparable and the 2–3% gap would not be a stable result. A cheaper proxy is to run a strong ASR system on the audio and compare its transcript to the BELEBELE text at the token level.","supporting_citations":[],"review_version":1}