{"id":"c33ead5a-8357-4123-95a4-33a054131fbc","arxiv_id":"2507.13264","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Voxtral Mini and Small are open-weight speech-text language models that report top results on several speech recognition and translation benchmarks while preserving strong text performance.","lead":"Voxtral is a pair of open-weight AI models that listen to spoken audio and read text, then answer questions, transcribe, translate, and summarize. The authors report state-of-the-art speech recognition and translation for models of this size, and release the weights under an Apache 2.0 license.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MCV state-of-the-art claim in Section 4.1 depends on omitting Arabic from the macro-average; the paper's own Table 5 shows Scribe wins once Arabic is included.","rationale":"The reader's verdict (CONDITIONAL) is appropriate, and the reader did flag the Arabic exclusion in the rationale, but the reader's identified 'weakest assumption' focused on long-context chunking. I find the Arabic exclusion more load-bearing because it directly falsifies part of the strongest claim using the paper's own reported numbers, whereas the long-context concern is more speculative and affects a secondary capability claim. The English Short-Form SOTA claim holds under arithmetic verification, and the translation claim is plausible within the narrow compared set, so the paper retains substantial value. However, the MCV claim as stated is not supported and must be corrected (either by including Arabic and reporting Scribe as the winner, or by explicitly narrowing the claim). The verdict remains CONDITIONAL, with the condition that the MCV analysis be fixed and the 'state-of-the-art' wording be re-scoped to the compared systems.","tokens_in":13860,"tokens_out":7336,"duration_ms":80882,"concrete_test":"Recompute the MCV macro-average WER from Table 5 including the Arabic column for all models. If Scribe's nine-language mean (≈10.44) is below Voxtral Small's (≈11.91), the Section 4.1 claim that Voxtral Small beats all models on MCV is unsupported. Also re-run Figure 3 with and without Arabic to confirm the winner flips.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper claims in Section 4.1 that 'Voxtral Small achieves state-of-the-art transcription results on English Short-Form and MCV, beating all open and closed-source models.' For MCV, Figure 3 uses a macro-average over nine languages, but Table 5's caption excludes Arabic 'for fairness, since all models score in excess of 45%.' This exclusion is not neutral: it removes Voxtral Small's worst language (61.97% WER) while keeping Scribe's worst-but-better Arabic score (47.03%). Recomputing the macro-average from Table 5 with Arabic included gives Voxtral Small 11.91% and Scribe 10.44%, making Scribe the better model. Without Arabic, Voxtral Small wins by 5.65% vs 5.87%. Thus the 'state-of-the-art on MCV' claim is an artifact of a post-hoc selection rule, not a robust result. The stated criterion 'all models score in excess of 45%' would also apply to other hard languages, but only Arabic is dropped, and it is the language that flips the outcome. This is a concrete, internal inconsistency that directly undermines a headline SOTA claim, more so than the long-context chunking concern the reader highlighted.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Voxtral Mini and Voxtral Small, two Apache-2.0-licensed audio-text language models built from Mistral text backbones (Ministral 3B and Mistral Small 3.1), a Whisper large-v3 audio encoder, and a temporal downsampling adapter. Training consists of speech-text pretraining with two interleaving patterns signaled by special tokens, supervised finetuning on synthetic and real data, and DPO/Online DPO alignment. The paper reports word error rates, speech translation BLEU scores, speech understanding accuracy, text-only benchmark scores, and ablations on padding, adapter downsampling, pretraining pattern ratios, and alignment. It also contributes speech-synthesized versions of GSM8K, TriviaQA, and MMLU, plus an in-house speech understanding (SU) benchmark. The central claims are that Voxtral Small achieves state-of-the-art speech recognition on English Short-Form and MCV, state-of-the-art translation on all tested FLEURS pairs, and competitive speech understanding while preserving text performance.","tokens_in":14072,"tokens_out":5763,"duration_ms":69809,"significance":"If the headline results hold after the issues below are addressed, this is a useful open-weights system contribution: both model sizes and the three contributed benchmarks are released under permissive licenses, and the ablations on padding, frame-rate, and pretraining patterns are informative for practitioners. The paper is also honest about several limitations, such as the padding degradation and the regression of online DPO on ASR. However, the state-of-the-art claims are currently supported by a narrow comparison set and at least one aggregation choice that reverses the headline MCV result, and the long-context capability is asserted without a direct cross-chunk evaluation. These issues are fixable, but they need to be corrected before the claims can be accepted as stated.","major_comments":[{"comment":"The claim that Voxtral Small achieves state-of-the-art transcription on MCV depends on omitting Arabic from the macro-average. Table 5's caption states that Arabic is omitted 'for fairness, since all models score in excess of 45%', but adding Arabic back to the macro-average reverses the ranking: without Arabic, Voxtral Small is 5.65% WER versus Scribe's 5.87%, while with Arabic, Voxtral Small is 11.91% versus Scribe's 10.44%. Arabic is the only language meeting the stated 45% threshold, and it is also the language that flips the outcome, so the exclusion is not a neutral quality filter. Please report both the with-Arabic and without-Arabic macro-averages, or use a pre-specified aggregation rule, and adjust the state-of-the-art claim accordingly.","section":"Section 4.1, Figure 3, Table 5"},{"comment":"The statement that Voxtral Small achieves 'state-of-the-art translation scores in every source/target combination' is not supported by the baseline set. Table 7 compares only against GPT-4o mini Audio, Gemini 2.5 Flash, and Whisper large-v3 (restricted to X-to-English). There are no dedicated open-weights speech translation systems (for example, SeamlessM4T-family models) and no other open-weights speech LLMs, so 'state-of-the-art' is a claim about this specific comparison set rather than the field. BLEU scores are also reported as point estimates without confidence intervals or significance testing. Please broaden the comparison, or rephrase the claim as 'outperforms the compared closed models on this subset.'","section":"Section 4.2, Table 7"},{"comment":"The long-context capability is asserted for audio files up to 40 minutes, but the architecture processes 30-second chunks independently with positional encodings reset per chunk and the chunk embeddings concatenated afterward. This design provides no cross-chunk relative positional information to the decoder, so tasks that require integrating information from two distant chunks are not guaranteed to work. The reported evaluations do not directly test cross-chunk integration: the SU benchmark has audio up to 19 minutes, the English long-form earnings data are segmented into 10-minute clips, and the ASR/translation benchmarks are short-form. Please add at least one evaluation that requires combining information across chunk boundaries, or qualify the long-context and 40-minute claims to the current evidence.","section":"Section 2.1, Abstract, Section 1"},{"comment":"The in-house SU benchmark is used both for checkpoint selection and as a headline evaluation result. In Table 2, the SU benchmark scores are used to justify the released Voxtral Mini Online DPO checkpoint and to discuss the Voxtral Small SFT versus Online DPO choice, and the same benchmark is then reported in Figure 5 and Table 8 as evidence of speech understanding capability. Without a clear train/evaluation split or a statement that the reported SU numbers were computed on examples not used for any selection decision, there is a real risk of selection-on-test-set bias. Please disclose how the SU benchmark was partitioned, or separate the model-selection set from the final evaluation set.","section":"Section 5.4, Figure 5, Table 8"}],"minor_comments":[{"comment":"The text says 'FLUERS French' and Table 3's caption says 'SPGISPeech'; both should be corrected to 'FLEURS' and 'SPGISpeech'.","section":"Section 5.2"},{"comment":"Section 5.1 refers to 'Figure 8' when describing the padding experiment, but the padding plots are in Figure 7; the downsampling experiment in Section 5.2 correctly refers to Figure 8.","section":"Section 5.1"},{"comment":"The model name is typeset with an intervening space as 'V oxtral' throughout, which appears to be a rendering artifact; please use the intended 'Voxtral' consistently.","section":"Throughout"},{"comment":"WER and BLEU values are reported without confidence intervals or significance tests, although Table 2 shows that the internal evaluation uses repeated judging. Please state whether the ASR/translation differences are stable, or provide variance estimates.","section":"Tables 3-7"},{"comment":"The acronyms 'MMAU' and 'AU Bench' in Table 8 are not defined in the text or appendix; please define them and cite the sources.","section":"Section 3.4"},{"comment":"The speech-synthesized benchmarks are generated with the same lab's TTS system and filtered/rewritten with Mistral Large; please report the number of examples kept after each filtering step to let readers assess the difficulty and potential selection bias.","section":"Section 3.4, Appendix A.3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the released checkpoints and the three synthesized benchmarks are real contributions, and the paper is worth engaging with. But the headline 'state-of-the-art on MCV' claim does not survive including Arabic. Table 5 shows Voxtral Small's Arabic WER is 61.97 versus Scribe's 47.03; with Arabic in the macro-average, Scribe wins (10.44 vs 11.91). The paper's stated reason for dropping Arabic — 'all models score in excess of 45%' — does not justify dropping the language that flips the outcome. The exclusion is disclosed, which is better than hiding it, but the justification is post-hoc.\n\nWhat the paper does well: Apache-licensed weights and evals are shipped, so independent verification is feasible. The transcribe mode and <repeat>/<next> tokens are small but sensible additions. The ablations on padding, downsampling, and pretraining pattern ratios are clearly presented and useful for anyone building speech-text models. The synthesized evaluations for GSM8K, TriviaQA, and MMLU fill a real gap, though they inherit the usual TTS domain-shift caveat.\n\nSoft spots beyond the MCV issue: the SOTA translation claim rests on a narrow comparison set with no open-weights speech LLMs and no dedicated translation systems. The internal SU benchmark is used for both checkpoint selection and the headline DPO gains, which is mildly circular; the paper could disclose the selection rule or hold out a validation set. WER/BLEU tables lack confidence intervals, so it is hard to tell whether small differences are noise. The chunked encoder with per-chunk positional reset is a genuine limitation for cross-chunk reasoning, but the paper is upfront about it and the long-context claims are modest enough that it is secondary.\n\nBottom line: worth a serious referee. The MCV claim needs either a principled inclusion rule that keeps Arabic or a softened wording. The rest of the empirical work and the released artifacts stand on their own.","headline":"The open-weights release and benchmarks are genuinely useful, but the MCV state-of-the-art claim is an artifact of dropping Arabic, and the paper should fix or soften that claim before it is trusted.","tokens_in":15219,"tokens_out":3428,"would_cite":true,"duration_ms":36205,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An open-weight 24B audio model claims state-of-the-art transcription and translation across tested speech benchmarks.","keywords":["speech recognition","speech translation","multimodal language model","audio understanding","open-weights model","long-context audio","speech question answering","direct preference optimization"],"falsifier":"Build a QA set where the answer requires combining facts from two chunks separated by more than 30 seconds, and swap the chunk order to create a control: if accuracy is statistically unchanged or near chance after the swap while short-form WER stays low, the reset-position chunk design, not the decoder, is the ceiling on long-context reasoning.","tokens_in":13616,"feed_emoji":"🎙️","tokens_out":6948,"duration_ms":70478,"temperature":0.7,"pith_summary":"Voxtral Mini and Voxtral Small are two open-weight chat models trained to understand both spoken audio and text, released under an Apache 2.0 license. The paper's central claim is that the larger Small model, at 24B parameters, achieves state-of-the-art word error rates on English short-form and multilingual Common Voice transcription and beats all tested open and closed systems, and that it reaches the best BLEU scores on every FLEURS speech-translation pair tested. The authors argue this is possible without sacrificing text performance and while keeping the model small enough to run locally, with a 32K context window for audio up to 40 minutes. They also release three speech-synthesized benchmarks and an internal understanding benchmark for knowledge and trivia questions.","feed_headline":"Apache-licensed 24B model beats closed speech systems","feed_subtitle":"Voxtral Small posts state-of-the-art transcription and translation scores while staying small enough to run locally.","key_machinery":"The mechanism that carries the argument is the chunked audio front end: the encoder uses Whisper's fixed 30-second receptive field, processing each chunk independently with positional encodings reset per chunk, then concatenating the chunk embeddings and passing them through an MLP adapter that downsamples the frame rate 4x to 12.5 Hz before the language decoder autoregressively generates text. Two special tokens, <repeat> and <next>, disambiguate the two pretraining patterns, audio-to-text repetition and cross-modal continuation, whose balanced mixture is shown in ablations to be necessary for both transcription and understanding.","core_discovery":"On its own terms, the paper establishes that a 24B audio-text model built from a Whisper-style encoder, a 4x downsampling adapter, and a text decoder can surpass larger and closed-source comparison systems on speech recognition and translation: Voxtral Small posts the best English short-form and MCV word error rates, the best FLEURS translation score in every tested source/target pair, and competitive accuracy on speech question answering while retaining text-only benchmark performance. The smaller Mini model, when trained only for transcription, also beats the closed API transcribe systems on every recognition task tested.","pith_inferences":["My inference: the reset-per-chunk positional encoding puts a testable ceiling on cross-chunk reasoning; tasks whose answer depends on the relative order of two chunks more than 30 seconds apart should be fragile even if short-form ASR stays strong.","My inference: the synthesized benchmarks mix speaker diversity and TTS filtering with spoken-input difficulty, so their scores are not directly comparable to text MMLU numbers and should be read as a new capability axis rather than a substitute.","My inference: the observation that a text reward model fed only the transcription improves audio response quality suggests that preference alignment for speech can reuse text infrastructure, which would lower the cost of aligning other audio models.","My inference: the 6.25 Hz frame-rate penalty on FLEURS French suggests a floor on how far temporal downsampling can go before fine phonetic detail is lost, so scaling to longer context by more downsampling is unlikely to be free."],"forward_implications":["If the central claims hold, a 24B Apache-licensed model is a viable local drop-in for cloud transcription and translation services on the tested benchmarks.","The 32K context, at 12.5 Hz audio embeddings, makes single-pass handling of 40-minute audio practical for question answering and summarization rather than requiring segmentation and stitching.","The ablation results imply that training recipe choices, balanced repetition and continuation patterns and a 4x downsampling rate, are load-bearing for the balanced performance rather than incidental details.","Online DPO improves response-quality grades on the internal speech-understanding benchmark, but for Small it costs about 0.2 word error rate on English short-form, which is why the released Small checkpoint is the SFT version.","The released speech-synthesized GSM8K, TriviaQA, and MMLU subsets give the field a standard way to measure spoken knowledge and reasoning that did not previously exist."],"supporting_citations":[{"why":"Supplies the Whisper large-v3 audio encoder and its fixed 30-second receptive field that the chunking design builds on.","marker":"[Radford et al., 2023]"},{"why":"Provides the Transformer architecture underlying the encoder, adapter, and decoder stack.","marker":"[Vaswani et al., 2017]"},{"why":"Justifies the chunk-wise attention equivalence the encoder uses to handle long audio efficiently.","marker":"[Zhang et al., 2023]"},{"why":"Provides the 24B text decoder backbone that Voxtral Small extends to audio.","marker":"[Mistral AI Team, 2025]"},{"why":"Supplies the DPO objective used in preference alignment.","marker":"[Rafailov et al., 2024]"},{"why":"Supplies the online DPO variant that yielded the larger response-quality gains.","marker":"[Guo et al., 2024]"},{"why":"One of the prior works whose audio-text repetition and cross-modal continuation patterns the pretraining adapts.","marker":"[Nguyen et al., 2025]"},{"why":"The other prior source for the interleaved audio-text pretraining patterns and synthetic interleaved data.","marker":"[Zeng et al., 2024]"},{"why":"Provides the Llama QA benchmark and the speech-synthesized text-benchmark approach the paper extends.","marker":"[Nachmani et al., 2024]"},{"why":"Provides the Openbook QA benchmark that anchors the speech understanding comparisons.","marker":"[Chen et al., 2024]"}],"fun_headline_variants":["24B audio model beats closed rivals on speech tasks","Open-source Voxtral Small trumps proprietary speech systems","Speech AI that fits locally, beats big closed models","Small open speech model outperforms big proprietary ones","Apache-licensed 24B speech model sets new records"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything about long-form understanding rests on the assumption that concatenating independently encoded 30-second chunks, with positions reset per chunk, gives the decoder enough cross-chunk information to reason about the full audio.","fun_headline_variants_meta":{"raw":{"variants":["24B audio model beats closed rivals on speech tasks","Open-source Voxtral Small trumps proprietary speech systems","Speech AI that fits locally, beats big closed models","Small open speech model outperforms big proprietary ones","Apache-licensed 24B speech model sets new records"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000696,"raw_usage":{"total_tokens":3049,"prompt_tokens":750,"completion_tokens":2299,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":366,"completion_tokens_details":{"reasoning_tokens":2222}},"tokens_in":366,"tokens_out":2299,"duration_ms":19644,"temperature":1.0,"reasoning_tokens":2222,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:26:28.126407+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a QA set where the answer requires combining facts from two chunks separated by more than 30 seconds, and swap the chunk order to create a control: if accuracy is statistically unchanged or near chance after the swap while short-form WER stays low, the reset-position chunk design, not the decoder, is the ceiling on long-context reasoning.","supporting_citations":[],"review_version":1}