REVIEW 34 cited by
SeamlessM4T: Massively Multilingual & Multimodal Machine Translation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
What does it take to create the Babel Fish, a tool that can help individuals translate speech between any two languages? While recent breakthroughs in text-based models have pushed machine translation coverage beyond 200 languages, unified speech-to-speech translation models have yet to achieve similar strides. More specifically, conventional speech-to-speech translation systems rely on cascaded systems that perform translation progressively, putting high-performing unified systems out of reach. To address these gaps, we introduce SeamlessM4T, a single model that supports speech-to-speech translation, speech-to-text translation, text-to-speech translation, text-to-text translation, and automatic speech recognition for up to 100 languages. To build this, we used 1 million hours of open speech audio data to learn self-supervised speech representations with w2v-BERT 2.0. Subsequently, we created a multimodal corpus of automatically aligned speech translations. Filtered and combined with human-labeled and pseudo-labeled data, we developed the first multilingual system capable of translating from and into English for both speech and text. On FLEURS, SeamlessM4T sets a new standard for translations into multiple target languages, achieving an improvement of 20% BLEU over the previous SOTA in direct speech-to-text translation. Compared to strong cascaded models, SeamlessM4T improves the quality of into-English translation by 1.3 BLEU points in speech-to-text and by 2.6 ASR-BLEU points in speech-to-speech. Tested for robustness, our system performs better against background noises and speaker variations in speech-to-text tasks compared to the current SOTA model. Critically, we evaluated SeamlessM4T on gender bias and added toxicity to assess translation safety. Finally, all contributions in this work are open-sourced and accessible at https://github.com/facebookresearch/seamless_communication
Forward citations
Cited by 34 Pith papers
-
OLMoASR: Open Models and Data for Training Robust Speech Recognition Models
An open 1M-hour English speech dataset plus Whisper-architecture models trained on it match Whisper's word error rates on short and long-form benchmarks.
-
SimulS2ST-Omni: Data-Efficient Streaming Speech-to-Speech Translation via Explicit Trajectory Supervision
A joint text-code trajectory supervision recipe lets a two-stream speech LM achieve competitive long-form streaming S2ST with ~2k hours of paired speech.
-
When to Use Extra Context: Evidence-Grounded Terminology Adaptation for Simultaneous Speech Translation
Selectively passing evidence-grounded paper terminology into a streaming speech translator improves technical-term recall and translation quality more than injecting whole-document context.
-
Bridging the Stability-Expressivity Gap: Synthetic Data Scaling and Preference Alignment for Low-Resource Spoken Language Models
Synthetic data for low-resource spoken language models creates a Stability-Expressivity Gap that DGSA and TDSC self-alignment close, enabling SOTA Thai TTS and first Lao zero-shot voice cloning.
-
Frequency-Guided Deformable Networks for Continuous Phase Alignment
RFFT-derived periods guide deformable convolutions with Gaussian RBF interpolation and asymmetric routing to improve multi-task time-series modeling over rigid grids and bilinear sampling.
-
Cross-Attention is Half Explanation in Speech-to-Text Models
Cross-attention in speech-to-text models correlates with saliency-based explanations (Pearson r roughly 0.49-0.75 in the best aggregations) but explains only a minority of the variance, so it should complement, not re...
-
Geolocation-Aware Robust Spoken Language Identification
Auxiliary geolocation prediction with injected conditioning signals improves dialect and accent robustness in SSL-based spoken language identification, reaching new SOTA on FLEURS and ML-SUPERB 2.0.
-
Data Quality Issues in Multilingual Speech Datasets: The Need for Sociolinguistic Awareness and Proactive Language Planning
A quality audit of Common Voice, FLEURS, and VoxPopuli finds serious micro- and macro-level data defects concentrated in less institutionalized languages, with a case study of Taiwanese Southern Min.
-
FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models
A new benchmark shows state-of-the-art LLMs perform poorly on three Taiwanese indigenous languages across MT, ASR, and summarization.
-
Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasia
Across six ASR services, speakers with aphasia receive worse transcriptions than controls, and standard audit methods mask within-group disparities and hallucination risks.
-
Multimodal Zero-Shot Framework for Deepfake Hate Speech Detection in Low-Resource Languages
A contrastive audio-text framework detects hate speech in synthesized speech across six languages and outperforms baselines, with a new 127k-sample dataset.
-
Different Speech Translation Models Encode and Translate Speaker Gender Differently
Traditional encoder-decoder speech translation models encode speaker gender in hidden states, while newer adapter-based models largely do not; lower gender encoding tracks with masculine-default translation bias.
-
NAVER LABS Europe Submission to the Instruction-following Track
A separately-trained speech projector and text LoRA can be merged with only 1K steps of multimodal fine-tuning to yield a competitive ASR, ST, and spoken QA system.
-
CASPER: A Large Scale Spontaneous Speech Dataset
CASPER presents a 102-hour spontaneous English conversation dataset with per-speaker channels, speaker metadata, and baseline ASR and diarization results.
-
The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence
In 150k-hour speech-to-text training, a sub-exponential learning-rate warmup prevents divergence, while a faster warmup only speeds early convergence and does not improve the final model.
-
FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian
FAMA provides the first large-scale, fully open-source-licensed speech foundation models for English and Italian, with competitive accuracy and much higher speed than existing models.
-
Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation
A text-like 'unit language' mined from discrete speech units via n-gram modeling, plus task-prompt multi-task training, improves textless speech-to-speech translation to near text-supervised performance.
-
OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models
A new open suite of 13 multilingual speech models, up to 18B parameters, yields empirical scaling laws for ASR and speech translation performance.
-
Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders
MSRT-4B uses resource-aware routing between a frozen and a trainable Whisper encoder, plus a five-stage curriculum, reaching 83.3 average COMET over all 45×44 FLEURS translation directions with about 10 hours of paire...
-
X-Translator: A Real-Time Multilingual Speaker-Aware Speech-to-Speech Translation System
An open, modular cascaded system (streaming ASR + MT + prompt-conditioned TTS) preserves speaker identity in long-form multi-speaker translation, at higher latency and slightly lower translation quality than proprietary APIs.
-
Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Survey
The paper offers the first focused review of MLLM-based video translation organized by a three-role taxonomy of Semantic Reasoner, Expressive Performer, and Visual Synthesizer, plus open challenges.
-
COCO-Urdu: A Large-Scale Urdu Image-Caption Dataset with Multimodal Quality Estimation
A machine-translated, quality-filtered Urdu caption set covering 59,000 MS COCO images with 319,000 captions, presented as the largest public Urdu image-caption dataset.
-
WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learning
WeThink, a 120K-image QA dataset with AI-generated reasoning paths, combined with hybrid-reward reinforcement learning, improves a 7B vision-language model across 14 benchmarks.
-
Seamless Dysfluent Speech Text Alignment for Disordered Speech Analysis
Neural LCS uses learned phoneme and word similarity instead of exact matches to align dysfluent speech to intended text, and it outperforms DTW and Hard LCS on simulated benchmarks.
-
TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment
TESU-LLM shows that a frozen LLM can answer spoken queries after training only a 13M-parameter projector on text, using SeamlessM4T's shared speech-text encoder.
-
Continual Learning Beyond Experience Rehearsal and Full Model Surrogates
SPARC achieves strong continual learning accuracy with a fraction of the parameters of surrogate-based methods by combining task-specific depthwise filters with shared pointwise filters updated by exponential averaging.
-
Selective Invocation for Multilingual ASR: A Cost-effective Approach Adapting to Speech Recognition Difficulty
A spoken large language model learns to judge its own transcription difficulty and routes only hard speech to a stronger ASR model, cutting cost and improving word error rate.
-
CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages
A contrastive learning framework (CLaMP 3) aligns three music modalities with multilingual text, enabling text-to-music retrieval, cross-lingual retrieval for unseen languages, and emergent cross-modal retrieval.
-
VibeVoice-ASR-BitNet Technical Report
Heterogeneous quantization (INT8 tokenizer + 2-bit ternary LM) makes a 1.5B-parameter LLM-based ASR system run at real-time speed on CPUs with 2.9x compression and modest measured WER increases.
-
NAVER LABS System Re-implementation for the IWSLT 2026 Instruction-Following Task
Open re-implementation of the NAVER LABS three-stage speech-LLM pipeline for IWSLT 2026 yields COMET 0.781 EN–ZH ST and BERTScore-F1 0.346 English SQA, with 100k synthetic data released.
-
IIITH-BUT system for IWSLT 2025 low-resource Bhojpuri to Hindi speech translation
Fine-tuning SeamlessM4T on 20 hours of Bhojpuri-Hindi data with tuned hyperparameters and SpecAugment reaches 36.4 dev BLEU but only 9.9 test BLEU in the IWSLT 2025 low-resource task.
-
GMU Systems for the IWSLT 2025 Low-Resource Speech Translation Shared Task
Fine-tuning SeamlessM4T-v2 directly for end-to-end speech translation is competitive, and ASR-encoder initialization adds about 1 to 5 BLEU for languages unseen by the base model.
-
EZ-VC: Easy Zero-shot Any-to-Any Voice Conversion
EZ-VC combines discrete units from a multilingual self-supervised encoder (Xeus) with an F5-TTS flow-matching decoder to achieve zero-shot any-to-any voice conversion, without text labels or multiple disentangling encoders.
-
TransBench: Benchmarking Machine Translation for Industrial-Scale Applications
TransBench is a proposed e-commerce MT benchmark with a three-level evaluation framework and a fine-tuned quality-scoring model, but the paper contains no results and no released data or code.
Discussion (0). Sign in to comment.