REVIEW 31 cited by
Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks. We also demonstrate that despite using a labeled training set 1/7-th the size of that used for the Whisper model, our model exhibits comparable or better performance on both in-domain and out-of-domain speech recognition tasks across many languages.
Forward citations
Cited by 31 Pith papers
-
OLMoASR: Open Models and Data for Training Robust Speech Recognition Models
An open 1M-hour English speech dataset plus Whisper-architecture models trained on it match Whisper's word error rates on short and long-form benchmarks.
-
Gemma 4 Technical Report
Gemma 4 open multimodal models (dense + MoE) with thinking mode, encoder-free 12B path, and KV/memory optimizations leap prior Gemma and rival larger open models on STEM, multimodal, long-context, and Arena benchmarks.
-
UniPASE: A Generative Model for Universal Speech Enhancement with High Fidelity and Low Hallucinations
UniPASE extends the PASE framework with DeWavLM-Omni to convert degraded speech into high-fidelity, low-hallucination audio across sampling rates via phonetic enhancement, acoustic adaptation, and multi-rate vocoding.
-
UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models
A single LLM can do ASR and zero-shot TTS on continuous speech features by switching between causal and bidirectional attention, reaching competitive but not state-of-the-art results.
-
CAM\~OES: A Comprehensive Automatic Speech Recognition Benchmark for European Portuguese
CAMOES is a new open benchmark and model collection for European Portuguese ASR, cutting word error rate by about 35% over the best zero-shot model.
-
Geolocation-Aware Robust Spoken Language Identification
Auxiliary geolocation prediction with injected conditioning signals improves dialect and accent robustness in SSL-based spoken language identification, reaching new SOTA on FLEURS and ML-SUPERB 2.0.
-
LCS-CTC: Leveraging Soft Alignments to Enhance Phonetic Transcription Robustness
LCS-CTC, a phoneme recognizer trained with similarity-aware LCS alignment masks constraining CTC, outperforms vanilla CTC on all reported PER, WPER, boundary-loss, and articulatory metrics.
-
Identifying Hearing Difficulty Moments in Conversational Audio
Prompted Gemini audio models detect hearing difficulty moments in conversation audio with F1 0.87, beating ASR hotword and Wav2Vec baselines.
-
OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder
OpenBEATs releases the BEATs audio pretraining pipeline, trains 300M-parameter models on 20k hours of multi-domain audio, and reports strong results across 25 datasets, including bioacoustics and reasoning tasks.
-
NIRANTAR: Continual Learning with New Languages and Domains on Real-world Speech Data
A real-world continual learning benchmark for multilingual ASR built from 3,250 hours of Indian language speech shows that no current CL method performs consistently across language- and domain-incremental scenarios.
-
A Self-Training Approach for Whisper to Enhance Long Dysarthric Speech Recognition
An iterative segmentation-based self-training method for Whisper improved long dysarthric speech recognition and achieved second place in both WER and SemScore at the SAP Challenge.
-
Data Quality Issues in Multilingual Speech Datasets: The Need for Sociolinguistic Awareness and Proactive Language Planning
A quality audit of Common Voice, FLEURS, and VoxPopuli finds serious micro- and macro-level data defects concentrated in less institutionalized languages, with a case study of Taiwanese Southern Min.
-
Early Attentive Sparsification Accelerates Neural Speech Transcription
Attention-based early audio-token sparsification at 40-60% sparsity accelerates Whisper ASR up to 1.6x with under 1% relative WER loss, across ten model variants, with no fine-tuning.
-
OWSM-Biasing: Contextualizing Open Whisper-Style Speech Models for Automatic Speech Recognition with Dynamic Vocabulary
Freezing OWSM v3.1 and adding dynamic-vocabulary biasing modules improves rare-word recognition and reduces real-time factor on LibriSpeech 100.
-
GigaAM: Efficient Self-Supervised Learner for Speech Recognition
GigaAM, a Russian ASR model family pretrained with CTC-teacher cluster targets, reports roughly 50% lower WER than Whisper-large-v3 on three Russian benchmarks.
-
NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction
NTPP models dual-channel spoken dialogue by predicting both speakers' next speech tokens as a pair, achieving speaker-independent full-duplex generation in a decoder-only transformer.
-
The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence
In 150k-hour speech-to-text training, a sub-exponential learning-rate warmup prevents divergence, while a faster warmup only speeds early convergence and does not improve the final model.
-
OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models
A new open suite of 13 multilingual speech models, up to 18B parameters, yields empirical scaling laws for ASR and speech translation performance.
-
A Semi-spontaneous Dutch Speech Dataset for Speech Enhancement and Speech Recognition
On real noisy Dutch semi-spontaneous speech, five of eight SOTA ASR models reach WER under 22%, yet five single-channel SE methods fail to improve and often degrade ASR.
-
MoLGE: Mixture of Language Group Experts for Efficient Scaling of Massively Multilingual Speech Recognition
Grouping 495 languages into roughly 16 clusters and routing speech to group-specific LoRA experts improves multilingual ASR error rates over dense and random baselines.
-
GigaAM Multilingual: Foundation Model for Underrepresented Languages
Cluster-balanced HuBERT-style pre-training on 2M hours plus domain-aware fine-tuning yields a compact encoder that outperforms larger open multilingual ASR models on Kazakh, Kyrgyz and Uzbek.
-
Towards Improved Speech Recognition through Optimized Synthetic Data Generation
A fine-tuned TTS with Whisper-based filtering generates synthetic Quebec French speech that trains ASR models well when combined with 10-60 hours of real audio, though a 13-14% WER gap to real-data training remains.
-
Efficient Multilingual ASR Finetuning via LoRA Language Experts
Monolingual LoRA language experts, combined by weighted merging (MoLE) or layer-wise knowledge distillation, improve Whisper-based multilingual ASR by about 10-15% relative WER over a plain multilingual LoRA baseline.
-
A Self-Refining Framework for Enhancing ASR Using TTS-Synthesized Data
Fine-tuning Whisper-large-v2 on 10,000 hours of synthesized Mandarin plus small real English/code-switching sets yields Twister, cutting mixed error rate by up to 56% on code-switching and 19% on Taiwanese Mandarin.
-
OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaning
OWSM v4 models, trained on a cleaned 166k-hour multilingual YODAS subset, beat prior open OWSM models and are competitive with Whisper and MMS on several benchmarks.
-
Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis
A Data2Vec2 speech encoder pre-trained on 300,000 hours of unlabeled Chinese dialect speech, connected to a small Qwen LLM via a linear projector and fine-tuned in four stages, sets a new state of the art on Chinese d...
-
DuRep: Dual-Mode Speech Representation Learning via ASR-Aware Distillation
A single speech encoder trained via ASR-aware distillation with variable attention masking performs competitively in both streaming and full-context modes at 200M and 2B scale.
-
VietASR: Achieving Industry-level Vietnamese ASR with 50-hour labeled data and Large-Scale Speech Pretraining
A 68M-parameter Vietnamese ASR model, pretrained on 70,000 hours of unlabeled audio and fine-tuned on 50 hours of labels, reports average WER 8.31, beating Whisper Large-v3 and commercial systems.
-
Selective Invocation for Multilingual ASR: A Cost-effective Approach Adapting to Speech Recognition Difficulty
A spoken large language model learns to judge its own transcription difficulty and routes only hard speech to a stronger ASR model, cutting cost and improving word error rate.
-
DONDO: Open w2v-BERT Speech-Recognition Base Models for African Languages
Open w2v-BERT ASR base models for 27 African languages, with a two-step annealing recipe and prefix-frame language conditioning.
-
Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach
Llama-SMoP-DEDR, a sparse mixture of projectors with modality-specific experts and routers, lowers word error rate for LLM-based AVSR on LRS3, mainly with smaller LLMs.
Discussion (0). Continue with ORCID to comment.