Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

OpusLM: A Family of Open Unified Speech Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fully open family of speech language models, trained on public data alone and initialized from text LLMs, matches or beats prior systems on speech recognition, speech synthesis, and text-only benchmarks.

desk verdict Solid open SpeechLM family with valuable scaling and annealing findings; headline cross-system comparisons need a common harness. read the letter →

arxiv 2506.17611 v1 pith:S76DLTVV submitted 2025-06-21 cs.CL cs.SDeess.AS

Jinchuan Tian , William Chen , Yifan Peng , Jiatong Shi , Siddhant Arora , Shikhar Bharadwaj , Takashi Maekaku , Yusuke Shinohara
show 4 more authors
Keita Goto Xiang Yue Huck Yang Shinji Watanabe
This is my paper · ORCID
classification cs.CLcs.SDeess.AS
keywords speechlanguagemodelfoundationautomaticrecognitiontext-to-speechdelayinterleavearchitecturemulti-streamtokenizationopen-sourcereleasescaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a competitive speech language model can be built entirely from public materials. Its OpusLMs, initialized from pretrained text LLMs and pre-trained on 213K hours of open English speech-text pairs plus 292B text-only tokens, reach a 2.3% word error rate on LibriSpeech Test-Clean, a 4.0% WER in text-to-speech, and an MMLU score of 59.0 on the 7B model, while losing only 2.7 points of text capability relative to the base LLM. The paper's case is that the entire recipe — architecture, data, and training schedule — for a speech-text model that rivals larger or proprietary-data systems is now reproducible and auditable rather than hidden inside closed training pipelines. It also argues that the joint speech-text setup needs at least about 1.7B parameters to work, and that annealing on curated data helps but is sensitive to which data is chosen.

What carries the argument

The load-bearing mechanism is the delay interleave architecture operating on a multi-stream token grid. Each frame's $N=9$ tokens — $x_{t,1}$ semantic, $x_{t,2},\dots,x_{t,9}$ acoustic — are delayed by $n-1$ frames, the delayed embeddings are summed frame-wise, and one causal Transformer predicts all tokens of the next frame in parallel with trainable level biases $b_n$, which is what keeps the inference cost at $O(T)$ independent of the number of streams. Two further pieces carry the results: the joint vocabulary whose padding token has zero embedding, so that text-only sequences behave identically to the source text LLM and the pretrained text knowledge is inherited; and the two-stage schedule in which a long pre-training phase with 1M-frame batches, an in-domain text corpus, and target-region loss is followed by annealing with curated data and spliced long-form speech, which the paper shows turns one-to-two-minute recognition and synthesis from infeasible into usable.

What would settle it

Run the Table 1 baselines, including the strongest single-task ASR and TTS models and the multi-task speech LMs, through the exact OpusLM inference protocol — greedy decoding for ASR, top-30 sampling at temperature 0.7 for TTS, the same TTS scoring harness, no speaker-prompt or sample selection, and identical text normalization — and check whether the OpusLMs' 2.3% Test-Clean WER, 4.0% TTS WER, and 59.0 MMLU still beat them; if re-measured baselines close the gap, the 'comparable or superior' claim would not survive.

Watch

Extended reading notes

Core claim

OpusLMs are decoder-only Transformers that read and write a multi-stream token grid rather than a single token string: each 50Hz audio frame carries one semantic token and eight acoustic codec tokens ($N=9$), while text frames are padded to the same width with a zero-embedding pad. The central design, adopted from prior interleaving work, is the delay interleave architecture, in which token $x_{t,n}$ is moved $n-1$ frames later, the delayed embeddings in each frame are summed, and one causal Transformer predicts the next frame's tokens in parallel using per-stream bias vectors; this preserves intra-frame autoregression, keeps inference at $O(T)$ regardless of the stream count, and for text-only input reduces exactly to the behaviour of the source text LLM. Initialized from the SmolLM2 series (135M, 360M, and 1.7B) and from OLMo-2-7B, the models are trained on spliced task sequences for ASR, TTS, speech-only, and text-only with a joint vocabulary and task identifier tokens, then annealed on curated high-quality data spliced to up to two minutes of speech. The paper reports 2.3% WER on LibriSpeech Test-Clean (7B), 4.0% TTS WER (1.7B), and 59.0 MMLU (7B) with a 2.7-point degradation, and uses these to argue that fully open, public-data SpeechLMs are comparable or superior to prior systems. Its stated contribution is the articulation of the whole system design — tokenization, multi-stream modeling, and multi-stage training — rather than any single new component.

Load-bearing premise

The load-bearing premise is that the headline comparisons are fair: the baseline numbers are the ones each prior system reported under its own evaluation setup, and OpusLM was not re-measured against them in one shared harness, so the claimed margins could narrow if every system were scored identically.

Editorial extensions

If this is right

  • Future SpeechLM work can start from fully released weights, data, code, and training logs, because the paper shows public resources are enough to reach the reported performance.
  • One model can serve ASR, TTS, and text tasks with text skill largely intact, so applications do not need separate speech and text stacks.
  • The 135M and 360M variants collapse on all three tasks in this joint setup, making scale past roughly 1.7B a precondition for the unified paradigm rather than an optimization detail.
  • Annealing with curated data improves out-of-domain performance and makes long-form speech tractable, so the annealing practice from text LLM development carries over to speech pre-training.
  • Because text-only input collapses to the base text LLM's behavior, the trained models can be dropped into existing text LLM inference pipelines as replacements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension: if the pretrained text backbone carries much of the gain, initializing from a weaker public base LLM with the same speech data should shrink the ASR and TTS advantages proportionally.
  • Since inference cost is $O(T)$ regardless of stream count $N$, a wider codec with more acoustic tokens per frame could be plugged in at the same inference cost, offering a direct test of whether synthesis quality improves with stream width — a knob this paper does not turn.
  • The two annealing trials differ in data composition but not in schedule, so the reported gains leave open whether the decay itself or the better data produces them; rerunning annealing with the compositions swapped would separate the two effects.
  • The 7B model is an intermediate checkpoint stopped halfway through its scheduled updates by the paper's own account, so the published 7B numbers are plausibly a floor, and the paper's scaling story implies full convergence would widen the gap over the 1.7B model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper describes OpusLM, a family of decoder-only speech language models (135M, 360M, 1.7B, and 7B) initialized from pretrained text LLMs and continuously pretrained on 213K hours of public English speech-text pairs and 292B text-only tokens. Audio is represented by semantic plus acoustic discrete tokens in a delay-interleaved multi-stream architecture, with a text:semantic:acoustic loss reweighting. Training proceeds in a pretraining stage followed by a short annealing stage with two data compositions. The authors report LibriSpeech Test-Clean ASR WER of 2.3% (7B), TTS ASR-WER of 4.0% (1.7B), and MMLU of 59.0 (7B) with only 2.7 points of text degradation from the base LLM. They also present scaling experiments (135M/360M/1.7B), design-choice ablations (loss region, batch size, text corpus domain), and annealing experiments showing in-domain and partially out-of-domain gains. All code, data, checkpoints, and training logs are released.

Significance. If the headline results hold under comparable evaluation, the paper would be a valuable demonstration that open, public-data SpeechLMs can approach or match much larger or proprietary systems, and the full release of code, data, checkpoints, and logs is a genuine contribution to reproducibility in this area. The internal scaling and annealing ablations are self-consistent and provide useful practical guidance for SpeechLM pretraining. The architecture description is clear and the use of standard, publicly available tokenizers and LLM initializations lowers the barrier for follow-up work. However, the paper's central comparative claim—'comparable or even superior' performance—rests on cross-system numbers whose evaluation protocols are not matched, and the most prominent 7B results come from an explicitly unconverged checkpoint. These issues need to be addressed before the comparative claims can be accepted as stated.

major comments (3)
  1. [Section 3.1 and Table 1] The TTS comparison in Table 1 is not based on a matched evaluation protocol. The paper states in Section 3.1 that its own TTS evaluation uses VERSA with ASR-WER, speaker similarity, and proxy MOS, but it does not state that the ChatTTS, CosyVoice, and Parler-TTS rows (Table 1, rows 3-5) were re-evaluated under the same VERSA pipeline. Those numbers are almost certainly taken from each system's own publications, which involve different recognizers, test sets, and text normalization. Because TTS ASR-WER is highly sensitive to the ASR model and normalization, the claimed 4.0% versus 4.7% margin over Parler-TTS cannot be taken as evidence of superiority. Please either re-run the comparison systems under the identical VERSA evaluation or explicitly label Table 1 as a collection of numbers from heterogeneous protocols and soften the 'outperforms all other competitors' claim accordingly.
  2. [Section 2.3 footnote 9, Table 1, Section 3.2] The 7B model is an intermediate, unconverged checkpoint. Footnote 9 says OpusLM-7B was trained for only 250k updates and 'is not fully converged,' and footnote 15 says its TTS is weaker because 'TTS converges slower than ASR.' Despite this, the abstract and Table 1 present the 7B ASR WER of 2.3% and MMLU of 59.0 as headline results, and Section 3.2 states the 7B model 'outperforms all other SpeechLMs on the MMLU metric.' The main text should clearly mark these as intermediate-checkpoint numbers and either defer comparative claims for 7B or report the final converged numbers. As written, the paper gives the impression that OpusLM-7B is a finished model, which conflicts with its own appended limitation.
  3. [Section 3.4 and Table 3] The claim that annealing improves performance 'in general rather than adapting to the annealing data domain' is not fully supported. For Opt-B, out-of-domain ASR WER worsens from 13.9% to 14.5% (Table 3), and the only clear out-of-domain wins are in TTS. Moreover, the in-domain evaluation uses LibriSpeech test sets while Opt-A contains LibriSpeech and Opt-B contains LibriTTS (which the paper itself, in footnote 14, calls a re-segmented version of LibriSpeech), so the in-domain gains are at least partly explained by direct overlap with the annealing data. The paper should separate the evidence for ASR and TTS, report an out-of-domain ASR setting where annealing helps, or explicitly hedge the generalization claim to the TTS task.
minor comments (5)
  1. [Section 2.3] Typo: 'two trails of annealing' should be 'two trials of annealing.'
  2. [Section 3.1] 'using lm eval' should be written as 'using lm-evaluation-harness' with a proper citation, since the current text is ambiguous about the exact harness version and settings.
  3. [Section 2.2, Eq. (4)] The word 'parallelly' is nonstandard; use 'in parallel' or 'simultaneously.'
  4. [Table 2.b] The sentence 'the 4M batch size is translated to 125k updates to the model' is unclear; please clarify whether this refers to the number of optimizer steps or the effective number of epochs, and how it relates to the comparison at 30k and 60k updates.
  5. [Abstract and Figure 2] The abstract contains a spacing error ('presentsOpenUnifiedSpeechLanguage Models'), and the Figure 2 caption has a stray space in 'T ext MMLU'; these should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: OpusLM's performance claims are empirical training/evaluation outcomes, not derived from a fitted equation or from the authors' prior results.

full rationale

OpusLM's central claims are measured outcomes of training runs with released checkpoints; no target metric is defined in terms of an input parameter, and no fitted parameter is renamed as a prediction. The main design choices (delay-interleave architecture, token loss ratio 1:1/2:1/(N-1), annealing data compositions) are stated as assumptions or as empirically compared options, and the paper provides out-of-domain (GigaSpeech) and long-form (LibriHeavy) checks rather than relying only on in-domain test sets. The only domain-overlap caveat is disclosed in the footnote 'Note LibriTTS is roughly a re-segmented version of LibriSpeech so this evaluation should still be considered in-domain for Opt-B'; this limits the annealing TTS result but does not make it circular, because the result is a training/evaluation outcome, not a construction. Citations to the authors' own ESPnet-SpeechLM toolkit [25] and OWSM v3.2 [22] are infrastructure and data provenance, not load-bearing proofs of the headline capabilities. The Table 1 cross-system WER comparison may be questionable because OpusLM's TTS WER uses VERSA while baseline rows likely cite their own published numbers, but that is an evaluation-comparability issue, not a circularity of derivation.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claims rest on a handful of hand-chosen hyperparameters (loss weights, batch size, learning rates, decoding settings), a set of domain assumptions about tokenization and evaluation proxies, and the availability of public data and base LLMs. No new theoretical entities or fitted constants are introduced; the work is an empirical system built from known components.

free parameters (6)
  • Loss reweighting ratio (text:semantic:acoustic) = 1 : 0.5 : 1/(N-1), with N=9
    Chosen from an importance assumption stated in section 2.2; not tuned on dev sets but controls the training objective.
  • Number of tokens per frame (N) = 9 (1 semantic, 8 acoustic)
    Determined by the public tokenizers at 50 Hz with 8 codebooks; a design choice that fixes the sequence width.
  • Peak learning rate = 2e-4 for 135M/360M/1.7B; 1e-4 for 7B
    Hand-selected values; no sweep reported.
  • Batch size = 1M frames
    Chosen after 4M frames showed no decisive advantage in Table 2.b; not derived.
  • Annealing hyperparameters = start LR 5e-5, 85k updates, 10x upsample of four curated datasets
    Fixed for the annealing experiments in section 3.4; only two compositions are tried.
  • TTS decoding settings = top-k=30, temperature=0.7
    Constant sampling configuration for TTS evaluation; no tuning.
assumptions (5)
  • domain assumption Semantic plus acoustic discrete tokens are sufficient for speech understanding and generation
    Relied on throughout section 2.1; the paper inherits this from AudioLM/SoundStream-style tokenizers and does not prove it here.
  • domain assumption The public English data mixture (YODAS, Emilia, OWSM v3.2, 213K hours) is a sufficient training distribution
    The paper assumes the mixture is representative for competitive ASR and TTS; results provide indirect evidence but no formal guarantee.
  • domain assumption Delay-interleave with sum-of-embeddings preserves the dependencies needed for multi-stream speech modeling
    Adopted from MusicGen and prior SpeechLMs in section 2.2; the claimed intra-frame autoregression and O(T) inference are not formally analyzed.
  • domain assumption Automatic metrics (ASR-WER for TTS intelligibility, UTMOS proxy MOS, speaker similarity) are adequate proxies for speech quality
    All TTS conclusions in section 3.2 rest on these proxies; no human listening evaluation is reported.
  • domain assumption Keeping the text-only corpus in-domain with the base LLM pretraining corpus is necessary to retain text capability
    Empirical finding in Table 2.c; treated as a design rule rather than a demonstrated universal principle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OpusLM: A Family of Open Unified Speech Language Models." pith.science (2026). https://pith.science/paper/S76DLTVV

@misc{pith2026250617611,
  author       = {Pith},
  title        = {Pith review of: OpusLM: A Family of Open Unified Speech Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S76DLTVV}},
  note         = {Machine review of arXiv:2506.17611}
}
read the original abstract

This paper presents Open Unified Speech Language Models (OpusLMs), a family of open foundational speech language models (SpeechLMs) up to 7B. Initialized from decoder-only text language models, the OpusLMs are continuously pre-trained on 213K hours of speech-text pairs and 292B text-only tokens. We demonstrate our OpusLMs achieve comparable (or even superior) performance with existing SpeechLMs in speech recognition, speech synthesis, and text-only capabilities. Technically, this paper articulates our SpeechLM designs on tokenization, multi-stream language models, and multi-stage training strategies. We experimentally demonstrate the importance of model size scaling and the effect of annealing data selection. The OpusLMs are all built from publicly available materials and are fully transparent models. We release our code, data, checkpoints, and training logs to facilitate open SpeechLM research

Figures

Figures reproduced from arXiv: 2506.17611 by the authors.

Figure 1
Figure 1. An example of multi-stream speech-text interleaved sequence (upper) and its delay-interleaved sequence (lower). Assume N = 3, we append extra N −1 padding frames to avoid vertical overlap on the speech-text border. Tokens connected by arrows belong to one unique frame. Task Sequences: OpusLMs are trained on four tasks: speech￾only, text-only, speech recognition, and speech synthesis. The training sequence compositio… view at source ↗
Figure 2
Figure 2. Scaling property of OpusLM on 135M, 360M and 1.7B over ASR, TTS and text-only task Data and Open Source Practice: We follow the prior works to make our OpusLMs fully transparent [21,23,24], with all code, data, checkpoints, and training logs released. Additionally, al￾though our data volume is much smaller than prior works with proprietary data sources (e.g., No.13-15), we demonstrate our public data is still suffic… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UniVoice: Unifying Autoregressive ASR and Flow-Matching based TTS with Large Language Models

    eess.AS 2025-10 conditional novelty 6.0 of 10

    A single LLM can do ASR and zero-shot TTS on continuous speech features by switching between causal and bidirectional attention, reaching competitive but not state-of-the-art results.

Reference graph

Works this paper leans on

57 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    This prosperity quickly spread to the speech domain and invoked the popular- ity of speech language models (SpeechLMs) [4–6]

    Introduction Recent advances in large language models (LLMs) have yielded remarkable progress in text processing [1–3]. This prosperity quickly spread to the speech domain and invoked the popular- ity of speech language models (SpeechLMs) [4–6]. Specifi- cally, SpeechLMs are sequential models that convert speech- related tasks into sequential modeling pro...

    work page

  2. [2]

    The to- kenization is described in §2.1

    OpusLM This section introduces the design of the OpusLMs. The to- kenization is described in §2.1. The language model design is in §2.2. We finally describe our training process in §2.3. Note most modular features in our SpeechLM implementation are supported by the existing ESPnet-SpeechLM toolkit [25]. This paper explores the holistic system design of Sp...

    arXiv 2025

  3. [3]

    Experimental Setup Data:Our speech data is a mixture of YODAS [41], Emilia [42], and OWSM v3.2 suite [22]

    Experiments 3.1. Experimental Setup Data:Our speech data is a mixture of YODAS [41], Emilia [42], and OWSM v3.2 suite [22]. We restrict our corpus to English-only and obtain a total volume of 213K hours 10. We apply this data mixture to all ASR, TTS, and audio-only tasks, which gives 128B frames11. For text corpus, we follow the com- position of the pre-t...

    work page arXiv

  4. [4]

    The code, data, checkpoints, and training logs are released to support open speech language model research in the community

    Conclusion This paper presents OpusLM, a family of pre-trained speech language models that achieve competitive performance on speech recognition, speech synthesis, and text. The code, data, checkpoints, and training logs are released to support open speech language model research in the community

    work page

  5. [5]

    18Note the out-domain corpus in [25] is of sufficiently high quality, so we believe this degradation is more from domain mismatch

    Acknowledgement Experiments of this work used the Bridges2 system at PSC and Delta system at NCSA through allocations CIS210014 and IRI120008P from the Advanced Cyberinfrastructure Co- ordination Ecosystem: Services & Support (ACCESS) pro- gram, supported by National Science Foundation grants #2138259,#:2138286, #:2138307, #:2137603, and #:2138296. 18Note...

    work page

  6. [6]

    GPT-4 technical report,

    J. Achiamet al., “GPT-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  7. [7]

    The Llama 3 herd of models,

    A. Dubeyet al., “The Llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  8. [8]

    A survey of large language models,

    W. X. Zhaoet al., “A survey of large language models,”arXiv preprint arXiv:2303.18223, 2023

    arXiv 2023

Show all 57 references
  1. [9]

    Recent advances in speech language models: A survey,

    W. Cuiet al., “Recent advances in speech language models: A survey,”arXiv preprint arXiv:2410.03751, 2024

    2024 arXiv

  2. [10]

    A survey on speech large language models,

    J. Penget al., “A survey on speech large language models,”arXiv preprint arXiv:2410.18908, 2024

    2024

  3. [11]

    Wavchat: A survey of spoken dialogue models,

    S. Jiet al., “Wavchat: A survey of spoken dialogue models,”arXiv preprint arXiv:2411.13577, 2024

    2024 arXiv

  4. [12]

    Qwen2-audio technical report,

    Y . Chuet al., “Qwen2-audio technical report,”arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  5. [13]

    V oxtlm: Unified decoder-only models for consoli- dating speech recognition, synthesis and speech, text continuation tasks,

    S. Maitiet al., “V oxtlm: Unified decoder-only models for consoli- dating speech recognition, synthesis and speech, text continuation tasks,” inICASSP, 2024

    2024

  6. [14]

    Uniaudio: Towards universal audio generation with large language models,

    D. Yanget al., “Uniaudio: Towards universal audio generation with large language models,” inICML, 2024

    2024

  7. [15]

    SpeechX: Neural codec language model as a ver- satile speech transformer,

    X. Wanget al., “SpeechX: Neural codec language model as a ver- satile speech transformer,”TASLP, vol. 32, pp. 3355–3364, 2024

    2024

  8. [16]

    Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,

    Z. Zhanget al., “Speak foreign languages with your own voice: Cross-lingual neural codec language modeling,”arXiv preprint arXiv:2303.03926, 2023

    2023 arXiv

  9. [17]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D ´efossezet al., “Moshi: a speech-text foundation model for real-time dialogue,”arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  10. [18]

    VITA-1.5: Towards gpt-4o level real-time vision and speech interaction,

    C. Fuet al., “VITA-1.5: Towards gpt-4o level real-time vision and speech interaction,”arXiv:2501.01957, 2025

    2025 arXiv

  11. [19]

    Mini-omni2: Towards open-source gpt- 4o with vision, speech and duplex capabilities,

    Z. Xie and C. Wu, “Mini-omni2: Towards open-source gpt- 4o with vision, speech and duplex capabilities,”arXiv preprint arXiv:2410.11190, 2024

    2024 arXiv

  12. [20]

    GLM-4-V oice: Towards intelligent and human- like end-to-end spoken chatbot,

    A. Zenget al., “GLM-4-V oice: Towards intelligent and human- like end-to-end spoken chatbot,” 2024

    2024

  13. [21]

    Ocean-omni: To understand the world with omni- modality,

    Y . Liet al., “Ocean-omni: To understand the world with omni- modality,”arXiv preprint arXiv:2410.08565, 2024

    2024

  14. [22]

    V oiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech- Text Supervised Fine-Tuning,

    Y . Penget al., “V oiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech- Text Supervised Fine-Tuning,” inNAACL, 2025

    2025

  15. [23]

    Role of intelligence tests in speech/language,

    R. Sparkset al., “Role of intelligence tests in speech/language,” 1996

    1996

  16. [24]

    GPT-4o system card,

    A. Hurstet al., “GPT-4o system card,”arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  17. [25]

    Gemini: a family of highly capable multimodal mod- els,

    G. Team, “Gemini: a family of highly capable multimodal mod- els,”arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  18. [26]

    Owsm v3.1: Better and faster open whisper-style speech models based on e-branchformer,

    Y . Penget al., “Owsm v3.1: Better and faster open whisper-style speech models based on e-branchformer,” inInterspeech, 2024

    2024

  19. [27]

    On the effects of heterogeneous data sources on speech-to-text foundation models,

    J. Tianet al., “On the effects of heterogeneous data sources on speech-to-text foundation models,” inInterspeech, 2024

    2024

  20. [28]

    Olmo 2 furious,

    T. OLMo, “Olmo 2 furious,”arXiv preprint arXiv:2501.00656, 2024

    2024 arXiv

  21. [29]

    Parler-tts,

    Y . Lacombe and other, “Parler-tts,” https://github.com/ huggingface/parler-tts, 2024

    2024

  22. [30]

    ESPnet-SpeechLM: An open speech language model toolkit,

    J. Tianet al., “ESPnet-SpeechLM: An open speech language model toolkit,”arXiv, 2024

    2024

  23. [31]

    Librispeech: an ASR corpus based on public domain audio books,

    V . Panayotovet al., “Librispeech: an ASR corpus based on public domain audio books,” inICASSP, 2015, pp. 5206–5210

    2015

  24. [32]

    Measuring massive multitask language un- derstanding,

    D. Hendryckset al., “Measuring massive multitask language un- derstanding,”arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  25. [33]

    SALMONN: Towards generic hearing abilities for large language models,

    C. Tanget al., “SALMONN: Towards generic hearing abilities for large language models,” inICLR, 2024

    2024

  26. [34]

    Speak, read and prompt: High-fidelity text- to-speech with minimal supervision,

    E. Kharitonovet al., “Speak, read and prompt: High-fidelity text- to-speech with minimal supervision,”TACL, vol. 11, pp. 1703– 1718, 2023

    2023

  27. [35]

    Audiolm: a language modeling approach to au- dio generation,

    Z. Borsoset al., “Audiolm: a language modeling approach to au- dio generation,”TASLP, vol. 31, pp. 2523–2533, 2023

    2023

  28. [36]

    Soundstream: An end-to-end neural audio codec,

    N. Zeghidouret al., “Soundstream: An end-to-end neural audio codec,”TASLP, vol. 30, pp. 495–507, 2021

    2021

  29. [37]

    Robust speech recognition via large-scale weak supervision,

    A. Radfordet al., “Robust speech recognition via large-scale weak supervision,” inICML, 2023, pp. 28 492–28 518

    2023

  30. [38]

    Chattts: A generative speech model for daily dialogue

    2Noise, “Chattts: A generative speech model for daily dialogue.” 2024, available at https://github.com/2noise/ChatTTS. [Online]. Available: https://github.com/2noise/ChatTTS

    2024

  31. [39]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,

    Z. Duet al., “Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens,” arXiv preprint arXiv:2407.05407, 2024

    2024 arXiv

  32. [40]

    Gemma: Open models based on gemini research and technology,

    G. Team, “Gemma: Open models based on gemini research and technology,”arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  33. [41]

    Smollm2: When smol goes big–data- centric training of a small language model,

    L. B. Allalet al., “Smollm2: When smol goes big–data- centric training of a small language model,”arXiv preprint arXiv:2502.02737, 2025

    2025 arXiv

  34. [42]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvronet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  35. [43]

    Spirit-lm: Interleaved spoken and written language model,

    T. A. Nguyenet al., “Spirit-lm: Interleaved spoken and written language model,”arXiv preprint arXiv:2402.05755, 2024

    2024 arXiv

  36. [44]

    Neural codec language models are zero-shot text to speech synthesizers,

    C. Wanget al., “Neural codec language models are zero-shot text to speech synthesizers,”arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  37. [45]

    Simple and controllable music generation,

    J. Copetet al., “Simple and controllable music generation,” NeurIPS, vol. 36, 2024

    2024

  38. [46]

    Yodas: Youtube-oriented dataset for audio and speech,

    X. Liet al., “Yodas: Youtube-oriented dataset for audio and speech,” inASRU, 2023, pp. 1–8

    2023

  39. [47]

    Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,

    H. Heet al., “Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation,” inSLT, 2024, pp. 885–890

    2024

  40. [48]

    Zero: Memory optimizations toward train- ing trillion parameter models,

    S. Rajbhandariet al., “Zero: Memory optimizations toward train- ing trillion parameter models,” inSC20: International Conference for High Performance Computing, Networking, Storage and Anal- ysis. IEEE, 2020, pp. 1–16

    2020

  41. [49]

    PaLM: Scaling language modeling with pathways,

    A. Chowdheryet al., “PaLM: Scaling language modeling with pathways,”JMLR, vol. 24, no. 240, pp. 1–113, 2023

    2023

  42. [50]

    FLEURS: Few-Shot Learning Evaluation of Universal Representations of Speech,

    A. Conneauet al., “FLEURS: Few-Shot Learning Evaluation of Universal Representations of Speech,” inSLT, 2022

    2022

  43. [51]

    Espnet-spk: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models,

    J. weon Junget al., “Espnet-spk: full pipeline speaker embedding toolkit with reproducible recipes, self-supervised front-ends, and off-the-shelf models,” inInterspeech, 2024, pp. 4278–4282

    2024

  44. [52]

    Utmos: Utokyo-sarulab system for voicemos challenge 2022,

    T. Saekiet al., “Utmos: Utokyo-sarulab system for voicemos challenge 2022,” inInterspeech, 2022, pp. 4521–4525

    2022

  45. [53]

    Versa: A versatile evaluation toolkit for speech, au- dio, and music,

    J. Shiet al., “Versa: A versatile evaluation toolkit for speech, au- dio, and music,”arXiv preprint arXiv:2412.17667, 2024

    2024 arXiv

  46. [54]

    Libritts: A corpus derived from librispeech for text-to-speech,

    H. Zenet al., “Libritts: A corpus derived from librispeech for text-to-speech,” inInterspeech, 2019, pp. 1526–1530

    2019

  47. [55]

    CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR V oice Cloning Toolkit,

    J. Yamagishiet al., “CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR V oice Cloning Toolkit,” 2019

    2019

  48. [56]

    GigaSpeech: An Evolving, Multi-Domain ASR Corpus with 10,000 Hours of Transcribed Audio,

    G. Chenet al., “GigaSpeech: An Evolving, Multi-Domain ASR Corpus with 10,000 Hours of Transcribed Audio,” inInterSpeech, 2021

    2021

  49. [57]

    Libriheavy: a 50,000 hours asr corpus with punc- tuation casing and context,

    W. Kanget al., “Libriheavy: a 50,000 hours asr corpus with punc- tuation casing and context,” 2023

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.