LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors

Bhuvana Ramabhadran; Kartik Audhkhasi; Rao Ma; Tongzhou Chen

REVIEW 2 major objections 4 minor 37 references

LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors

T0 review · 2 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A frozen speech encoder can feed an LLM through CTC posteriors and be swapped zero-shot.

desk verdict A clean new interface for speech-LLM integration with solid experiments; the zero-shot swapping claim overreaches the evidence. read the letter →

arxiv 2505.11352 v1 pith:VLYWXGDM submitted 2025-05-16 cs.CL cs.SDeess.AS

Rao Ma , Tongzhou Chen , Kartik Audhkhasi , Bhuvana Ramabhadran This is my paper · ORCID

classification cs.CLcs.SDeess.AS

keywords CTCposteriorsspeech-LLMinterfaceautomaticspeechrecognitiontranslationzero-shotencoderswapembedding-tablereconstructionmodularASRsoftmaxtemperaturecontrol

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes LegoSLM, a way to connect a pre-trained speech encoder to a large language model without training a special mapping network or decoding into text first. The speech encoder is fine-tuned with CTC loss over the LLM's own token vocabulary, and each frame's posterior distribution is used as weights to combine the LLM's input embeddings into a pseudo-audio embedding. These reconstructed embeddings are concatenated with text embeddings and fed to the LLM, which is fine-tuned to transcribe or translate. On multilingual MLS ASR the method reports an average 49% word-error-rate reduction over the USM-CTC baseline, matches a speech-prompt baseline on ASR while keeping the encoder frozen, and beats AEC and speech-prompt baselines on speech translation. The deeper claim is modularity: because only CTC posteriors cross the interface, a differently-trained speech encoder can be plugged into the already-fine-tuned LLM with no retraining.

What carries the argument

The load-bearing object is Eq. (4): $s_t = E \cdot o_t$, where $E$ is the LLM's input embedding table and $o_t$ is the CTC posterior vector over the LLM vocabulary (including a special blank mapped to a newly initialized embedding). This turns the LLM's text embeddings into a codebook for speech: the encoder never sends continuous features, only a distribution over text tokens, and the LLM's own embeddings reconstruct an audio-like input. The CTC blank token, the optional top-K variants, and the temperature $\tau$ in the softmax all modify this one reconstruction step.

What would settle it

A decisive test: train the same LegoSLM pipeline but replace the pretrained LLM embedding table $E$ with a randomly initialized table, keeping the rest of the LLM pretrained and training the system as in the paper. If WER stays near 5.6% on MLS_en, the linguistic structure of the embeddings is not what carries the signal; if WER degrades sharply, the codebook hypothesis is supported.

Watch

Extended reading notes

Core claim

The central claim is that a CTC posterior matrix over the LLM's word-piece vocabulary is a sufficient interface between speech and language. Formally, with embedding table $E$ and per-frame posteriors $o_t$, the speech representation is $s_t = E \cdot o_t$; this weighted sum is then concatenated with text embeddings for the LLM. The paper shows this representation carries enough information to support both ASR and speech translation, and that because the interface is a probability distribution over tokens rather than continuous encoder features, a different encoder trained on the same CTC vocabulary can be substituted after LLM fine-tuning with no weight updates. The paper also reports that downweighting the CTC blank token (LegoSLM*) improves ASR, and that a softmax temperature on the CTC posteriors acts as an acoustic-model/language-model weight for zero-shot combinations.

Load-bearing premise

The load-bearing premise is that a probability distribution over the LLM's text tokens, averaged through the token embedding table, is a rich enough representation of speech for the LLM to recover the utterance; the paper validates this by experiment, not by analysis.

Editorial extensions

If this is right

On MLS-en ASR, LegoSLM* reaches 5.6% WER with a frozen USM encoder, close to the 5.5% of the speech-prompt system and well below the 7.8% of 10-best AEC.
On multilingual MLS ASR across 8 languages, LegoSLM* gives an average 49% WERR over the USM-CTC baseline, versus 50% for speech prompts and 22% for AEC.
In zero-shot encoder swapping, a Gemma fine-tuned on MLS-en USM outputs transfers to public, multi, and LibriSpeech encoders with 32-37% WERR, while speech-prompt systems fail with WER above 165.
On CoVoST 2 speech translation, LegoSLM* achieves the best BLEU in fr->en, de->en, and en->de, and keeps competitive BLEU when the encoder is swapped.
A softmax temperature tau around 0.6-1.1 improves zero-shot combinations, with lower tau favoring a stronger encoder and higher tau giving the LLM more freedom.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the reconstruction step is as information-preserving as the experiments suggest, the same interface could be used for tasks beyond ASR and translation, such as speech summarization or spoken language understanding, without retraining the encoder.
The success of the top-K softmax variant suggests the effective interface is sparse; a testable extension is whether keeping far fewer than 100 tokens per frame, or selecting them dynamically, can cut compute while preserving WER.
Because the blank token dominates CTC posteriors and carries little content, its downweighting is a hidden hyperparameter; other encoders with different alignment statistics may need different blank weights, and the temperature control partially automates this.
The vocabulary-mismatch experiment (16K vs 256K tokens with an extra learned embedding table) implies the approach does not require shared tokenizers, so it could connect encoders and LLMs in different languages or with different subword units through a small learned map.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Ma et al. propose LegoSLM: fine-tune a speech encoder (USM) with CTC over the LLM's vocabulary, then reconstruct the LLM's input embeddings as a weighted sum of the embedding table using the CTC posteriors. That's a genuinely new and elegant way to connect a speech encoder to an LLM, and it works. On MLS multilingual ASR they get 49% average WERR over USM-CTC (Table 2: 17.8 to 9.1, internally consistent), matching a tuned speech-prompt baseline on average and beating AEC. On CoVoST2 AST, LegoSLM* outperforms all baselines including AEC n=10. The ablations (blank downscaling, temperature control, top-K variants, vocab mismatch) are sensible and clearly reported. The paper is honest about training details and includes a thorough appendix.

The main soft spot is the scope of the zero-shot modularity claim. Every swapped encoder in Tables 3 and 7 is a USM-CTC checkpoint: same architecture, same pre-trained base, differing only in ASR fine-tuning data. The abstract says 'the speech encoder can be switched' which sounds general, and Section 3.4 promises any encoder operating on the same CTC vocabulary. That is not demonstrated. The result is still useful—it shows the LLM can be reused when its own encoder is updated or adapted, which is a real practical scenario—but the claim should be narrowed.

Second, Table 7 reports 'the best BLEU score achieved with the optimal temperature value' without saying whether the temperature was selected on a held-out set. Appendix C shows test-set curves; if the temperature is chosen on the test set, the zero-shot AST comparison is optimistic. This is a moderate concern, not a fatal one: the main ASR results use default temperature, and the temperature trends are otherwise clear.

Minor: no error bars or significance tests, and no code release. Also, the speech-prompt baseline is a simple projection layer; the comparison is fair enough given they also report U+P+G.

All that said, the central idea is sound and the empirical execution is above average. The paper deserves serious peer review; I'd ask the authors to (a) re-scope the zero-shot claim or test a genuinely different encoder family, and (b) specify how temperature was chosen. For a reading group, it's worth a slot as a good example of a modular speech-LLM interface.

Referee Report

2 major / 4 minor

Summary. The paper proposes LegoSLM, a method to connect a pre-trained CTC-based speech encoder with a decoder-only LLM. Instead of feeding continuous encoder representations or ASR hypotheses, the encoder is fine-tuned to produce CTC posteriors over the LLM's vocabulary; the posteriors are then linearly combined with the LLM's input embedding table (Eq. 4) to form pseudo-speech embeddings, which are concatenated with text embeddings for the LLM input. The speech encoder is frozen during LLM fine-tuning. Experiments on ASR (MLS, LibriSpeech, SpeechStew) and AST (CoVoST 2) show that the method outperforms ASR error correction baselines and is competitive with or better than simple speech-prompt baselines, with a reported 49% average WERR over the USM-CTC baseline on 8 MLS languages. The paper also introduces a temperature parameter to control the acoustic/language influence at decode time, and demonstrates zero-shot combination of LLMs with USM-CTC encoders trained on different data.

Significance. If the claims hold, LegoSLM is a simple and modular interface between speech encoders and LLMs, with potential advantages in privacy (only posteriors exchanged), cost (no beam search or N-best generation), and flexibility (encoder can be swapped). The paper provides extensive experiments across multiple languages and datasets, and the headline results are internally consistent; for example, the 49% WERR matches the average of the per-language WERRs derivable from Table 2. The proposed method and ablations are clearly described, and the comparison to AEC and speech prompts is useful. However, the modularity claim is only supported for sibling USM-CTC checkpoints, and the selection protocol for key hyperparameters (blk_downscale and temperature) is not reported, which tempers the strength of the conclusions.

major comments (2)

[Section 3.4 and Tables 3, 5, 7 (also Appendix C)] The zero-shot modularity claim is broader than the evidence. Every zero-shot experiment swaps USM-CTC models that share the same USM architecture and pre-trained base, differing only in the supervised ASR data (mls-en, multi, public, lbs). The claim in Section 3.4 that the LLM can accept outputs from a different speech encoder as long as the model also operates on the same CTC vocabulary is not tested with a genuinely different encoder family. Shared frame-level features, blank-posterior statistics, and error patterns across sibling checkpoints may make transfer much easier than across arbitrary encoders. Please either add an experiment with a different encoder (e.g., a different architecture such as Whisper, or a CTC model initialized from a different pretrained checkpoint) or explicitly limit the claim to swapping USM-CTC checkpoints fine-tuned on different data.
[Sections 3.3, 5.1, 5.3 and Tables 2, 7] The selection protocol for blk_downscale and temperature is not reported. The headline 49% WERR (Table 2) uses LegoSLM* with blk_downscale=1e4 in Eq. (5), and Table 7 reports the best BLEU score achieved with the optimal temperature value without stating whether the temperature was chosen on a held-out set. If these values were selected using the test set, the reported gains are optimistic. Please state whether blk_downscale=1e4 and the optimal tau values were chosen on a held-out development set, and if so, report the held-out selection details. If not, report results at a fixed default (e.g., blk_downscale=1 and tau=1) alongside the tuned results.

minor comments (4)

[Appendix C] The first paragraph says 'Table 10 presents the detailed WER results' but the correct table reference is Table 15; additionally, in the last row of Table 15, '5,7' should be '5.7'.
[Equation (7)] The notation it = argmax_k(zt) is nonstandard; it should refer to the indices of the top-K elements. Consider defining it = top-k(zt) for clarity.
[Tables 1 and 2] The comparison between LegoSLM* and SP (P+G) relies on WER differences as small as 0.1-0.3 (e.g., Table 2, en: 5.7 vs 5.5; es: 5.8 vs 5.4). Without error bars or significance tests, the relative ordering of these systems should be presented more cautiously.
[Section 3.3] It would be helpful to clarify whether the reconstructed speech embeddings for blank frames are used as the randomly initialized vector during training or whether those frames are masked, since the random embedding may inject noise early in adaptation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LegoSLM is an empirical interface design; its results are measured, not derived from the method's definitions.

full rationale

The paper makes no first-principles derivation that could reduce to its inputs. Equation (4) (s_t = E * o_t) defines the proposed speech-to-LLM interface by construction, but every performance claim is an empirical outcome measured against external baselines (USM-CTC, speech prompts, AEC) on held-out MLS, LibriSpeech, and CoVoST 2 test sets; the 49% WERR and BLEU numbers are not algebraic consequences of Eq. (4). The CTC posterior generation (Eqs. 2-3) is standard supervised training and is not fitted to the LLM targets in a way that would make the downstream ASR/AST results forced. The temperature knob (Eq. 6) is an explicit decode-time hyperparameter, and its tuning is reported transparently in Appendix C; even if the "optimal temperature" in Table 7 may have been selected on test data, that is an evaluation-protocol weakness rather than a circular derivation. The zero-shot modularity experiments swap USM-CTC checkpoints that share the same USM architecture and pre-trained base, so the modularity claim is narrower than the abstract's wording; the Limitations section itself acknowledges that other encoder architectures are not tested. This is a scope/validity concern, not circularity. Self-citations (e.g., Ma et al. 2023a,b) are used only for baseline construction and related work, and no load-bearing argument is reduced to a self-cited uniqueness theorem or fitted parameter renamed as a prediction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method is empirical; the main load-bearing assumptions are that CTC posteriors over the LLM vocabulary are informative enough, and that the LLM embedding table is a good codebook for weighted-sum reconstruction. The paper validates these only empirically, without error bars or theoretical justification.

free parameters (3)

blk_downscale = 1e4 (LegoSLM* variant; default 1)
Logit offset applied to the CTC blank token in Eq. (5). The value 1e4 was chosen because further increase gave no improvement; it materially changes results, e.g., 5.6 vs 6.1 WER on MLS_en.
CTC softmax temperature tau = varies per setup, e.g., 1.1 and 0.6 in Figure 4; 'optimal' per Table 7
Decode-time temperature in Eq. (6) controls the AM/LM balance. The optimal value is selected per encoder/LLM combination, and Table 7 reports the best BLEU across temperatures, so the reported zero-shot numbers depend on this tuned parameter.
Additional input embedding table for mismatched vocab experiment = 12M parameters, randomly initialized and jointly trained
Introduced in Section 5.4 to map 16K ASR logits into the 256K LLM embedding space; trained on MLS en-us data, so it is fitted to the evaluation setup.

assumptions (4)

domain assumption Fine-tuning a pre-trained speech encoder with CTC loss over the LLM vocabulary yields frame-level posteriors that retain enough phonetic information for the LLM to use.
Section 3.1, Eq. (1)-(3). The method depends on the CTC posteriors being informative after fine-tuning; no theoretical guarantee is provided.
domain assumption The LLM embedding table is a suitable codebook for reconstructing speech information via a weighted sum.
Section 3.2, Eq. (4) s_t = E * o_t. This is the core interface assumption and is validated only empirically.
domain assumption A decoder-only LLM can be fine-tuned to interpret reconstructed speech embeddings without modifications to the speech encoder.
Section 3.2. The method assumes the LLM can learn to read the pseudo-audio embeddings after fine-tuning on supervised speech data.
ad hoc to paper Mapping the CTC blank token to a randomly initialized embedding and optionally downscaling its logit is a valid way to handle the blank token.
Section 3.3, Eq. (5). The blank-token embedding and blk_downscale are introduced specifically for this method and affect performance.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors." pith.science (2026). https://pith.science/paper/VLYWXGDM

@misc{pith2026250511352,
  author       = {Pith},
  title        = {Pith review of: LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VLYWXGDM}},
  note         = {Machine review of arXiv:2505.11352}
}

read the original abstract

Recently, large-scale pre-trained speech encoders and Large Language Models (LLMs) have been released, which show state-of-the-art performance on a range of spoken language processing tasks including Automatic Speech Recognition (ASR). To effectively combine both models for better performance, continuous speech prompts, and ASR error correction have been adopted. However, these methods are prone to suboptimal performance or are inflexible. In this paper, we propose a new paradigm, LegoSLM, that bridges speech encoders and LLMs using the ASR posterior matrices. The speech encoder is trained to generate Connectionist Temporal Classification (CTC) posteriors over the LLM vocabulary, which are used to reconstruct pseudo-audio embeddings by computing a weighted sum of the LLM input embeddings. These embeddings are concatenated with text embeddings in the LLM input space. Using the well-performing USM and Gemma models as an example, we demonstrate that our proposed LegoSLM method yields good performance on both ASR and speech translation tasks. By connecting USM with Gemma models, we can get an average of 49% WERR over the USM-CTC baseline on 8 MLS testsets. The trained model also exhibits modularity in a range of settings -- after fine-tuning the Gemma model weights, the speech encoder can be switched and combined with the LLM in a zero-shot fashion. Additionally, we propose to control the decode-time influence of the USM and LLM using a softmax temperature, which shows effectiveness in domain adaptation.

Figures

Figures reproduced from arXiv: 2505.11352 by the authors.

**Figure 2.** Depiction of the proposed LegoSLM method. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Illustration of the zero-shot system combina [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Effect of changing the temperature value in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Effect of changing the temperature value in LegoSLM on the CoVoST 2 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 22 canonical work pages

[1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report . arXiv preprint arXiv:2303.08774

arXiv 2023
[2]

Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common Voice: A Massively-Multilingual Speech Corpus . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4218--4222

2020
[3]

Lo \" c Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023. SeamlessM4T-Massively Multilingual & Multimodal Machine Translation . arXiv preprint arXiv:2308.11596

arXiv 2023
[4]

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models . arXiv preprint arXiv:2108.07258

arXiv 2021
[5]

Rami Botros, Rohit Prabhavalkar, Johan Schalkwyk, Ciprian Chelba, Tara N Sainath, and Fran c oise Beaufays. 2023. Lego-Features: Exporting modular encoder features for streaming and deliberation ASR . In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5. IEEE

work page 2023
[6]

William Chan, Daniel Park, Chris Lee, Yu Zhang, Quoc Le, and Mohammad Norouzi. 2021. Speechstew: Simply mix all available speech recognition data to train one large neural network . arXiv preprint arXiv:2104.02133

arXiv 2021
[7]

Chen Chen, Yuchen Hu, Chao-Han Huck Yang, Sabato Marco Siniscalchi, Pin-Yu Chen, and Eng-Siong Chng. 2023. Hyporadise: An open baseline for generative speech recognition with large language models . In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 31665--31688

work page 2023
[8]

Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. 2022. Self-supervised learning with random-projection quantizer for speech recognition . In International Conference on Machine Learning, pages 3915--3924. PMLR

work page 2022

Show all 37 references

[9]

Siddharth Dalmia, Dmytro Okhonko, Mike Lewis, Sergey Edunov, Shinji Watanabe, Florian Metze, Luke Zettlemoyer, and Abdelrahman Mohamed. 2023. LegoNN: Building modular encoder-decoder models . IEEE/ACM Transactions on Audio, Speech, and Language Processing

2023
[10]

Rahhal Errattahi, Asmaa El Hannani, and Hassan Ouahmane. 2018. Automatic speech recognition errors detection and correction: A review . Procedia Computer Science, 128:32--37

2018
[11]

Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, et al. 2024. Prompting large language models with speech recognition abilities . In ICASSP 2024-2024 IEEE International Conference on Acous...

2024
[12]

Marco Gaido, Sara Papi, Matteo Negri, and Luisa Bentivogli. 2024. Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1...

2024
[13]

Alex Graves, Santiago Fern \'a ndez, Faustino Gomez, and J \"u rgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks . In Proceedings of the 23rd international conference on Machine learning, pages 369--376

2006
[14]

Yukiya Hono, Koh Mitsuda, Tianyu Zhao, Kentaro Mitsui, Toshiaki Wakatsuki, and Kei Sawada. 2023. An integration of pre-trained speech and language models for end-to-end speech recognition . arXiv preprint arXiv:2312.03668

2023 arXiv
[15]

Sheng Li, Chen Chen, Chin Yuen Kwok, Chenhui Chu, Eng Siong Chng, and Hisashi Kawai. 2024. Investigating ASR error correction with large language model and multilingual 1-best hypotheses . In Proc. Interspeech 2024, pages 1315--1319

2024
[16]

Rao Ma, Mark J. F. Gales, Kate M. Knill, and Mengjie Qian. 2023 a . N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space . In Proc. INTERSPEECH, pages 3267--3271

2023
[17]

Rao Ma, Mengjie Qian, Mark J. F. Gales, and Kate M. Knill. 2023 b . Adapting an Unadaptable ASR System . In Proc. INTERSPEECH 2023, pages 989--993

2023
[18]

Rao Ma, Mengjie Qian, Potsawee Manakul, Mark Gales, and Kate Knill. 2023 c . Can generative large language models perform ASR error correction? arXiv preprint arXiv:2307.04172

2023 arXiv
[19]

Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, et al. 2024. An Embarrassingly Simple Approach for LLM with Strong ASR Capacity . arXiv preprint arXiv:2402.08846

2024 arXiv
[20]

Anirudh Mani, Shruti Palaskar, Nimshi Venkat Meripo, Sandeep Konam, and Florian Metze. 2020. ASR error correction and domain adaptation using machine translation . In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6344...

2020
[21]

Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an ASR corpus based on public domain audio books . In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206--5210. IEEE

2015
[22]

Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D

Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. 2019. SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition . In Interspeech 2019, pages 2613--2617

2019
[23]

Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, et al. 2024. Scaling speech technology to 1,000+ languages . Journal of Machine Learning Research, 25(97):1--52

2024
[24]

Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. MLS: A Large-Scale Multilingual Dataset for Speech Research . In Interspeech 2020, pages 2757--2761

2020
[25]

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, MA Zejun, and Chao Zhang. 2024. SALMONN: Towards Generic Hearing Abilities for Large Language Models . In The Twelfth International Conference on Learning Representations

2024
[26]

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology . arXiv preprint arXiv:2403.08295

2024 arXiv
[27]

Shubham Toshniwal, Anjuli Kannan, Chung-Cheng Chiu, Yonghui Wu, Tara N Sainath, and Karen Livescu. 2018. A comparison of techniques for language model integration in encoder-decoder speech recognition . In 2018 IEEE spoken language technology workshop (SLT), pages 369--375. IEEE

2018
[28]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models . arXiv preprint arXiv:2302.13971

2023 arXiv
[29]

Francesco Verdini, Pierfrancesco Melucci, Stefano Perna, Francesco Cariaggi, Marco Gaido, Sara Papi, Szymon Mazurek, Marek Kasztelnik, Luisa Bentivogli, S \'e bastien Brati \`e res, et al. 2024. How to Connect Speech Foundation Models and Large Language Models? What Matters an...

2024 arXiv
[30]

Changhan Wang, Anne Wu, Jiatao Gu, and Juan Pino. 2021. CoVoST 2 and Massively Multilingual Speech Translation

2021
[31]

Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. 2024. VisionLLM: Large language model is also an open-ended decoder for vision-centric tasks . Advances in Neural Information Processing Systems, 36

2024
[32]

Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, et al. 2023. On decoder-only architecture for speech-to-text and large language model integration . In 2023 IEEE Automatic Speech Recognition and Understanding Wor...

2023
[33]

Steve Young, Gunnar Evermann, Mark Gales, Thomas Hain, Dan Kershaw, Xunying Liu, Gareth Moore, Julian Odell, Dave Ollason, Dan Povey, et al. 2002. The HTK book . Cambridge university engineering department, 3(175):12

2002
[34]

Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, et al. 2023. Google USM: Scaling automatic speech recognition beyond 100 languages . arXiv preprint arXiv:2303.01037

2023 arXiv
[35]

Linchen Zhu, Wenjie Liu, Linquan Liu, and Edward Lin. 2021. Improving ASR error correction using n-best hypotheses . In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 83--89. IEEE

2021
[36]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[37]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report . arXiv preprint arXiv:2303.08774

arXiv 2023

[2] [2]

Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common Voice: A Massively-Multilingual Speech Corpus . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4218--4222

2020

[3] [3]

Lo \" c Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023. SeamlessM4T-Massively Multilingual & Multimodal Machine Translation . arXiv preprint arXiv:2308.11596

arXiv 2023

[4] [4]

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models . arXiv preprint arXiv:2108.07258

arXiv 2021

[5] [5]

Rami Botros, Rohit Prabhavalkar, Johan Schalkwyk, Ciprian Chelba, Tara N Sainath, and Fran c oise Beaufays. 2023. Lego-Features: Exporting modular encoder features for streaming and deliberation ASR . In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5. IEEE

work page 2023

[6] [6]

William Chan, Daniel Park, Chris Lee, Yu Zhang, Quoc Le, and Mohammad Norouzi. 2021. Speechstew: Simply mix all available speech recognition data to train one large neural network . arXiv preprint arXiv:2104.02133

arXiv 2021

[7] [7]

Chen Chen, Yuchen Hu, Chao-Han Huck Yang, Sabato Marco Siniscalchi, Pin-Yu Chen, and Eng-Siong Chng. 2023. Hyporadise: An open baseline for generative speech recognition with large language models . In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 31665--31688

work page 2023

[8] [8]

Chung-Cheng Chiu, James Qin, Yu Zhang, Jiahui Yu, and Yonghui Wu. 2022. Self-supervised learning with random-projection quantizer for speech recognition . In International Conference on Machine Learning, pages 3915--3924. PMLR

work page 2022