Speech Encoder Fusion for LLM-based Automatic Speech Recognition
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-27 11:31 UTCgrok-4.3pith:CHJ2U3WQrecord.jsonopen to challenge →
The pith
Fusing multiple pre-trained speech encoders improves LLM-based ASR performance in mono-lingual, multi-lingual and diarized settings.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper establishes that carefully fusing multiple parallel pre-trained speech encoders into the embedding space of speech-aware LLMs improves downstream ASR performance across mono- and multilingual tasks as well as diarized speech recognition, using fusion strategies such as learned combinations and Transformer-based architectures that incur only limited extra computational cost.
What carries the argument
Fusion strategies (learned combinations and Transformer-based architectures) that merge outputs from several parallel pre-trained speech encoders before they enter the LLM embedding space.
If this is right
- ASR word error rates drop in single-language settings when multiple encoders are fused rather than used alone.
- Multilingual ASR accuracy rises without requiring separate models per language pair.
- Diarized speech recognition benefits from the same fusion approach with only modest added compute.
- Selection of a single best encoder becomes less critical once fusion is available.
- Overall system performance improves while keeping the LLM and most of the pipeline unchanged.
Where Pith is reading between the lines
- The same fusion logic might extend to other LLM-based speech tasks such as translation or summarization if the encoders supply complementary information there too.
- Future work could test whether the gains hold when one of the encoders is replaced by a much newer model.
- Hardware-aware fusion could further reduce the already-low overhead by pruning redundant encoder paths at inference time.
Load-bearing premise
The assumption that complementary strengths across encoders can be captured by the tested fusion methods without creating new failure modes or requiring heavy hyper-parameter search.
What would settle it
A controlled test in which any of the proposed fusion methods produces lower word error rate than the best single encoder on a held-out language or diarization condition would falsify the claim of improvement in all scenarios.
Figures
read the original abstract
Speech-aware large language models (LLMs) can incorporate speech through pre-trained acoustic encoders that project speech features into the LLM embedding space. While the choice of the speech encoder critically influences performance, different encoders often exhibit complementary strengths, motivating their combination. In this work, we investigate whether fusing multiple pre-trained speech encoders can enhance speech-aware LLMs for automatic speech recognition (ASR). We explore several fusion strategies beyond simple feature concatenation, including learned combinations and Transformer-based fusion architectures, and evaluate them across mono- and multilingual ASR settings as well as diarized speech recognition. Our results indicate that carefully fusing multiple parallel speech encoders improves downstream performance in all scenarios with limited computational overhead.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript investigates fusing multiple pre-trained speech encoders into speech-aware LLMs for ASR. It examines fusion strategies beyond simple concatenation (learned combinations and Transformer-based architectures) and reports performance gains on mono-lingual, multilingual, and diarized ASR tasks, achieved with limited additional computational overhead.
Significance. If the reported gains hold under controlled conditions, the work provides a practical route to exploit complementary encoder strengths in LLM-based ASR without large increases in compute or parameters. This could be useful for robust multilingual and diarized recognition where single-encoder performance is limited.
major comments (2)
- [§4] §4 (Experiments): the central claim that fusion improves performance 'in all scenarios' with 'limited computational overhead' requires explicit reporting of total parameter counts and FLOPs for fused versus single-encoder baselines; without these numbers the overhead claim cannot be verified.
- [Table 2] Table 2 (multilingual results): the reported WER reductions are presented without error bars, number of runs, or statistical significance tests; this weakens the assertion that gains are robust across settings.
minor comments (2)
- [Abstract] Abstract: the specific encoders and fusion methods are not named; adding one sentence listing the main encoders (e.g., Whisper, HuBERT) and the two primary fusion approaches would improve readability.
- [§3.2] Notation: the description of the Transformer-based fusion module uses inconsistent variable names for the attention outputs across §3.2 and the appendix; standardizing these would aid reproducibility.
Simulated Author's Rebuttal
We thank the referee for the positive assessment and constructive comments. We address each major point below and will update the manuscript accordingly.
read point-by-point responses
-
Referee: [§4] §4 (Experiments): the central claim that fusion improves performance 'in all scenarios' with 'limited computational overhead' requires explicit reporting of total parameter counts and FLOPs for fused versus single-encoder baselines; without these numbers the overhead claim cannot be verified.
Authors: We agree that explicit parameter counts and FLOPs are needed to substantiate the overhead claim. In the revised version we will add a dedicated subsection in §4 (and a supplementary table) that reports total trainable parameters and estimated inference FLOPs for each fused configuration versus the single-encoder baselines. Our internal calculations confirm the added cost remains below 15 % parameters and 20 % FLOPs for the largest fusion models; these numbers will be included verbatim. revision: yes
-
Referee: [Table 2] Table 2 (multilingual results): the reported WER reductions are presented without error bars, number of runs, or statistical significance tests; this weakens the assertion that gains are robust across settings.
Authors: We acknowledge that the absence of error bars or significance tests limits the strength of the robustness claim. Because of the substantial compute required for multilingual training, all reported results are from single runs. In the revision we will (i) explicitly state this limitation in the caption of Table 2 and in §4, and (ii) add a short paragraph discussing the consistency of gains across three independent language partitions and two diarization settings as supporting evidence. We do not plan to rerun the full multilingual suite with multiple seeds for this minor revision. revision: partial
Circularity Check
No significant circularity; empirical performance claims only
full rationale
The paper presents an empirical investigation of fusion strategies for speech encoders in LLM-based ASR. No derivation chain, equations, or first-principles results are claimed; performance improvements are reported from experiments across mono-/multilingual and diarized settings. The central claim reduces to measured gains from tested architectures rather than any self-definitional mapping, fitted parameter renamed as prediction, or self-citation load-bearing step. No load-bearing mathematical step exists that could reduce to its inputs by construction.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Introduction Recent advancements have led to the development of multi- modal LLMs that can process other modalities besides text, such as images or audio [1, 2, 3, 4]. These systems typically leverage a modality-specific encoder which converts the input image or audio to feature vectors, which are then aligned with the embedding space of the LLM via an ad...
-
[2]
Related Work Most research on building speech-aware LLMs with pre-trained speech and text models focuses on single encoder systems [9, 10, 11]. Previous works have leveraged a combination of multiple encoders, combining Whisper with WavLM [20], a speaker encoder [21], or several weak encoders [22], but they typically just concatenate or sum the output fea...
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[3]
Method We explore several methods to extend single-encoder speech LLMs to multi-encoder speech LLMs through encoder fusion. 3.1. Speech LLM A speech encoder generates a sequence of feature vectors, which are downsampled and projected to the embedding dimen- sion of the LLM by the projector. Then, the whole cascaded sys- tem of speech encoder, projector an...
-
[4]
Setup Speech encoder(s):We use theWhisper-large-v3encoder
-
[5]
for all experiments. For Dutch experiments, we incor- porate the encoder from theNeLFASR model 2 [19], which is a Conformer speech encoder pre-trained on 14k hours of weakly supervised Belgian Dutch speech in an encoder-decoder model with CTC regularization [24] and two distinct decoders, where one is trained with subtitle data and the other is trained wi...
2048
-
[6]
0: Hello. 1: How are you? 0: I’m fine
Results 5.1. Monolingual speech recognition For Dutch experiments, we combine the encoders of the mul- tilingual Whisper and the monolingual NeLF model using the proposed fusion mechanisms. For English, we combine Whis- per and the monolingual (finetuned) Wav2vec2 model. Results on Dutch (NL) are in Table 1, results on English are in Table 2. For Dutch, w...
-
[7]
Although speech-LLMs have difficulties at- taining performance of dedicated ASR systems, augmenting LLMs with speech capabilities has much wider applications be- sides ASR
Discussion Encoder fusion for speech-LLM is a simple technique with lim- ited overhead to incorporate strengths from multiple pre-trained speech encoders. Although speech-LLMs have difficulties at- taining performance of dedicated ASR systems, augmenting LLMs with speech capabilities has much wider applications be- sides ASR. Note that all experiments in ...
-
[8]
We found that careful fusion outperforms standard feature concatenation in all cases
Conclusion We have explored several methods to fuse the outputs of multi- ple pre-trained speech encoders to attain stronger speech-LLM systems in a variety of scenarios. We found that careful fusion outperforms standard feature concatenation in all cases
-
[9]
Acknowledgments Research supported by the Research Foundation Flanders (FWO) under grant S004923N of the SBO programme and by the Flemish Government under the ”Flanders AI Research Pro- gram”. Part of the resources and services used in this work were provided by the VSC (Flemish Supercomputer Center), funded by the Research Foundation Flanders (FWO) and t...
-
[10]
No part of the manuscript’s content or ideas was produced by generative AI
Use of Generative AI Disclosure The authors used Generative AI exclusively for text formatting, editing, and polishing to improve the clarity of this manuscript. No part of the manuscript’s content or ideas was produced by generative AI. All authors take full responsibility and account- ability for the original work and content of this paper
-
[11]
BLIP-2: Bootstrap- ping language-image pre-training with frozen image encoders and large language models,
J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrap- ping language-image pre-training with frozen image encoders and large language models,” inProc. Conf. on Machine Learning (ICML), 2023, pp. 19 730–19 742
2023
-
[12]
Visual instruction tuning,
H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” inAdvances in Neural Information Processing Systems, vol. 36, 2023, pp. 34 892–34 916
2023
-
[13]
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-Audio: Advancing universal audio understand- ing via unified large-scale audio-language models,” 2023, arXiv eprint 2311.07919
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[14]
SALMONN: Towards generic hearing abilities for large language models,
C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang, “SALMONN: Towards generic hearing abilities for large language models,” inProc. Intl. Conf. on Learning Rep- resentations (ICLR), 2024
2024
-
[15]
AudioChatLlama: Towards general-purpose speech abilities for LLMs,
Y . Fathullah, C. Wu, E. Lakomkin, K. Li, J. Jia, Y . Shang- guan, J. Mahadeokar, O. Kalinli, C. Fuegen, and M. Seltzer, “AudioChatLlama: Towards general-purpose speech abilities for LLMs,” inProc. Conf. of the North American Chapter of the As- sociation for Computational Linguistics (NAACL): Human Lan- guage Technologies (Volume 1: Long Papers), 2024, pp...
2024
-
[16]
Speech ReaLLM – real-time speech recognition with multimodal lan- guage models by teaching the flow of time,
F. Seide, Y . Shi, M. Doulaty, Y . Gaur, J. Jia, and C. Wu, “Speech ReaLLM – real-time speech recognition with multimodal lan- guage models by teaching the flow of time,” inProc. Interspeech, 2024, pp. 1900–1904
2024
-
[17]
Wav2Prompt: End-to-end speech prompt learning and task-based fine-tuning for text-based LLMs,
K. Deng, G. Sun, and P. Woodland, “Wav2Prompt: End-to-end speech prompt learning and task-based fine-tuning for text-based LLMs,” inProc. Conf. of the Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL): Hu- man Language Technologies (Volume 1: Long Papers), 2025, pp. 6940–6956
2025
-
[18]
BESTOW: Efficient and streamable speech language model with the best of two worlds in GPT and T5,
Z. Chen, H. Huang, O. Hrinchuk, K. C. Puvvada, N. R. Koluguri, P. ˙Zelasko, J. Balam, and B. Ginsburg, “BESTOW: Efficient and streamable speech language model with the best of two worlds in GPT and T5,” inProc. IEEE Spoken Language Technology Work- shop (SLT), 2024, pp. 147–154
2024
-
[19]
Speech recognition meets large language model: benchmarking, models, and exploration,
Z. Ma, G. Yang, Y . Yang, Z. Gao, J. Wang, Z. Du, F. Yu, Q. Chen, S. Zheng, S. Zhang, and X. Chen, “Speech recognition meets large language model: benchmarking, models, and exploration,” inProc. AAAI Conf. on Artificial Intelligence, 2025
2025
-
[20]
Connecting speech encoder and large language model for ASR,
W. Yu, C. Tang, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Connecting speech encoder and large language model for ASR,” inProc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 12 637–12 641
2024
-
[21]
How to connect speech foundation models and large language models? What matters and what does not,
F. Verdini, P. Melucci, S. Perna, F. Cariaggi, M. Gaido, S. Papi, S. Mazurek, M. Kasztelnik, L. Bentivogli, S. Brati `eres, P. Meri- aldo, and S. Scardapane, “How to connect speech foundation models and large language models? What matters and what does not,” inProc. Interspeech, 2025, pp. 1813–1817
2025
-
[22]
Prompting large language models with speech recog- nition abilities,
Y . Fathullah, C. Wu, E. Lakomkin, J. Jia, Y . Shangguan, K. Li, J. Guo, W. Xiong, J. Mahadeokar, O. Kalinli, C. Fuegen, and M. Seltzer, “Prompting large language models with speech recog- nition abilities,” inProc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 13 351–13 355
2024
-
[23]
Robust speech recognition via large-scale weak su- pervision,
A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, “Robust speech recognition via large-scale weak su- pervision,” inProc. Intl. Conf. on Machine Learning (ICML), vol. 202, 2023, pp. 28 492–28 518
2023
-
[24]
OWSM v4: Improving open whisper-style speech models via data scaling and cleaning,
Y . Peng, M. Shakeel, Y . Sudo, W. Chen, J. Tian, C.-J. Lin, and S. Watanabe, “OWSM v4: Improving open whisper-style speech models via data scaling and cleaning,” inProc. Interspeech, 2025, pp. 2225–2229
2025
-
[25]
wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,
A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,” inAdvances in Neural Information Processing Systems, vol. 33, 2020, pp. 12 449–12 460
2020
-
[26]
WavLM: Large-scale self-supervised pre-training for full stack speech processing,
S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y . Qian, Y . Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022
2022
-
[27]
Better pseudo- labeling with multi-ASR fusion and error correction by speech- LLM,
J. Prakash, B. Kumar, K. Hacioglu, B. Sharma, S. Gopalan, M. Chetlur, S. Venkatesan, and A. Stolcke, “Better pseudo- labeling with multi-ASR fusion and error correction by speech- LLM,” inProc. Interspeech, 2025, pp. 579–583
2025
-
[28]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” inAdvances in Neural Information Processing Systems, vol. 30, 2017
2017
-
[29]
Leveraging broadcast media sub- title transcripts for automatic speech recognition and subtitling,
J. Poncelet and H. Van hamme, “Leveraging broadcast media sub- title transcripts for automatic speech recognition and subtitling,” Journal on Audio, Speech and Music Processing, vol. 11, 2026
2026
-
[30]
WavLLM: Towards robust and adaptive speech large language model,
S. Hu, L. Zhou, S. Liu, S. Chen, L. Meng, H. Hao, J. Pan, X. Liu, J. Li, S. Sivasankaran, L. Liu, and F. Wei, “WavLLM: Towards robust and adaptive speech large language model,” inFindings of the Association for Computational Linguistics: EMNLP, 2024, pp. 4552–4572
2024
-
[31]
Diarization-aware multi-speaker automatic speech recognition via large language models,
Y . Lin, M. Cheng, Z. Li, B. Tang, and M. Li, “Diarization-aware multi-speaker automatic speech recognition via large language models,” 2025, arXiv eprint 2506.05796
-
[32]
MoWE-Audio: Multitask audioLLMs with mixture of weak encoders,
W. Zhang, S. Sun, B. Wang, X. Zou, Z. Liu, Y . He, G. Lin, N. F. Chen, and A. Ti Aw, “MoWE-Audio: Multitask audioLLMs with mixture of weak encoders,” inProc. IEEE Intl. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2025
2025
-
[33]
Enhancing speech large language models with prompt-aware mixture of audio encoders,
W. Shan, Y . Li, Y . Zhang, Y . Luo, C. Xu, X. Zhao, L. Meng, Y . Lu, M. Zhang, H. Yang, T. Xiao, and J. Zhu, “Enhancing speech large language models with prompt-aware mixture of audio encoders,” inProc. Conf. on Empirical Methods in Natural Language Pro- cessing, 2025, pp. 19 305–19 320
2025
-
[34]
Hybrid CTC/Attention architecture for end-to-end speech recog- nition,
S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, “Hybrid CTC/Attention architecture for end-to-end speech recog- nition,”IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017
2017
-
[35]
Learning to jointly transcribe and subtitle for end-to-end spontaneous speech recognition,
J. Poncelet and H. Van Hamme, “Learning to jointly transcribe and subtitle for end-to-end spontaneous speech recognition,” in Proc. IEEE Spoken Language Technology Workshop (SLT), 2023, pp. 182–189
2023
-
[36]
Lib- rispeech: An ASR corpus based on public domain audio books,
V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Lib- rispeech: An ASR corpus based on public domain audio books,” inProc. IEEE Intl. Conf. on Acoustics, Speech and Signal Pro- cessing (ICASSP), 2015, pp. 5206–5210
2015
-
[37]
ECAPA2: a hybrid neural net- work architecture and training strategy for robust speaker embed- dings,
J. Thienpondt and K. Demuynck, “ECAPA2: a hybrid neural net- work architecture and training strategy for robust speaker embed- dings,” inProc. IEEE Automatic Speech Recognition and Under- standing Workshop (ASRU), 2023
2023
-
[38]
Trans-tokenization and cross-lingual vocabulary transfers: language adaptation of LLMs for low-resource NLP,
F. Remy, P. Delobelle, H. Avetisyan, A. Khabibullina, M. de Lhoneux, and T. Demeester, “Trans-tokenization and cross-lingual vocabulary transfers: language adaptation of LLMs for low-resource NLP,” inProc. Conf. on Language Modeling (COLM), 2024
2024
-
[39]
AlignFormer: Modality matching can achieve better zero-shot instruction- following speech-LLM,
R. Fan, B. Ren, Y . Hu, R. Zhao, S. Liu, and J. Li, “AlignFormer: Modality matching can achieve better zero-shot instruction- following speech-LLM,”IEEE Journal of Selected Topics in Sig- nal Processing, vol. 19, no. 7, pp. 1329–1337, 2025
2025
-
[40]
The spoken Dutch corpus. overview and first evalu- ation,
N. Oostdijk, “The spoken Dutch corpus. overview and first evalu- ation,” inProc. Intl. Conf. on Language Resources and Evaluation (LREC), 2000
2000
-
[41]
Seri- alized output training for end-to-end overlapped speech recogni- tion,
N. Kanda, Y . Gaur, X. Wang, Z. Meng, and T. Yoshioka, “Seri- alized output training for end-to-end overlapped speech recogni- tion,” inProc. Interspeech, 2020, pp. 2797–2801
2020
-
[42]
The rich transcription 2007 meeting recognition evaluation,
J. G. Fiscus, J. Ajot, and J. S. Garofolo, “The rich transcription 2007 meeting recognition evaluation,” inMultimodal Technolo- gies for Perception of Humans, 2008, pp. 373–389
2007
-
[43]
arXiv preprint arXiv:2510.06961 , year=
V . Srivastav, S. Zheng, E. Bezzam, E. L. Bihan, N. Koluguri, P. ˙Zelasko, S. Majumdar, A. Moumen, and S. Gandhi, “Open ASR leaderboard: Towards reproducible and transparent multilingual and long-form speech recognition evaluation,” 2025, arXiv eprint 2510.06961
-
[44]
Evaluation of LLMs in speech is often flawed: Test set contamination in large language models for speech recognition,
Y . Tseng, T. Parcollet, R. van Dalen, S. Zhang, and S. Bhat- tacharya, “Evaluation of LLMs in speech is often flawed: Test set contamination in large language models for speech recognition,” inProc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2025
2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.