REVIEW 3 major objections 5 minor 1 cited by
TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read TESU-LLM claims that a speech-capable LLM can be trained on text alone by projecting a unified encoder's shared latent space into a frozen LLM, with a 13M-parameter projector making the transfer work.
desk verdict A cheap and clever text-only recipe for speech LLMs with real promise, but the central alignment claim is unmeasured and the evaluation has some overlap leaks. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the unified text-speech encoder constraint $M(E_{\text{text}}(x_{\text{text}})) \approx M(E_{\text{speech}}(x_{\text{speech}}))$, realized by SeamlessM4T with an NLLB text encoder, a w2v-BERT 2.0 speech encoder, and a shared mapper. Training uses interleaved pretraining in which random spans of 3 to 10 words in long-form text are replaced by encoder outputs and the model is trained only on unmasked token positions, followed by supervised fine-tuning on instruction QA plus a repetition task whose system prompt asks the model to reproduce the user's query verbatim. The projector must convert encoder output into LLM input space so that text-derived and speech-derived representations become interchangeable, and the repetition prompt is reused at ASR evaluation to force surface-form recovery.
What would settle it
Feed the same utterance to TESU-LLM twice, once as speech and once as text through the text encoder, and compare outputs on verbatim reproduction and instruction following. If the speech side systematically degrades relative to the text side, the encoder alignment described by Equation 1 is not tight enough to carry the transfer; this is directly testable on LibriSpeech test-clean with the released checkpoints.
Extended reading notes
Core claim
The paper's central claim is that text-only supervision suffices for speech understanding because the alignment already baked into a unified multilingual speech-text encoder transfers across modalities. Concretely, with the text encoder, speech encoder, and unified mapper of SeamlessM4T all frozen, only a two-layer MLP encoder projector is trained, on text spans and text-only instruction data. At inference, spoken queries pass through the speech encoder and the same projector, and the paper reports that the resulting model handles question answering, instruction following, adversarial prompts, ASR, and even unseen speech-to-text translation competitively. The contribution is therefore not a new encoder or a new LLM but a demonstration that modality alignment learned elsewhere removes the need to ever show the model speech.
Load-bearing premise
The whole method depends on the claim that semantically matching speech and text produce nearly the same vector in SeamlessM4T's shared latent space, so the text-trained projector needs no adjustment to handle speech.
Editorial extensions
If this is right
- Speech instruction following can be built from text-only corpora, eliminating the need for paired audio-text data, TTS synthesis, or time-aligned segmentation.
- Because only a 13M-parameter projector is trained while the LLM and encoder stay frozen, adding speech capability becomes feasible on two GPUs rather than large multimodal training runs.
- ASR and zero-shot speech-to-text translation emerge without task-specific speech training, with the reported LibriSpeech WER of 3.61/6.20 and CoVoST2 En-De BLEU of 24.30.
- The repetition-task prompt becomes a general-purpose way to make a text-trained model recover verbatim surface content from encoder embeddings, which is what enables transcription.
Reading between the lines
- If SeamlessM4T's alignment is as tight as assumed, the same recipe should transfer to any language or domain where a unified speech-text encoder exists, without collecting speech data; the paper motivates this possibility but does not demonstrate it.
- The paper leaves the text-versus-speech latent distance unmeasured on instruction-style content, so a direct probe of $M(E_{\text{text}}(x_{\text{text}}))$ versus $M(E_{\text{speech}}(x_{\text{speech}}))$ would reveal how much headroom remains.
- Because the repetition task uses the same prompt at training and ASR evaluation, prompt sensitivity is an open question; testing alternative phrasings would separate genuine alignment quality from prompt tuning.
- The paper explicitly sets aside emotion and prosody, so the method's scope is content understanding rather than expressive spoken interaction; incorporating paralinguistic cues would likely require additional speech-side supervision.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes TESU-LLM, a speech-capable LLM trained entirely from text. The method freezes LLaMA-3.1-8B-Instruct and SeamlessM4T-v2 and trains only a 13M-parameter two-layer MLP projector that maps the SeamlessM4T unified-encoder output into the LLM input space. Pretraining uses synthetic long-form text, and SFT uses instruction datasets plus a repetition task built from LibriSpeech transcripts. At inference, speech is passed through the same SeamlessM4T speech encoder and mapper. The paper reports competitive VoiceBench scores, ASR WERs of 3.61/6.20 on LibriSpeech test-clean/test-other, and a zero-shot CoVoST2 EN-DE BLEU of 24.30. The central premise is Eq. (1): the unified encoder maps semantically equivalent text and speech to the same latent vector.
Significance. If the central premise holds, the contribution is significant: TESU-LLM would demonstrate a compute-efficient, text-only route to speech instruction following and speech understanding, using only 13M trainable parameters and two A100 GPUs. The paper's strengths include a simple architecture, fully frozen pretrained components, a small trainable projector, and direct comparison against large multimodal speech-LLMs. The approach also makes a falsifiable prediction: any text prompt encodable by SeamlessM4T's text encoder can be replaced by the corresponding speech without performance loss. However, the paper does not provide the experiment that would verify this prediction, and two evaluation choices (OpenBookQA in SFT and the ASR repetition task) make the benchmark numbers less decisive than the abstract suggests.
major comments (3)
- [II-A, Eq. (1)] The central assumption is that M(E_text(x_text)) ≈ M(E_speech(x_speech)) for arbitrary content, but SeamlessM4T is a translation model and its unified space is optimized for cross-lingual semantic equivalence, not for verbatim surface information or arbitrary instruction content. The paper never directly measures the distance between text-encoder and speech-encoder outputs on held-out instruction-style sentences, nor does it compare the LLM's answers for text-encoded versus speech-encoded inputs on the same prompts. This is load-bearing because the repetition task, the ASR results, and the VoiceBench claims all depend on Eq. (1). Please add a direct verification: cosine distances between M(E_text(x)) and M(E_speech(x)) over held-out VoiceBench-style prompts; output-level agreement between text-input and speech-input conditions; and a check that the projection does not collapse distinct surface forms (e.g., minimal pairs with different answers). Without this, the claim in Section II-A that the model can process speech 'without any degradation in performance' is unsupported.
- [III-A and Table I] The SFT stage includes OpenBookQA and CommonSenseQA, and the VoiceBench evaluation includes OpenBookQA. Training on OpenBookQA and then reporting a VoiceBench OpenBookQA score of 59.34 is a train/evaluation overlap that inflates the comparison with baselines that did not train on that benchmark. Because VoiceBench's speech inputs are generated from the same text, the model may rely on memorized text content rather than on speech understanding. Please retrain without OpenBookQA and CommonSenseQA, or report an ablation that excludes them, and disclose the overlap when reporting the OpenBookQA column of Table I. This is necessary before claiming performance 'comparable to baseline methods trained with large-scale multimodal datasets' in the abstract and Section VI.
- [II-B2 and Table II] The ASR evaluation uses the exact repetition prompt and LibriSpeech transcriptions from the SFT stage. The paper does not state which LibriSpeech splits were used during SFT; if test-clean/test-other or their transcripts appear in the repetition data, the WER numbers are not a clean measure. Even if there is no split overlap, the evaluation task is a near copy of the training objective, so the strong ASR result does not demonstrate generalized speech understanding. Please specify the splits, remove any overlap, report WER with a generic transcription prompt such as 'Transcribe the following audio,' and include an ASR dataset not used in SFT (for example, a CommonVoice subset or WSJ). This would materially strengthen the claim that the model performs ASR rather than only the repetition behavior it was fine-tuned for.
minor comments (5)
- [III-B] The heading 'Model Configuation' contains a typo; it should read 'Model Configuration.'
- [II-A] The equations and inline math are missing spacing and proper subscripts, e.g., 'E text(xtext)' and 'M(E text(xtext))'; these should be typeset as E_text(x_text) and M(E_text(x_text)).
- [Throughout] The benchmark name 'VoiceBench' is inconsistently typeset with spurious spaces, e.g., 'V oiceBench' in Table I and the caption, and 'CoV oST2' in Table II.
- [III-A] The dataset name is written both as 'OpenbookQA' and 'OpenBookQA'; please standardize to the official spelling.
- [References] Reference [24] is missing author names; the UltraChat citation should be completed for a proper reference entry.
Circularity Check
No significant circularity: text-to-speech transfer rests on external SeamlessM4T alignment; caveats are unmeasured Eq. (1) and train/eval overlap, not circularity.
full rationale
The paper's derivation chain is not circular. The only trained parameters are the 13M Encoder Projector (Sec. II-B), fitted on text-only encoder outputs M(E_text(x_text)); no parameter is fit to speech data or to the test benchmarks. The central premise, Eq. (1), is an assumed property of the externally pretrained SeamlessM4T encoder (Sec. II-A), not a quantity derived from or enforced by TESU-LLM's loss; the speech evaluations are therefore genuine transfer tests whose outcome depends on whether SeamlessM4T's text-speech alignment holds for non-translation content. There are no load-bearing self-citations: SeamlessM4T, Speechless, BLSP, and VoiceBench are external prior work. The main caveats are non-circular: Sec. II-A asserts 'without any degradation in performance' without directly measuring M(E_text(x_text)) vs M(E_speech(x_speech)) on held-out content, and Sec. V lists only paralinguistics as a limitation; and some evaluation settings overlap with training data/task design (OpenBookQA appears in both the SFT dataset and VoiceBench; the Repetition Task is trained on LibriSpeech transcripts and the same prompt is reused for ASR evaluation), which is benchmark-contamination/correctness risk rather than derivation circularity. Score 1 reflects these minor caveats, not a circular derivation.
Assumptions & free parameters
free parameters (2)
- Encoder projector weights =
13M parameters
- Training hyperparameters =
LR=1e-4, batch 64, 3 epochs, span length 3-10, 200K synthetic samples
assumptions (3)
- domain assumption SeamlessM4T's unified encoder achieves M(E_text(x_text)) is approximately M(E_speech(x_speech)) for arbitrary instruction/query content.
- domain assumption Pretrained LLaMA-3.1-8B-Instruct retains instruction-following and reasoning when fed projected encoder outputs instead of text embeddings.
- domain assumption The Repetition Task trained on LibriSpeech text transcripts transfers to speech input at inference.
Cite this review
Pith. "Pith review of TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment." pith.science (2026). https://pith.science/paper/DIO3CPKC
@misc{pith2026250606343,
author = {Pith},
title = {Pith review of: TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment},
year = {2026},
howpublished = {\url{https://pith.science/paper/DIO3CPKC}},
note = {Machine review of arXiv:2506.06343}
}
read the original abstract
Recent advances in speech-enabled language models have shown promising results in building intelligent voice assistants. However, most existing approaches rely on large-scale paired speech-text data and extensive computational resources, which pose challenges in terms of scalability and accessibility. In this paper, we present \textbf{TESU-LLM}, a novel framework that enables training speech-capable language models using only text data. Our key insight is to leverage a unified encoder that maps semantically equivalent text and speech inputs to a shared latent space. By aligning the encoder output with the embedding space of a LLM via a lightweight projection network, we enable the model to generalize from text-only supervision to speech-based inference. Despite being trained exclusively on text, TESU-LLM achieves strong performance on various speech-related benchmarks, comparable to baseline methods trained with large-scale multimodal datasets and substantial computational resources. These results highlight the effectiveness and efficiency of our approach, offering a scalable path toward building speech LLMs without speech data.
Figures
Forward citations
Cited by 1 Pith paper
-
MEUSLI: a Multilingual Projector for LLM-based ASR and Beyond
MEUSLI, a released family of linear projectors, enables open-source LLM-based ASR across 28 European languages and supports few-hour adaptation to unseen languages and extra speech tasks.
Reference graph
Works this paper leans on
-
[1]
Moshi: a speech-text foundation model for real-time dialogue,
A. D ´efossez, L. Mazar ´e, M. Orsini, A. Royer, P. P ´erez, H. J ´egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,”arXiv preprint arXiv:2410.00037, 2024
arXiv 2024
-
[2]
Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture- of-loras,
A. Abouelenin, A. Ashfaq, A. Atkinson, H. Awadalla, N. Bach, J. Bao, A. Benhaim, M. Cai, V . Chaudhary, C. Chenet al., “Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture- of-loras,”arXiv preprint arXiv:2503.01743, 2025
arXiv 2025
-
[3]
Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Linet al., “Qwen2-audio technical report,”arXiv preprint arXiv:2407.10759, 2024
arXiv 2024
-
[4]
Baichuan-omni-1.5 technical report,
Y . Li, J. Liu, T. Zhang, S. Chen, T. Li, Z. Li, L. Liu, L. Ming, G. Dong, D. Panet al., “Baichuan-omni-1.5 technical report,”arXiv preprint arXiv:2501.15368, 2025
arXiv 2025
-
[5]
Llama-omni: Seamless speech interaction with large language models,
Q. Fang, S. Guo, Y . Zhou, Z. Ma, S. Zhang, and Y . Feng, “Llama-omni: Seamless speech interaction with large language models,”arXiv preprint arXiv:2409.06666, 2024
arXiv 2024
-
[6]
Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm,
X. Wang, Y . Li, C. Fu, Y . Shen, L. Xie, K. Li, X. Sun, and L. Ma, “Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm,”arXiv preprint arXiv:2411.00774, 2024
arXiv 2024
-
[7]
Blsp: Bootstrapping language-speech pre-training via behavior align- ment of continuation writing,
C. Wang, M. Liao, Z. Huang, J. Lu, J. Wu, Y . Liu, C. Zong, and J. Zhang, “Blsp: Bootstrapping language-speech pre-training via behavior align- ment of continuation writing,”arXiv preprint arXiv:2309.00916, 2023
arXiv 2023
-
[8]
Inserter: Speech instruction following with unsupervised interleaved pre-training,
D. Wang, J. Xu, R. Chu, Z. Guo, X. Wang, J. Wu, D. Yang, S. Ji, and J. Lin, “Inserter: Speech instruction following with unsupervised interleaved pre-training,”arXiv preprint arXiv:2503.02769, 2025
arXiv 2025
Show all 29 references
-
[9]
Distilling an end-to-end voice assistant without instruction training data,
W. Held, E. Li, M. Ryan, W. Shi, Y . Zhang, and D. Yang, “Distilling an end-to-end voice assistant without instruction training data,”arXiv preprint arXiv:2410.02678, 2024
2024 arXiv
-
[10]
Speechless: Speech instruction training without speech for low resource languages,
A. Dao, D. B. Vu, H. H. Ha, T. L. D. Anh, S. Gopal, Y . H. Yeo, W. K. H. Low, E. S. Chng, and J. Q. Yip, “Speechless: Speech instruction training without speech for low resource languages,” 2025. [Online]. Available: https://arxiv.org/abs/2505.17417
2025 arXiv
-
[11]
Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing,
J. Ao, R. Wang, L. Zhou, C. Wang, S. Ren, Y . Wu, S. Liu, T. Ko, Q. Li, Y . Zhanget al., “Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing,”arXiv preprint arXiv:2110.07205, 2021
2021 arXiv
-
[12]
Slam: A unified encoder for speech and language modeling via speech-text joint pre-training,
A. Bapna, Y .-a. Chung, N. Wu, A. Gulati, Y . Jia, J. H. Clark, M. Johnson, J. Riesa, A. Conneau, and Y . Zhang, “Slam: A unified encoder for speech and language modeling via speech-text joint pre-training,”arXiv preprint arXiv:2110.10329, 2021
-
[13]
mslam: Massively multilingual joint pre-training for speech and text,
A. Bapna, C. Cherry, Y . Zhang, Y . Jia, M. Johnson, Y . Cheng, S. Khanuja, J. Riesa, and A. Conneau, “mslam: Massively multilingual joint pre-training for speech and text,”arXiv preprint arXiv:2202.01374, 2022
2022 arXiv
-
[14]
Seam- lessm4t: Massively multilingual & multimodal machine translation,
L. Barrault, Y .-A. Chung, M. C. Meglioli, D. Dale, N. Dong, P.-A. Duquenne, H. Elsahar, H. Gong, K. Heffernan, J. Hoffmanet al., “Seam- lessm4t: Massively multilingual & multimodal machine translation,” arXiv preprint arXiv:2308.11596, 2023
2023 arXiv
-
[15]
No language left behind: Scaling human-centered machine translation,
M. R. Costa-Juss `a, J. Cross, O. C ¸ elebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillardet al., “No language left behind: Scaling human-centered machine translation,” arXiv preprint arXiv:2207.04672, 2022
2022 arXiv
-
[16]
W2v-bert: Combining contrastive learning and masked language mod- eling for self-supervised speech pre-training,
Y .-A. Chung, Y . Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y . Wu, “W2v-bert: Combining contrastive learning and masked language mod- eling for self-supervised speech pre-training,” in2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021, ...
2021
-
[17]
V oicebench: Benchmarking llm-based voice assistants,
Y . Chen, X. Yue, C. Zhang, X. Gao, R. T. Tan, and H. Li, “V oicebench: Benchmarking llm-based voice assistants,”arXiv preprint arXiv:2410.17196, 2024
2024 arXiv
-
[18]
Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot,
A. Zeng, Z. Du, M. Liu, K. Wang, S. Jiang, L. Zhao, Y . Dong, and J. Tang, “Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot,”arXiv preprint arXiv:2412.02612, 2024
2024 arXiv
-
[19]
Paralinguistics-aware speech- empowered large language models for natural conversation,
H. Kim, S. Seo, K. Jeong, O. Kwon, S. Kim, J. Kim, J. Lee, E. Song, M. Oh, J.-W. Haet al., “Paralinguistics-aware speech- empowered large language models for natural conversation,”arXiv preprint arXiv:2402.05706, 2024
2024 arXiv
-
[20]
Spirit-lm: Interleaved spoken and written language model,
T. A. Nguyen, B. Muller, B. Yu, M. R. Costa-Jussa, M. Elbayad, S. Popuri, C. Ropers, P.-A. Duquenne, R. Algayres, R. Mavlyutov et al., “Spirit-lm: Interleaved spoken and written language model,” Transactions of the Association for Computational Linguistics, vol. 13, pp. 30–52, 2025
2025
-
[21]
Dissecting learning and forgetting in language model finetuning,
X. Zhang and J. Wu, “Dissecting learning and forgetting in language model finetuning,” inThe Twelfth International Conference on Learning Representations, 2024
2024
-
[22]
The llama 3 herd of models,
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024
2024 arXiv
-
[23]
Openwebtext corpus,
A. Gokaslan, V . Cohen, E. Pavlick, and S. Tellex, “Openwebtext corpus,” http://Skylion007.github.io/OpenWebTextCorpus, 2019
2019
-
[24]
Enhancing chat language models by scaling high-quality instructional conversations,
N. Ding, Y . Chen, B. Xu, Y . Qin, Z. Zheng, S. Hu, Z. Liu, M. Sun, and B. Zhou, “Enhancing chat language models by scaling high-quality instructional conversations,” 2023
2023
-
[25]
Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants,
Teknium, “Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants,” 2023. [Online]. Available: https: //huggingface.co/datasets/teknium/OpenHermes-2.5
2023
-
[26]
Can a suit of armor conduct electricity? a new dataset for open book question answering,
T. Mihaylov, P. Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,” inEMNLP, 2018
2018
-
[27]
CommonsenseQA: A question answering challenge targeting commonsense knowledge,
A. Talmor, J. Herzig, N. Lourie, and J. Berant, “CommonsenseQA: A question answering challenge targeting commonsense knowledge,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V...
2019
-
[28]
Librispeech: an asr corpus based on public domain audio books,
V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210
2015
-
[29]
Covost 2 and massively multilingual speech-to-text translation,
C. Wang, A. Wu, and J. Pino, “Covost 2 and massively multilingual speech-to-text translation,”arXiv preprint arXiv:2007.10310, 2020
2007 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.