REVIEW 3 major objections 6 minor 1 cited by
Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Stream-Omni claims that mapping speech to text in the layer dimension, via CTC, transfers text capabilities to speech using only 23,000 hours of speech data.
desk verdict A coherent omni-modal architecture worth a serious look, but the printed streaming algorithm is broken and the core data-efficiency claim needs tighter evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the CTC-based layer-dimension speech-text mapping. CTC (Connectionist Temporal Classification) is a loss and decoding scheme that predicts a per-timestep distribution over a vocabulary and collapses repeated and blank tokens into a text sequence; here it is applied over the discrete speech-unit sequence, so each speech unit carries a distribution over the merged text-plus-speech vocabulary. This mechanism carries the argument in three ways: it gives direct supervised training signal for speech-to-text alignment in the bottom speech layers; its decoded alignment tells the model exactly which text token a speech unit corresponds to, enabling the alignment-based fusion in the top speech layers that streams speech generation behind the text generation (with a lag of K tokens); and it naturally yields intermediate ASR text as a byproduct. All three claimed advantages—data efficiency, streaming interaction, and intermediate text output—flow from this single mechanism.
What would settle it
Run Stream-Omni on speech with heavy disfluency, background noise, code-switching, or long pauses, and compare the streamed CTC transcript against the final text response and the audited speech; if the intermediate ASR output diverges from the final answer, or if speech generation skips or repeats text tokens, the monotonic-alignment assumption is violated. Concretely, a dataset designed so that a phrase's speech units and text tokens are non-monotonic (e.g., 'um, it's...') should expose the failure.
Extended reading notes
Core claim
Stream-Omni claims that speech and text share such close semantic structure that they should be mapped in the layer dimension, not merely concatenated in the sequence dimension. The paper adds three bottom speech layers and five top speech layers around the LLM backbone: the bottom layers convert discrete speech units (from the CosyVoice tokenizer) into text via connectionist temporal classification (CTC), giving a non-autoregressive speech-to-text path, and the top layers convert text back into speech units in a streaming, wait-k fashion. The CTC decoder does double duty: during inference it supplies the positional alignment that tells the model when a generated speech unit corresponds to a completed text token, which drives the alignment-based fusion in the top speech layers. The paper reports that this design supports strong visual understanding, speech interaction, and vision-grounded speech interaction, and that it can simultaneously emit ASR transcripts and text replies while speaking.
Load-bearing premise
The whole design hinges on CTC over discrete speech units producing a reliable, monotonic alignment between speech and text, and on that alignment being usable online to decide when a generated speech unit completes a text token; if that alignment is brittle on realistic speech, the streamed ASR output and the streaming speech generation degrade together.
Editorial extensions
If this is right
- If the layer-dimension mapping holds up, an omni-modal assistant can be built on a text LLM backbone without re-learning textual knowledge from speech data.
- Streaming intermediate text (ASR and the in-progress response) becomes a free byproduct of the speech-text alignment, so users can read along while the model talks.
- Speech training data requirements could drop by an order of magnitude relative to concatenation-based omni-models, since the alignment supplies structure rather than relying on data-driven correlation.
- Stream-Omni's reported visual benchmark results suggest that adding speech layers does not measurably degrade the text-vision capability of the underlying LLM.
Reading between the lines
- One testable extension is whether the CTC layer-dimension mapping transfers to other language pairs or to code-switched speech, where speech-text monotonicity is less clean.
- The design implies a general recipe: modalities that share semantics with text (speech, perhaps video transcripts) can be mapped in the layer dimension, while modalities that add complementary information (images, structures) should stay in the sequence dimension.
- The streaming ASR output could be repurposed as a lightweight always-on keyword detector or as a text-based control channel, since the model already computes the transcript during speech generation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Stream-Omni proposes an 8B large language-vision-speech model in which vision is aligned with text via sequence-dimension concatenation and speech is aligned with text via a CTC-based layer-dimension mapping between bottom speech layers and an LLM backbone. The model is trained in three stages using roughly 23,000 hours of speech, including TTS-synthesized InstructOmni data, and supports text, vision, and speech inputs and outputs, with streaming ASR and simultaneous text generation during speech interaction. The paper reports vision benchmark scores, knowledge-based spoken QA results, a newly constructed SpokenVisIT benchmark, LibriSpeech ASR results, and an ablation of the alignment-based fusion mechanism.
Significance. If the central claims hold, Stream-Omni offers a credible path toward reducing speech training data for omni-modal LMMs and enabling simultaneous intermediate text outputs during speech interaction. The paper's strengths include releasing code and model weights, constructing and releasing datasets, and including a focused ablation of the fusion mechanism. The architecture's two-alignment strategy is clearly motivated, and the reported numbers are competitive with or better than several baselines. However, the significance is currently limited by an internally inconsistent streaming inference pseudocode and by evaluation evidence that does not yet isolate the claimed data-efficiency advantage.
major comments (3)
- [Algorithm 1 / §3.3] Algorithm 1 as printed cannot implement the streaming speech generation described in Section 3.3. Lines 4 and 9 index bY[-1] and bA[-1], bA[-2] after initializing both lists to empty, so the pseudocode would raise an IndexError on the first evaluation. More importantly, line 12 decodes CTCDec(F_bottom_speech(U)) with U the fixed input speech units from line 2; the generated units appended to bU in line 11 are never fed into the bottom speech layers or the CTC decoder. The inner loop condition therefore never changes: either it exits immediately or loops forever, and no speech is generated for any text token. This contradicts the prose in Section 3.3, which states that the CTC decoder identifies a new text token from the generated speech. Please correct the pseudocode so that the CTC decoder consumes the generated speech prefix (e.g., U concatenated with bU) and verify that the released implementation follows the corrected procedure.
- [§5.2–5.3, Tables 3–4] The central claim of data efficiency ('less data, especially speech') is not directly supported by the experiments. Tables 3 and 4 report single runs without error bars or significance tests, and the gaps are often small: Stream-Omni is 60.3 vs 59.7 average S→T against VITA-1.5 in Table 3, and 3.68 vs 3.45 on SpokenVisIT in Table 4. The comparison is also confounded by different training data, model sizes, and evaluation protocols. A matched ablation is needed: train a sequence-concatenation variant of Stream-Omni on the same 23K-hour corpus, or otherwise show that the CTC layer-dimension mapping, rather than the particular data mixture, drives the efficiency gain. Without this, the abstract's data-efficiency assertion remains plausible but unverified.
- [§5.3, Table 4, Appendix B] SpokenVisIT is a self-constructed benchmark: the spoken instructions are synthesized with CosyVoice, the same TTS system used to produce the model's training speech, and evaluation uses GPT-4o ratings on Whisper transcriptions with a provided image caption. This creates a domain-match advantage and an evaluation pipeline whose noise is not quantified. The paper should report a human-correlation study for the 1–5 ratings, include at least one established benchmark transformed to speech, and state how many instruction instances are evaluated after removing the eight math items. As it stands, Table 4 is too weak to support the 'superior real-world visual understanding' claim.
minor comments (6)
- [Table 1, Stage 2] The entry 'ASR (CTC Loss in Eq.(6))' should reference Eq. (4), since Eq. (6) is the fusion cross-attention formula, not the CTC loss.
- [Table 2] The average column appears to mix MME's 0–2000 scale with percentage-based benchmarks; please clarify how the Avg. is computed or exclude MME from the average.
- [Algorithm 1, line 14] If line 14 is taken literally, the speech decoder resynthesizes the entire bU sequence on every inner-loop iteration; please state whether synthesis is performed once after streaming or clarify that the line accumulates audio segments.
- [Table 6] The W=∞ row is not defined; please explain how full-context attention is realized in the alignment-based fusion module.
- [§3.1.2] There are minor grammatical issues, e.g., 'the bottom speech layersF_bottom_speech(·)maps' should be 'the bottom speech layers F_bottom_speech(·) map'.
- [§4.3] The configuration section reports the GPU count but not training hyperparameters such as batch size, learning rate, number of steps, or sequence length; adding these would improve reproducibility.
Circularity Check
No significant circularity: the core CTC alignment and text-to-speech transfer claims are trained on real ASR data and evaluated on external benchmarks, and the CosyVoice TTS/benchmark overlap is a distributional concern rather than a definitional reduction.
full rationale
Stream-Omni's central derivation is not circular. The speech-to-text mapping is trained with a standard CTC loss (Eq. 4) on real ASR corpora (LibriSpeech, WenetSpeech) plus TTS data, and the mapping quality is evaluated on the external LibriSpeech ASR benchmark (Table 5), not on the model's own outputs. The knowledge-transfer claim is tested on spoken QA benchmarks against external baselines, so it does not reduce to a fitted parameter or to the model's training signal. The use of CosyVoice both as tokenizer/decoder and as the TTS for InstructOmni/SpokenVisIT is a distributional-overlap concern: training and evaluation speech share the same synthesizer, which can inflate speech-interaction scores, but this is a data-construction issue rather than a 'prediction equals input by construction' pattern. Self-citations to Stream-Speech [43] supply the CTC-alignment idea, but that is a published, externally reviewed result, and the present paper's empirical validation is independent. One non-circularity issue should be flagged separately: Algorithm 1 line 12 decodes CTCDec(F_bottom_speech(U)) from the fixed input U rather than from the growing generated sequence bU, so the printed streaming termination condition is decoupled from the generated speech and cannot work as described; this is an internal correctness/consistency bug, not a circular derivation.
Assumptions & free parameters
free parameters (4)
- Bottom speech layer count =
3
- Top speech layer count =
5
- Fusion window size W =
5
- Lagging text tokens K =
3
assumptions (5)
- domain assumption CTC over discrete speech units produces a valid monotonic alignment between speech and text.
- domain assumption Speech and text are semantically consistent, while vision is semantically complementary to text.
- domain assumption TTS-synthesized speech from text instructions is representative of real speech interaction data.
- domain assumption Pretrained components (SigLIP vision encoder, LLaMA-3.1 backbone, CosyVoice tokenizer and decoder) provide sufficient representations.
- domain assumption Ground-truth transcriptions used in Stage 2 training of top speech layers are close enough to LLM-generated text at inference.
Cite this review
Pith. "Pith review of Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model." pith.science (2026). https://pith.science/paper/4KAVDJNR
@misc{pith2026250613642,
author = {Pith},
title = {Pith review of: Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/4KAVDJNR}},
note = {Machine review of arXiv:2506.13642}
}
read the original abstract
The emergence of GPT-4o-like large multimodal models (LMMs) has raised the exploration of integrating text, vision, and speech modalities to support more flexible multimodal interaction. Existing LMMs typically concatenate representation of modalities along the sequence dimension and feed them into a large language model (LLM) backbone. While sequence-dimension concatenation is straightforward for modality integration, it often relies heavily on large-scale data to learn modality alignments. In this paper, we aim to model the relationships between modalities more purposefully, thereby achieving more efficient and flexible modality alignments. To this end, we propose Stream-Omni, a large language-vision-speech model with efficient modality alignments, which can simultaneously support interactions under various modality combinations. Stream-Omni employs LLM as the backbone and aligns the vision and speech to the text based on their relationships. For vision that is semantically complementary to text, Stream-Omni uses sequence-dimension concatenation to achieve vision-text alignment. For speech that is semantically consistent with text, Stream-Omni introduces a CTC-based layer-dimension mapping to achieve speech-text alignment. In this way, Stream-Omni can achieve modality alignments with less data (especially speech), enabling the transfer of text capabilities to other modalities. Experiments on various benchmarks demonstrate that Stream-Omni achieves strong performance on visual understanding, speech interaction, and vision-grounded speech interaction tasks. Owing to the layer-dimensional mapping, Stream-Omni can simultaneously provide intermediate text outputs (such as ASR transcriptions and model responses) during speech interaction, offering users a comprehensive multimodal experience.
Figures
Forward citations
Cited by 1 Pith paper
-
Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory
Dual global+latent states with hierarchical episodic merging enable reflexive, low-latency long-video agents that beat iterative reasoning baselines on accuracy and efficiency.
Reference graph
Works this paper leans on
-
[1]
OpenAI. Hello gpt-4o, 2024. URLhttps://openai.com/index/hello-gpt-4o/
work page 2024
-
[2]
Gpt-4v(ision) system card, 2024
OpenAI. Gpt-4v(ision) system card, 2024. URL https://cdn.openai.com/papers/ GPTV_System_Card.pdf
work page 2024
-
[3]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc., 2023. 10 URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 6dcf277ea32ce3...
work page 2023
-
[4]
MiniGPT-4: Enhancing vision-language understanding with advanced large language models
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum?id=1tZbq88f27
work page 2024
-
[5]
Llava-next: Improved reasoning, ocr, and world knowledge, January 2024
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https: //llava-vl.github.io/blog/2024-01-30-llava-next/
2024
-
[6]
Llava-onevision: Easy visual task transfer, 2024
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. URL https://arxiv.org/abs/2408.03326
arXiv 2024
-
[7]
Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024
arXiv 2024
-
[8]
Mini-omni: Language models can hear, talk while thinking in streaming, 2024
Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming, 2024. URLhttps://arxiv.org/abs/2408.16725
arXiv 2024
Show all 79 references
-
[9]
LLaMA- omni: Seamless speech interaction with large language models
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. LLaMA- omni: Seamless speech interaction with large language models. InThe Thirteenth Interna- tional Conference on Learning Representations, 2025. URL https://openreview.net/ forum?id=PYmrUQmMEw
2025
-
[10]
Moshi: a speech-text foundation model for real-time dialogue, 2024
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue, 2024. URLhttps://arxiv.org/abs/2410.00037
2024 arXiv
-
[11]
Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot, 2024
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot, 2024. URLhttps://arxiv.org/abs/2412.02612
2024 arXiv
-
[12]
Vita-1.5: Towards gpt-4o level real-time vision and speech interaction, 2025
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, Long Ma, Xiawu Zheng, Rongrong Ji, Xing Sun, Caifeng Shan, and Ran He. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction, 2025. URLhttps:/...
2025 arXiv
-
[13]
Baichuan-omni technical report, 2024
Yadong Li, Haoze Sun, Mingan Lin, Tianpeng Li, Guosheng Dong, Tao Zhang, Bowen Ding, Wei Song, Zhenglin Cheng, Yuqi Huo, Song Chen, Xu Li, Da Pan, Shusen Zhang, Xin Wu, Zheng Liang, Jun Liu, Tao Zhang, Keer Lu, Yaqi Zhao, Yanjun Shen, Fan Yang, Kaicheng Yu, Tao Lin, Jianhua Xu...
2024
-
[14]
Qwen2.5-omni technical report, 2025
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report, 2025. URLhttps://arxiv.org/abs/2503.20215
2025 arXiv
-
[15]
Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. InPro- ceedings of the 23rd International Conference on Machine Learning, ICML ’06, page 369–37...
2006
-
[16]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila an...
2021
-
[17]
Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023. URL https://arxiv.org/abs/2308. 12966
2023
-
[18]
Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qin- glong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedi...
2024
-
[19]
Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models, 2023
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, and Yu Qiao. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large...
2023 arXiv
-
[20]
mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CV...
2024
-
[21]
Internvideo: General video foundation models via generative and discriminative learning, 2022
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative lear...
2022 arXiv
-
[22]
Videochat: Chat-centric video understanding, 2024
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024. URL https: //arxiv.org/abs/2305.06355
2024 arXiv
-
[23]
Video-ChatGPT: Towards detailed video understanding via large vision and language models
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Comput...
2024
-
[24]
Llama-vid: An image is worth 2 tokens in large language models, 2023
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models, 2023. URLhttps://arxiv.org/abs/2311.17043
2023 arXiv
-
[25]
Video-LLaV A: Learning united visual representation by alignment before projection
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaV A: Learning united visual representation by alignment before projection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in...
2024
-
[26]
Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm, 2024
Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm, 2024. URLhttps://arxiv.org/abs/2411.00774
2024 arXiv
-
[27]
Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation, 2024
Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, and Chao Zhang. Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation, 2024. URLhttps://arxiv.org/abs/2411.18138
2024 arXiv
-
[28]
Slam-omni: Timbre-controllable voice interaction system with single-stage training,
Wenxi Chen, Ziyang Ma, Ruiqi Yan, Yuzhe Liang, Xiquan Li, Ruiyang Xu, Zhikang Niu, Yanqiao Zhu, Yifan Yang, Zhanxun Liu, Kai Yu, Yuxuan Hu, Jinyu Li, Yan Lu, Shujie Liu, and Xie Chen. Slam-omni: Timbre-controllable voice interaction system with single-stage training,
-
[29]
Robust speech recognition via large-scale weak supervision, 2022
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. 12
2022
-
[30]
URLhttps://arxiv.org/abs/2412.15649
-
[31]
Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3...
2021
-
[32]
SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Findings of the Association for Computat...
2023 doi
-
[33]
Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens, 2024
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijie Yan. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens, 2024. URL https: //arxiv.org/ab...
2024 arXiv
-
[34]
Speechtokenizer: Uni- fied speech tokenizer for speech language models
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Uni- fied speech tokenizer for speech language models. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id= AF9Q8Vip84
2024
-
[35]
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, and Emmanuel Dupoux. SpiRit-LM: Inter...
2025 doi
-
[36]
Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In H. Larochelle, M. Ran- zato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Information Processing Systems, volume 33, ...
2020
-
[37]
M2-omni: Advancing omni-mllm for comprehensive modality support with competitive performance, 2025
Qingpei Guo, Kaiyou Song, Zipeng Feng, Ziping Ma, Qinglong Zhang, Sirui Gao, Xuzheng Yu, Yunxiao Sun, Tai-Wei Chang, Jingdong Chen, Ming Yang, and Jun Zhou. M2-omni: Advancing omni-mllm for comprehensive modality support with competitive performance, 2025. URL https://arxiv.or...
2025 arXiv
-
[38]
Megrez-omni technical report, 2025
Boxun Li, Yadong Li, Zhiyuan Li, Congyi Liu, Weilin Liu, Guowei Niu, Zheyue Tan, Haiyang Xu, Zhuyu Yao, Tao Yuan, Dong Zhou, Yueqing Zhuang, Shengen Yan, Guohao Dai, and Yu Wang. Megrez-omni technical report, 2025. URL https://arxiv.org/abs/2502. 15803
2025
-
[39]
Kwok, Hengshuang Zhao, Xiaodan Liang, Dit-Yan Yeung, Xiao Chen, Zhenguo Li, Wei Zhang, Qun Liu, Jun Yao, Lanqing Hong, Lu Hou, and Hang Xu
Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, Dingdong Wang, Kun Xiang, Haoyuan Li, Haoli Bai, Jianhua Han, Xiaohui Li, Weike Jin, Nian Xie, Yu Zhang, James T. Kwok, Hengshuang Zhao, Xiaodan Liang, Dit-Yan Yeung,...
2025 arXiv
-
[40]
Capybara-omni: An efficient paradigm for building omni-modal language models, 2025
Xingguang Ji, Jiakang Wang, Hongzhi Zhang, Jingyuan Zhang, Haonan Zhou, Chenxi Sun, Yahui Liu, Qi Wang, and Fuzheng Zhang. Capybara-omni: An efficient paradigm for building omni-modal language models, 2025. URLhttps://arxiv.org/abs/2504.12315
2025 arXiv
-
[41]
Omni-emotion: Extending video mllm with detailed face and audio modeling for multimodal emotion analysis, 2025
Qize Yang, Detao Bai, Yi-Xing Peng, and Xihan Wei. Omni-emotion: Extending video mllm with detailed face and audio modeling for multimodal emotion analysis, 2025. URL https://arxiv.org/abs/2501.09502
2025 arXiv
-
[42]
Openomni: Advancing open-source omnimodal large language models with progressive multimodal alignment and real-time self-aware emotional speech synthesis, 2025
Run Luo, Ting-En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, Yangyi Chen, Xiaobo Xia, Hamid Alinejad-Rokny, and Fei Huang. Openomni: Advancing open-source omnimodal large language models with progressive multimodal alignm...
2025
-
[43]
Stream- Speech: Simultaneous speech-to-speech translation with multi-task learning
Shaolei Zhang, Qingkai Fang, Shoutao Guo, Zhengrui Ma, Min Zhang, and Yang Feng. Stream- Speech: Simultaneous speech-to-speech translation with multi-task learning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the As- soci...
2024 doi
-
[44]
LLaV A-mini: Efficient image and video large multimodal models with one vision token
Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. LLaV A-mini: Efficient image and video large multimodal models with one vision token. InThe Thirteenth International Con- ference on Learning Representations, 2025. URL https://openreview.net/forum? id=UQJ7CDW8nb
2025
-
[45]
Future-guided incremental transformer for si- multaneous translation.Proceedings of the AAAI Conference on Artificial Intelligence, 35 (16):14428–14436, May 2021
Shaolei Zhang, Yang Feng, and Liangyou Li. Future-guided incremental transformer for si- multaneous translation.Proceedings of the AAAI Conference on Artificial Intelligence, 35 (16):14428–14436, May 2021. URL https://ojs.aaai.org/index.php/AAAI/ article/view/17696
2021
-
[46]
STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework
Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework. InPro...
2019 doi
-
[47]
Information-transport-based policy for simultaneous trans- lation
Shaolei Zhang and Yang Feng. Information-transport-based policy for simultaneous trans- lation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors,Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 992–1013, Abu Dhabi, United...
2022 doi
-
[48]
Universal simultaneous machine translation with mixture-of- experts wait-k policy
Shaolei Zhang and Yang Feng. Universal simultaneous machine translation with mixture-of- experts wait-k policy. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7306–7317, Online and Punta Cana, Dominican Republic, Novem- ber 2021...
2021 doi
-
[49]
Librispeech: An asr corpus based on public domain audio books
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210, 2015. doi: 10.1109/ ICASSP.2015.7178964
2015
-
[50]
End-to-end simultaneous speech translation with differentiable segmentation
Shaolei Zhang and Yang Feng. End-to-end simultaneous speech translation with differentiable segmentation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Findings of the Association for Computational Linguistics: ACL 2023, pages 7659–7680, Toronto, Canada, July...
2023 doi
-
[51]
Making the v in vqa matter: Elevating the role of image understanding in visual question answering
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. 14
2017
-
[52]
Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition, 2022
Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, and Zhendong Peng. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition, 2022. URL https://arxiv.org/ abs/2110.03370
2022 arXiv
-
[53]
Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
2018
-
[54]
Hudson and Christopher D
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019
2019
-
[55]
Towards vqa models that can read
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019
2019
-
[56]
Learn to explain: Multimodal rea- soning via thought chains for science question answering
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal rea- soning via thought chains for science question answering. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, ...
2022
-
[57]
Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL https://arxiv. org/abs/2306.13394
2024 arXiv
-
[58]
Evaluating object hallucination in large vision-language models
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id= xozJw0kZXF
2023
-
[59]
Seed-bench: Benchmarking multimodal large language models
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13299– 13308, June 2024
2024
-
[60]
Mmbench: Is your multi-modal model an all-around player?, 2024
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024. URLhttps://arxiv.org/abs/2307.06281
2024 arXiv
-
[61]
Mm-vet: Evaluating large multimodal models for integrated capabilities,
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities,
-
[62]
Visual instruction tuning
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc., 2023. U...
2023
-
[63]
Semantic parsing on Freebase from question-answer pairs
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors,Proceedings of the 2013 Conference on Empirical Methods in Natural...
2013
-
[64]
Visit-bench: A dynamic benchmark for evaluating instruction-following vision-and-language models
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. Visit-bench: A dynamic benchmark for evaluating instruction-following vision-and-language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M...
2023
-
[65]
Spoken question answering and speech continuation using spectrogram-powered LLM
Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. Spoken question answering and speech continuation using spectrogram-powered LLM. InThe Twelfth International ...
2024
-
[66]
Improved baselines with visual instruction tuning
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, June 2024
2024
-
[67]
Introducing idefics: An open reproduction of state-of-the-art visual language model, 2023.URL https://huggingface
H Laurençon, Daniel van Strien, Stas Bekman, Leo Tronchon, Lucile Saulnier, Thomas Wang, Siddharth Karamcheti, Amanpreet Singh, Giada Pistilli, Yacine Jernite, et al. Introducing idefics: An open reproduction of state-of-the-art visual language model, 2023.URL https://huggingf...
2023
-
[68]
Textually pretrained speech language models
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Défossez, Gabriel Synnaeve, Emmanuel Dupoux, Roy Schwartz, and Yossi Adi. Textually pretrained speech language models. InThirty-seventh Conference on Neural Information Proce...
2023
-
[69]
BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Pr...
2023
-
[70]
The llama 3 herd of models, 2024
Meta. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783
2024 arXiv
-
[71]
Sigmoid loss for language image pre-training
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986, October 2023
2023
-
[72]
Llama-omni2: Llm- based real-time spoken chatbot with autoregressive streaming speech synthesis, 2025
Qingkai Fang, Yan Zhou, Shoutao Guo, Shaolei Zhang, and Yang Feng. Llama-omni2: Llm- based real-time spoken chatbot with autoregressive streaming speech synthesis, 2025. URL https://arxiv.org/abs/2505.02625
2025 arXiv
-
[73]
Anygpt: Unified multimodal llm with discrete sequence modeling, 2024
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yugang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal llm with discrete sequence modeling, 2024. URL https://arxiv...
2024 arXiv
-
[74]
Enhancing chat language models by scaling high-quality instructional conversations
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on E...
2023 doi
-
[75]
does not allow traveling to the second floor
Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li. Aishell-3: A multi-speaker mandarin tts corpus. InInterspeech 2021, pages 2756–2760, 2021. doi: 10.21437/Interspeech.2021-755. 16 A Construction of InstructOmni Existing publicly available text and vision instruction data are...
2021 doi
-
[77]
Scaling speech-text pre-training with synthetic interleaved data, 2024
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong, and Jie Tang. Scaling speech-text pre-training with synthetic interleaved data, 2024. URL https: //arxiv.org/abs/2411.17607
2024 arXiv
-
[2013]
URL https://aclanthology.org/ D13-1160/
Association for Computational Linguistics. URL https://aclanthology.org/ D13-1160/. 15
-
[2023]
URLhttps://arxiv.org/abs/2308.02490
-
[2024]
doi: 10.18653/v1/2024.emnlp-main.342
Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.342. URLhttps://aclanthology.org/2024.emnlp-main.342/
2024 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.