Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Stream-Omni claims that mapping speech to text in the layer dimension, via CTC, transfers text capabilities to speech using only 23,000 hours of speech data.

desk verdict A coherent omni-modal architecture worth a serious look, but the printed streaming algorithm is broken and the core data-efficiency claim needs tighter evidence. read the letter →

arxiv 2506.13642 v2 pith:4KAVDJNR submitted 2025-06-16 cs.AI cs.CLcs.CVcs.SDeess.AS

classification cs.AIcs.CLcs.CVcs.SDeess.AS
keywords multimodallargelanguagemodelconnectionisttemporalclassificationspeech-textalignmentmodalitystreamingspeechgenerationdataefficiencyvision-language-speech
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Stream-Omni is a large multimodal model that handles text, vision, and speech in one system, and it argues that the three modalities should not be aligned the same way. Because speech is semantically consistent with text, Stream-Omni aligns speech to text through a CTC-based layer-dimension mapping, while vision, which is semantically complementary to text, is aligned by the usual sequence concatenation. The paper's central claim is that this deliberate separation makes modality alignment data-efficient, transferring the text capabilities of an 8B LLM backbone to speech with only 23,000 hours of speech data. A byproduct of the CTC mapping is that the model can stream intermediate text results, such as ASR transcriptions and the ongoing text response, while it is speaking. If correct, this points a way toward omni-modal assistants that are cheaper to train and more transparent to users than today's concatenation-based LMMs.

What carries the argument

The central object is the CTC-based layer-dimension speech-text mapping. CTC (Connectionist Temporal Classification) is a loss and decoding scheme that predicts a per-timestep distribution over a vocabulary and collapses repeated and blank tokens into a text sequence; here it is applied over the discrete speech-unit sequence, so each speech unit carries a distribution over the merged text-plus-speech vocabulary. This mechanism carries the argument in three ways: it gives direct supervised training signal for speech-to-text alignment in the bottom speech layers; its decoded alignment tells the model exactly which text token a speech unit corresponds to, enabling the alignment-based fusion in the top speech layers that streams speech generation behind the text generation (with a lag of K tokens); and it naturally yields intermediate ASR text as a byproduct. All three claimed advantages—data efficiency, streaming interaction, and intermediate text output—flow from this single mechanism.

What would settle it

Run Stream-Omni on speech with heavy disfluency, background noise, code-switching, or long pauses, and compare the streamed CTC transcript against the final text response and the audited speech; if the intermediate ASR output diverges from the final answer, or if speech generation skips or repeats text tokens, the monotonic-alignment assumption is violated. Concretely, a dataset designed so that a phrase's speech units and text tokens are non-monotonic (e.g., 'um, it's...') should expose the failure.

Watch

Extended reading notes

Core claim

Stream-Omni claims that speech and text share such close semantic structure that they should be mapped in the layer dimension, not merely concatenated in the sequence dimension. The paper adds three bottom speech layers and five top speech layers around the LLM backbone: the bottom layers convert discrete speech units (from the CosyVoice tokenizer) into text via connectionist temporal classification (CTC), giving a non-autoregressive speech-to-text path, and the top layers convert text back into speech units in a streaming, wait-k fashion. The CTC decoder does double duty: during inference it supplies the positional alignment that tells the model when a generated speech unit corresponds to a completed text token, which drives the alignment-based fusion in the top speech layers. The paper reports that this design supports strong visual understanding, speech interaction, and vision-grounded speech interaction, and that it can simultaneously emit ASR transcripts and text replies while speaking.

Load-bearing premise

The whole design hinges on CTC over discrete speech units producing a reliable, monotonic alignment between speech and text, and on that alignment being usable online to decide when a generated speech unit completes a text token; if that alignment is brittle on realistic speech, the streamed ASR output and the streaming speech generation degrade together.

Editorial extensions

If this is right

  • If the layer-dimension mapping holds up, an omni-modal assistant can be built on a text LLM backbone without re-learning textual knowledge from speech data.
  • Streaming intermediate text (ASR and the in-progress response) becomes a free byproduct of the speech-text alignment, so users can read along while the model talks.
  • Speech training data requirements could drop by an order of magnitude relative to concatenation-based omni-models, since the alignment supplies structure rather than relying on data-driven correlation.
  • Stream-Omni's reported visual benchmark results suggest that adding speech layers does not measurably degrade the text-vision capability of the underlying LLM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One testable extension is whether the CTC layer-dimension mapping transfers to other language pairs or to code-switched speech, where speech-text monotonicity is less clean.
  • The design implies a general recipe: modalities that share semantics with text (speech, perhaps video transcripts) can be mapped in the layer dimension, while modalities that add complementary information (images, structures) should stay in the sequence dimension.
  • The streaming ASR output could be repurposed as a lightweight always-on keyword detector or as a text-based control channel, since the model already computes the transcript during speech generation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. Stream-Omni proposes an 8B large language-vision-speech model in which vision is aligned with text via sequence-dimension concatenation and speech is aligned with text via a CTC-based layer-dimension mapping between bottom speech layers and an LLM backbone. The model is trained in three stages using roughly 23,000 hours of speech, including TTS-synthesized InstructOmni data, and supports text, vision, and speech inputs and outputs, with streaming ASR and simultaneous text generation during speech interaction. The paper reports vision benchmark scores, knowledge-based spoken QA results, a newly constructed SpokenVisIT benchmark, LibriSpeech ASR results, and an ablation of the alignment-based fusion mechanism.

Significance. If the central claims hold, Stream-Omni offers a credible path toward reducing speech training data for omni-modal LMMs and enabling simultaneous intermediate text outputs during speech interaction. The paper's strengths include releasing code and model weights, constructing and releasing datasets, and including a focused ablation of the fusion mechanism. The architecture's two-alignment strategy is clearly motivated, and the reported numbers are competitive with or better than several baselines. However, the significance is currently limited by an internally inconsistent streaming inference pseudocode and by evaluation evidence that does not yet isolate the claimed data-efficiency advantage.

major comments (3)
  1. [Algorithm 1 / §3.3] Algorithm 1 as printed cannot implement the streaming speech generation described in Section 3.3. Lines 4 and 9 index bY[-1] and bA[-1], bA[-2] after initializing both lists to empty, so the pseudocode would raise an IndexError on the first evaluation. More importantly, line 12 decodes CTCDec(F_bottom_speech(U)) with U the fixed input speech units from line 2; the generated units appended to bU in line 11 are never fed into the bottom speech layers or the CTC decoder. The inner loop condition therefore never changes: either it exits immediately or loops forever, and no speech is generated for any text token. This contradicts the prose in Section 3.3, which states that the CTC decoder identifies a new text token from the generated speech. Please correct the pseudocode so that the CTC decoder consumes the generated speech prefix (e.g., U concatenated with bU) and verify that the released implementation follows the corrected procedure.
  2. [§5.2–5.3, Tables 3–4] The central claim of data efficiency ('less data, especially speech') is not directly supported by the experiments. Tables 3 and 4 report single runs without error bars or significance tests, and the gaps are often small: Stream-Omni is 60.3 vs 59.7 average S→T against VITA-1.5 in Table 3, and 3.68 vs 3.45 on SpokenVisIT in Table 4. The comparison is also confounded by different training data, model sizes, and evaluation protocols. A matched ablation is needed: train a sequence-concatenation variant of Stream-Omni on the same 23K-hour corpus, or otherwise show that the CTC layer-dimension mapping, rather than the particular data mixture, drives the efficiency gain. Without this, the abstract's data-efficiency assertion remains plausible but unverified.
  3. [§5.3, Table 4, Appendix B] SpokenVisIT is a self-constructed benchmark: the spoken instructions are synthesized with CosyVoice, the same TTS system used to produce the model's training speech, and evaluation uses GPT-4o ratings on Whisper transcriptions with a provided image caption. This creates a domain-match advantage and an evaluation pipeline whose noise is not quantified. The paper should report a human-correlation study for the 1–5 ratings, include at least one established benchmark transformed to speech, and state how many instruction instances are evaluated after removing the eight math items. As it stands, Table 4 is too weak to support the 'superior real-world visual understanding' claim.
minor comments (6)
  1. [Table 1, Stage 2] The entry 'ASR (CTC Loss in Eq.(6))' should reference Eq. (4), since Eq. (6) is the fusion cross-attention formula, not the CTC loss.
  2. [Table 2] The average column appears to mix MME's 0–2000 scale with percentage-based benchmarks; please clarify how the Avg. is computed or exclude MME from the average.
  3. [Algorithm 1, line 14] If line 14 is taken literally, the speech decoder resynthesizes the entire bU sequence on every inner-loop iteration; please state whether synthesis is performed once after streaming or clarify that the line accumulates audio segments.
  4. [Table 6] The W=∞ row is not defined; please explain how full-context attention is realized in the alignment-based fusion module.
  5. [§3.1.2] There are minor grammatical issues, e.g., 'the bottom speech layersF_bottom_speech(·)maps' should be 'the bottom speech layers F_bottom_speech(·) map'.
  6. [§4.3] The configuration section reports the GPU count but not training hyperparameters such as batch size, learning rate, number of steps, or sequence length; adding these would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the core CTC alignment and text-to-speech transfer claims are trained on real ASR data and evaluated on external benchmarks, and the CosyVoice TTS/benchmark overlap is a distributional concern rather than a definitional reduction.

full rationale

Stream-Omni's central derivation is not circular. The speech-to-text mapping is trained with a standard CTC loss (Eq. 4) on real ASR corpora (LibriSpeech, WenetSpeech) plus TTS data, and the mapping quality is evaluated on the external LibriSpeech ASR benchmark (Table 5), not on the model's own outputs. The knowledge-transfer claim is tested on spoken QA benchmarks against external baselines, so it does not reduce to a fitted parameter or to the model's training signal. The use of CosyVoice both as tokenizer/decoder and as the TTS for InstructOmni/SpokenVisIT is a distributional-overlap concern: training and evaluation speech share the same synthesizer, which can inflate speech-interaction scores, but this is a data-construction issue rather than a 'prediction equals input by construction' pattern. Self-citations to Stream-Speech [43] supply the CTC-alignment idea, but that is a published, externally reviewed result, and the present paper's empirical validation is independent. One non-circularity issue should be flagged separately: Algorithm 1 line 12 decodes CTCDec(F_bottom_speech(U)) from the fixed input U rather than from the growing generated sequence bU, so the printed streaming termination condition is decoupled from the generated speech and cannot work as described; this is an internal correctness/consistency bug, not a circular derivation.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper adds modest free hyperparameters (layer counts, fusion window, lagging tokens) and relies on standard CTC and pretrained components. No new physical entities are introduced. The main unproven premise is that synthetic TTS data and monotonic CTC alignment are faithful enough to transfer text knowledge to speech.

free parameters (4)
  • Bottom speech layer count = 3
    Architecture hyperparameter chosen by the authors for speech-to-text mapping; no sweep reported.
  • Top speech layer count = 5
    Architecture hyperparameter for text-to-speech mapping; no sweep reported.
  • Fusion window size W = 5
    Cross-attention window over aligned text tokens; ablation shows W=5 is best among tested values but it is still a hand-selected hyperparameter.
  • Lagging text tokens K = 3
    Wait-k lag before streaming speech generation begins; fixed value with no latency-quality sweep reported.
assumptions (5)
  • domain assumption CTC over discrete speech units produces a valid monotonic alignment between speech and text.
    Invoked in Eq. (3)-(4) and during streaming inference in Algorithm 1; if CTC alignment is unreliable, both ASR and fusion degrade.
  • domain assumption Speech and text are semantically consistent, while vision is semantically complementary to text.
    Used to justify choosing sequence concatenation for vision and layer-dimension mapping for speech in Sections 1 and 3.
  • domain assumption TTS-synthesized speech from text instructions is representative of real speech interaction data.
    InstructOmni is created with CosyVoice TTS and used for Stages 2 and 3; if synthetic speech differs from real user speech, data-efficiency results may not transfer.
  • domain assumption Pretrained components (SigLIP vision encoder, LLaMA-3.1 backbone, CosyVoice tokenizer and decoder) provide sufficient representations.
    The model relies entirely on these pretrained components without fine-tuning them, assuming their representations are adequate for the new alignment tasks.
  • domain assumption Ground-truth transcriptions used in Stage 2 training of top speech layers are close enough to LLM-generated text at inference.
    Section 3.2.2 states that text representations fed to top speech layers during training come from ground-truth transcriptions, while inference uses LLM outputs; a mismatch could degrade streaming speech generation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model." pith.science (2026). https://pith.science/paper/4KAVDJNR

@misc{pith2026250613642,
  author       = {Pith},
  title        = {Pith review of: Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4KAVDJNR}},
  note         = {Machine review of arXiv:2506.13642}
}
read the original abstract

The emergence of GPT-4o-like large multimodal models (LMMs) has raised the exploration of integrating text, vision, and speech modalities to support more flexible multimodal interaction. Existing LMMs typically concatenate representation of modalities along the sequence dimension and feed them into a large language model (LLM) backbone. While sequence-dimension concatenation is straightforward for modality integration, it often relies heavily on large-scale data to learn modality alignments. In this paper, we aim to model the relationships between modalities more purposefully, thereby achieving more efficient and flexible modality alignments. To this end, we propose Stream-Omni, a large language-vision-speech model with efficient modality alignments, which can simultaneously support interactions under various modality combinations. Stream-Omni employs LLM as the backbone and aligns the vision and speech to the text based on their relationships. For vision that is semantically complementary to text, Stream-Omni uses sequence-dimension concatenation to achieve vision-text alignment. For speech that is semantically consistent with text, Stream-Omni introduces a CTC-based layer-dimension mapping to achieve speech-text alignment. In this way, Stream-Omni can achieve modality alignments with less data (especially speech), enabling the transfer of text capabilities to other modalities. Experiments on various benchmarks demonstrate that Stream-Omni achieves strong performance on visual understanding, speech interaction, and vision-grounded speech interaction tasks. Owing to the layer-dimensional mapping, Stream-Omni can simultaneously provide intermediate text outputs (such as ASR transcriptions and model responses) during speech interaction, offering users a comprehensive multimodal experience.

Figures

Figures reproduced from arXiv: 2506.13642 by the authors.

Figure 1
Figure 1. Comparison of modality alignments in Stream-Omni and previous works. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Architecture of Stream-Omni. Right: Interactions under various modality combinations. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Diagram of top speech layers. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Case Study of Stream-Omni (detail understanding). [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Case Study of Stream-Omni (long response). [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory

    cs.CV 2026-07 conditional novelty 6.5 of 10

    Dual global+latent states with hierarchical episodic merging enable reflexive, low-latency long-video agents that beat iterative reasoning baselines on accuracy and efficiency.

Reference graph

Works this paper leans on

79 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024. URLhttps://openai.com/index/hello-gpt-4o/

  2. [2]

    Gpt-4v(ision) system card, 2024

    OpenAI. Gpt-4v(ision) system card, 2024. URL https://cdn.openai.com/papers/ GPTV_System_Card.pdf

  3. [3]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc., 2023. 10 URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 6dcf277ea32ce3...

  4. [4]

    MiniGPT-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum?id=1tZbq88f27

  5. [5]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https: //llava-vl.github.io/blog/2024-01-30-llava-next/

  6. [6]

    Llava-onevision: Easy visual task transfer, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. URL https://arxiv.org/abs/2408.03326

  7. [7]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

  8. [8]

    Mini-omni: Language models can hear, talk while thinking in streaming, 2024

    Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming, 2024. URLhttps://arxiv.org/abs/2408.16725

Show all 79 references
  1. [9]

    LLaMA- omni: Seamless speech interaction with large language models

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. LLaMA- omni: Seamless speech interaction with large language models. InThe Thirteenth Interna- tional Conference on Learning Representations, 2025. URL https://openreview.net/ forum?id=PYmrUQmMEw

  2. [10]

    Moshi: a speech-text foundation model for real-time dialogue, 2024

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue, 2024. URLhttps://arxiv.org/abs/2410.00037

  3. [11]

    Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot, 2024

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot, 2024. URLhttps://arxiv.org/abs/2412.02612

  4. [12]

    Vita-1.5: Towards gpt-4o level real-time vision and speech interaction, 2025

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, Long Ma, Xiawu Zheng, Rongrong Ji, Xing Sun, Caifeng Shan, and Ran He. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction, 2025. URLhttps:/...

  5. [13]

    Baichuan-omni technical report, 2024

    Yadong Li, Haoze Sun, Mingan Lin, Tianpeng Li, Guosheng Dong, Tao Zhang, Bowen Ding, Wei Song, Zhenglin Cheng, Yuqi Huo, Song Chen, Xu Li, Da Pan, Shusen Zhang, Xin Wu, Zheng Liang, Jun Liu, Tao Zhang, Keer Lu, Yaqi Zhao, Yanjun Shen, Fan Yang, Kaicheng Yu, Tao Lin, Jianhua Xu...

  6. [14]

    Qwen2.5-omni technical report, 2025

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report, 2025. URLhttps://arxiv.org/abs/2503.20215

  7. [15]

    Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks

    Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. InPro- ceedings of the 23rd International Conference on Machine Learning, ICML ’06, page 369–37...

  8. [16]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila an...

  9. [17]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023. URL https://arxiv.org/abs/2308. 12966

  10. [18]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qin- glong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedi...

  11. [19]

    Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models, 2023

    Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, and Yu Qiao. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large...

  12. [20]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CV...

  13. [21]

    Internvideo: General video foundation models via generative and discriminative learning, 2022

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative lear...

  14. [22]

    Videochat: Chat-centric video understanding, 2024

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024. URL https: //arxiv.org/abs/2305.06355

  15. [23]

    Video-ChatGPT: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the Association for Comput...

  16. [24]

    Llama-vid: An image is worth 2 tokens in large language models, 2023

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models, 2023. URLhttps://arxiv.org/abs/2311.17043

  17. [25]

    Video-LLaV A: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaV A: Learning united visual representation by alignment before projection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in...

  18. [26]

    Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm, 2024

    Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm, 2024. URLhttps://arxiv.org/abs/2411.00774

  19. [27]

    Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation, 2024

    Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, and Chao Zhang. Salmonn-omni: A codec-free llm for full-duplex speech understanding and generation, 2024. URLhttps://arxiv.org/abs/2411.18138

  20. [28]

    Slam-omni: Timbre-controllable voice interaction system with single-stage training,

    Wenxi Chen, Ziyang Ma, Ruiqi Yan, Yuzhe Liang, Xiquan Li, Ruiyang Xu, Zhikang Niu, Yanqiao Zhu, Yifan Yang, Zhanxun Liu, Kai Yu, Yuxuan Hu, Jinyu Li, Yan Lu, Shujie Liu, and Xie Chen. Slam-omni: Timbre-controllable voice interaction system with single-stage training,

  21. [29]

    Robust speech recognition via large-scale weak supervision, 2022

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. 12

  22. [30]

    URLhttps://arxiv.org/abs/2412.15649

  23. [31]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units.IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3...

  24. [32]

    SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Findings of the Association for Computat...

  25. [33]

    Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens, 2024

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijie Yan. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens, 2024. URL https: //arxiv.org/ab...

  26. [34]

    Speechtokenizer: Uni- fied speech tokenizer for speech language models

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Uni- fied speech tokenizer for speech language models. InThe Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id= AF9Q8Vip84

  27. [35]

    Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoît Sagot, and Emmanuel Dupoux. SpiRit-LM: Inter...

  28. [36]

    Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In H. Larochelle, M. Ran- zato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Information Processing Systems, volume 33, ...

  29. [37]

    M2-omni: Advancing omni-mllm for comprehensive modality support with competitive performance, 2025

    Qingpei Guo, Kaiyou Song, Zipeng Feng, Ziping Ma, Qinglong Zhang, Sirui Gao, Xuzheng Yu, Yunxiao Sun, Tai-Wei Chang, Jingdong Chen, Ming Yang, and Jun Zhou. M2-omni: Advancing omni-mllm for comprehensive modality support with competitive performance, 2025. URL https://arxiv.or...

  30. [38]

    Megrez-omni technical report, 2025

    Boxun Li, Yadong Li, Zhiyuan Li, Congyi Liu, Weilin Liu, Guowei Niu, Zheyue Tan, Haiyang Xu, Zhuyu Yao, Tao Yuan, Dong Zhou, Yueqing Zhuang, Shengen Yan, Guohao Dai, and Yu Wang. Megrez-omni technical report, 2025. URL https://arxiv.org/abs/2502. 15803

  31. [39]

    Kwok, Hengshuang Zhao, Xiaodan Liang, Dit-Yan Yeung, Xiao Chen, Zhenguo Li, Wei Zhang, Qun Liu, Jun Yao, Lanqing Hong, Lu Hou, and Hang Xu

    Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, Dingdong Wang, Kun Xiang, Haoyuan Li, Haoli Bai, Jianhua Han, Xiaohui Li, Weike Jin, Nian Xie, Yu Zhang, James T. Kwok, Hengshuang Zhao, Xiaodan Liang, Dit-Yan Yeung,...

  32. [40]

    Capybara-omni: An efficient paradigm for building omni-modal language models, 2025

    Xingguang Ji, Jiakang Wang, Hongzhi Zhang, Jingyuan Zhang, Haonan Zhou, Chenxi Sun, Yahui Liu, Qi Wang, and Fuzheng Zhang. Capybara-omni: An efficient paradigm for building omni-modal language models, 2025. URLhttps://arxiv.org/abs/2504.12315

  33. [41]

    Omni-emotion: Extending video mllm with detailed face and audio modeling for multimodal emotion analysis, 2025

    Qize Yang, Detao Bai, Yi-Xing Peng, and Xihan Wei. Omni-emotion: Extending video mllm with detailed face and audio modeling for multimodal emotion analysis, 2025. URL https://arxiv.org/abs/2501.09502

  34. [42]

    Openomni: Advancing open-source omnimodal large language models with progressive multimodal alignment and real-time self-aware emotional speech synthesis, 2025

    Run Luo, Ting-En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, Yangyi Chen, Xiaobo Xia, Hamid Alinejad-Rokny, and Fei Huang. Openomni: Advancing open-source omnimodal large language models with progressive multimodal alignm...

  35. [43]

    Stream- Speech: Simultaneous speech-to-speech translation with multi-task learning

    Shaolei Zhang, Qingkai Fang, Shoutao Guo, Zhengrui Ma, Min Zhang, and Yang Feng. Stream- Speech: Simultaneous speech-to-speech translation with multi-task learning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of the As- soci...

  36. [44]

    LLaV A-mini: Efficient image and video large multimodal models with one vision token

    Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. LLaV A-mini: Efficient image and video large multimodal models with one vision token. InThe Thirteenth International Con- ference on Learning Representations, 2025. URL https://openreview.net/forum? id=UQJ7CDW8nb

  37. [45]

    Future-guided incremental transformer for si- multaneous translation.Proceedings of the AAAI Conference on Artificial Intelligence, 35 (16):14428–14436, May 2021

    Shaolei Zhang, Yang Feng, and Liangyou Li. Future-guided incremental transformer for si- multaneous translation.Proceedings of the AAAI Conference on Artificial Intelligence, 35 (16):14428–14436, May 2021. URL https://ojs.aaai.org/index.php/AAAI/ article/view/17696

  38. [46]

    STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework

    Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework. InPro...

  39. [47]

    Information-transport-based policy for simultaneous trans- lation

    Shaolei Zhang and Yang Feng. Information-transport-based policy for simultaneous trans- lation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors,Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 992–1013, Abu Dhabi, United...

  40. [48]

    Universal simultaneous machine translation with mixture-of- experts wait-k policy

    Shaolei Zhang and Yang Feng. Universal simultaneous machine translation with mixture-of- experts wait-k policy. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7306–7317, Online and Punta Cana, Dominican Republic, Novem- ber 2021...

  41. [49]

    Librispeech: An asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210, 2015. doi: 10.1109/ ICASSP.2015.7178964

  42. [50]

    End-to-end simultaneous speech translation with differentiable segmentation

    Shaolei Zhang and Yang Feng. End-to-end simultaneous speech translation with differentiable segmentation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Findings of the Association for Computational Linguistics: ACL 2023, pages 7659–7680, Toronto, Canada, July...

  43. [51]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. 14

  44. [52]

    Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition, 2022

    Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang, Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, Di Wu, and Zhendong Peng. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition, 2022. URL https://arxiv.org/ abs/2110.03370

  45. [53]

    Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P

    Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

  46. [54]

    Hudson and Christopher D

    Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019

  47. [55]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019

  48. [56]

    Learn to explain: Multimodal rea- soning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal rea- soning via thought chains for science question answering. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, ...

  49. [57]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL https://arxiv. org/abs/2306.13394

  50. [58]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/forum?id= xozJw0kZXF

  51. [59]

    Seed-bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13299– 13308, June 2024

  52. [60]

    Mmbench: Is your multi-modal model an all-around player?, 2024

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024. URLhttps://arxiv.org/abs/2307.06281

  53. [61]

    Mm-vet: Evaluating large multimodal models for integrated capabilities,

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities,

  54. [62]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors,Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc., 2023. U...

  55. [63]

    Semantic parsing on Freebase from question-answer pairs

    Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors,Proceedings of the 2013 Conference on Empirical Methods in Natural...

  56. [64]

    Visit-bench: A dynamic benchmark for evaluating instruction-following vision-and-language models

    Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. Visit-bench: A dynamic benchmark for evaluating instruction-following vision-and-language models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M...

  57. [65]

    Spoken question answering and speech continuation using spectrogram-powered LLM

    Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. Spoken question answering and speech continuation using spectrogram-powered LLM. InThe Twelfth International ...

  58. [66]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296–26306, June 2024

  59. [67]

    Introducing idefics: An open reproduction of state-of-the-art visual language model, 2023.URL https://huggingface

    H Laurençon, Daniel van Strien, Stas Bekman, Leo Tronchon, Lucile Saulnier, Thomas Wang, Siddharth Karamcheti, Amanpreet Singh, Giada Pistilli, Yacine Jernite, et al. Introducing idefics: An open reproduction of state-of-the-art visual language model, 2023.URL https://huggingf...

  60. [68]

    Textually pretrained speech language models

    Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Défossez, Gabriel Synnaeve, Emmanuel Dupoux, Roy Schwartz, and Yossi Adi. Textually pretrained speech language models. InThirty-seventh Conference on Neural Information Proce...

  61. [69]

    BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Pr...

  62. [70]

    The llama 3 herd of models, 2024

    Meta. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783

  63. [71]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986, October 2023

  64. [72]

    Llama-omni2: Llm- based real-time spoken chatbot with autoregressive streaming speech synthesis, 2025

    Qingkai Fang, Yan Zhou, Shoutao Guo, Shaolei Zhang, and Yang Feng. Llama-omni2: Llm- based real-time spoken chatbot with autoregressive streaming speech synthesis, 2025. URL https://arxiv.org/abs/2505.02625

  65. [73]

    Anygpt: Unified multimodal llm with discrete sequence modeling, 2024

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yugang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal llm with discrete sequence modeling, 2024. URL https://arxiv...

  66. [74]

    Enhancing chat language models by scaling high-quality instructional conversations

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on E...

  67. [75]

    does not allow traveling to the second floor

    Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li. Aishell-3: A multi-speaker mandarin tts corpus. InInterspeech 2021, pages 2756–2760, 2021. doi: 10.21437/Interspeech.2021-755. 16 A Construction of InstructOmni Existing publicly available text and vision instruction data are...

  68. [77]

    Scaling speech-text pre-training with synthetic interleaved data, 2024

    Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong, and Jie Tang. Scaling speech-text pre-training with synthetic interleaved data, 2024. URL https: //arxiv.org/abs/2411.17607

  69. [2013]

    URL https://aclanthology.org/ D13-1160/

    Association for Computational Linguistics. URL https://aclanthology.org/ D13-1160/. 15

  70. [2023]

    URLhttps://arxiv.org/abs/2308.02490

  71. [2024]

    doi: 10.18653/v1/2024.emnlp-main.342

    Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.342. URLhttps://aclanthology.org/2024.emnlp-main.342/

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.