Pith. sign in

REVIEW 9 cited by

InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.09596 v1 pith:HFTS3HNI submitted 2024-12-12 cs.CV cs.AIcs.CL

Pan Zhang , Xiaoyi Dong , Yuhang Cao , Yuhang Zang , Rui Qian , Xilin Wei , Lin Chen , Yifei Li
show 21 more authors
Junbo Niu Shuangrui Ding Qipeng Guo Haodong Duan Xin Chen Han Lv Zheng Nie Min Zhang Bin Wang Wenwei Zhang Xinyue Zhang Jiaye Ge Wei Li Jingwen Li Zhongying Tu Conghui He Xingcheng Zhang Kai Chen Yu Qiao Dahua Lin Jiaqi Wang
This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords memoryreasoningstreaminglong-termmultimodalperceptionlongmodule
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Creating AI systems that can interact with environments over long periods, similar to human cognition, has been a longstanding research goal. Recent advancements in multimodal large language models (MLLMs) have made significant strides in open-world understanding. However, the challenge of continuous and simultaneous streaming perception, memory, and reasoning remains largely unexplored. Current MLLMs are constrained by their sequence-to-sequence architecture, which limits their ability to process inputs and generate responses simultaneously, akin to being unable to think while perceiving. Furthermore, relying on long contexts to store historical data is impractical for long-term interactions, as retaining all information becomes costly and inefficient. Therefore, rather than relying on a single foundation model to perform all functions, this project draws inspiration from the concept of the Specialized Generalist AI and introduces disentangled streaming perception, reasoning, and memory mechanisms, enabling real-time interaction with streaming video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive (IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module: Processes multimodal information in real-time, storing key details in memory and triggering reasoning in response to user queries. (2) Multi-modal Long Memory Module: Integrates short-term and long-term memory, compressing short-term memories into long-term ones for efficient retrieval and improved accuracy. (3) Reasoning Module: Responds to queries and executes reasoning tasks, coordinating with the perception and memory modules. This project simulates human-like cognition, enabling multimodal large language models to provide continuous and adaptive service over time.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory

    cs.CV 2026-07 conditional novelty 6.5 of 10

    Dual global+latent states with hierarchical episodic merging enable reflexive, low-latency long-video agents that beat iterative reasoning baselines on accuracy and efficiency.

  2. FOLIO: Focused Semantic Memory for Streaming Video Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Entity-centered focus-guided streaming memory lifts Qwen3-VL-8B to 82.0/69.1 Perception/Backward on OVO-Bench and 74.5 on StreamingBench while cutting writer tokens by ~32%.

  3. HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A hierarchical benchmark for multimodal models on human-centric visual understanding finds frontier models average under 60% and miss question-uncued visual evidence, with test-time scaling helping only marginally.

  4. HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Requiring omni-modal models to summarize context before reasoning, with LLM-judged context and logical rewards, improves human-intent reasoning benchmarks.

  5. SurgVLM: A Large Vision-Language Model and Systematic Evaluation Benchmark for Surgical Intelligence

    cs.CV 2025-06 conditional novelty 6.0 of 10

    SurgVLM, a family of surgical vision-language models trained on 1.81M frames and 7.79M conversations, outperforms 14 commercial VLMs on a six-dataset surgical benchmark.

  6. Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Codec-guided sparse patch selection plus a lightweight speak/silent gate yields a 4B streaming VLM that is competitive on static tasks, stronger on video/spatial benchmarks, and much cheaper at inference.

  7. X$^3$-OPD: Distilling Reasoning into Large Audio-Language Models via On-Policy Alignment

    cs.LG 2026-07 conditional novelty 5.0 of 10

    X3-OPD improves audio-grounded reasoning by training the audio student on its own rollouts with token-level teacher feedback, using a three-tier paired text-audio corpus.

  8. Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A new resolution-focused benchmark and an open-source native-resolution training framework show that preserving original image resolution improves VLM performance on fine-grained visual tasks.

  9. Know-MRI: A Knowledge Mechanisms Revealer&Interpreter for Large Language Models

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Know-MRI combines eleven existing LLM interpretation methods into one extensible toolkit with automatic input-to-method matching and dual UI and code interfaces.

Pith tools