Pith. sign in

REVIEW 2 major objections 4 minor 3 cited by

LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale

T0 review · 2 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A video LLM trained on timestamp-aligned speech transcripts can deliver real-time commentary and match much larger models on video QA.

desk verdict Solid systems paper: timestamp-dense ASR interleaving works for QA and enables real-time commentary, but the headline commentary win-rate is measured against the model's own training target, so read that claim with caution. read the letter →

arxiv 2504.16030 v1 pith:X345NTZD submitted 2025-04-22 cs.CV

Joya Chen , Ziyun Zeng , Yiqi Lin , Wei Li , Zejun Ma , Mike Zheng Shou This is my paper · ORCID
classification cs.CV
keywords streamingvideoLLMASRpre-trainingreal-timecommentarytimestampinterleavingclosedcaptionsQALiveSports-3Klow-latencyinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that timestamp-aligned automatic speech recognition (ASR) transcripts—the closed captions that already exist for millions of YouTube videos—are enough to train a video large language model, without the expensive human- or GPT-crafted annotations usually required. The training scheme interleaves each short block of ASR words with the video frames that were on screen while the words were spoken, so the model learns to generate the few words of commentary current for each frame. On five million YouTube clips this yields a 7B model that, even before instruction tuning, can produce real-time play-by-play commentary, and after tuning outperforms 72B models on the commentary benchmark while reaching state-of-the-art 7B/8B results on standard video QA. The paper also contributes the LiveSports-3K benchmark for measuring free-form streaming commentary and temporal Who/When/What question answering.

What carries the argument

The load-bearing mechanism is a training sequence that interleaves visual and textual tokens at frame-level granularity: $$[\text{Con}]\langle F_{t:t+k}\rangle \langle W_{t:t+k}\rangle \langle F_{t+k:t+2k}\rangle \langle W_{t+k:t+2k}\rangle \ldots$$ where $\langle F\rangle$ holds the visual tokens of one or more frames, $\langle W\rangle$ holds the ASR words assigned to the same time interval, and $[\text{Con}]$ is the video title, previous ASR text, or user prompt. At 2 FPS with $k=1$ second, the model predicts only the two seconds' worth of words per frame block, with an ellipsis token marking pauses, which is what makes low-latency streaming inference possible. This interleaving is the object that lets the model bind spoken language to specific visual moments rather than to the whole clip.

What would settle it

A decisive test would be a human rating study on a sample of LiveSports-3K clips where raters, seeing only the video and the candidate commentaries, rank LiveCC-7B-Instruct against Qwen2.5-VL-72B-Instruct and LLaVA-Video-72B; if human preferences diverge from the GPT-4o judge's win rates, the commentary-quality claim loses its support. A second test—shuffling word-to-frame timestamps during pre-training—would isolate whether the dense temporal interleaving itself is the cause of the gains.

Watch

Extended reading notes

Core claim

The central discovery is that a streaming task—predicting the few spoken words that align with each pair of video frames—is a powerful and scalable pre-training objective for video LLMs. Dense timestamp-level interleaving of ASR words and frames, rather than treating the transcript as one global caption, is what teaches temporally fine-grained vision-language alignment, produces a genuinely new real-time commentary ability, and transfers to general video question answering. The resulting LiveCC-7B-Instruct, initialized from a 7B vision-language base and trained on Live-CC-5M plus instruction data, is reported to surpass 72B models in commentary quality while running at sub-half-second latency per frame, and to lead the 7B/8B class on VideoMME and OVOBench.

Load-bearing premise

The headline result rests on trusting GPT-4o's judgment that a commentary is good because it resembles the ground-truth speech transcript in meaning and style, a metric that may favor a model trained to imitate exactly that style.

Editorial extensions

If this is right

  • Live sports commentary and other frame-by-frame narration can be produced by a 7B model with sub-0.5-second latency per frame, making real-time deployment at 2 FPS plausible.
  • ASR and closed captions can substitute for expensive proprietary annotations during video pre-training, reducing data cost and allowing the training set to scale to millions of clips.
  • Streaming ASR pre-training transfers to general video QA: the ASR-only base model is already competitive, and the SFT model reaches state-of-the-art results at the 7B/8B scale on VideoMME and OVOBench.
  • The per-frame ellipsis/pause token lets the model distinguish temporary pauses from the true end of a response, a necessary piece for streaming generation.
  • The LiveSports-3K benchmark gives the community a reusable way to measure free-form streaming commentary alongside Who/When/What question answering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether the LiveSports-3K-CC win-rate metric tracks human judgment; if GPT-4o's preference is driven by stylistic mimicry of ASR output, the reported margin over 72B models may not transfer to human listeners. A human rating study would settle this.
  • The paper observes QA performance degrading beyond 5M pre-training clips, which suggests single-source streaming ASR eventually saturates general understanding; mixing streaming ASR with captioning or QA objectives during pre-training is a natural extension it defers to future work.
  • The same interleaving machinery could be pointed at any visually grounded narration stream—tutorials, weather reports, live events—and the title-plus-previous-ASR context design hints at a text-memory architecture in which visual tokens are dropped periodically while text context persists, supporting arbitrarily long streams.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes LiveCC, a video large language model trained by densely interleaving timestamp-aligned ASR word transcripts with video frames in a streaming, autoregressive fashion. The authors construct a 5M-clip pretraining dataset (Live-CC-5M) from YouTube closed captions and a 526K-clip SFT dataset (Live-WhisperX-526K) using WhisperX, with filtering for visual grounding and talking-head removal. They introduce LiveSports-3K, a sports benchmark with a commentary win-rate track (LiveSports-3K-CC) and a QA track (LiveSports-3K-QA), and they report that LiveCC-7B-Instruct surpasses 72B models (Qwen2.5-VL-72B, LLaVA-Video-72B) in commentary quality and achieves state-of-the-art results at 7B/8B scale on VideoMME and OVOBench. The paper also releases all resources.

Significance. If the central claims hold, the contribution is substantial: a low-cost, scalable training paradigm that uses freely available ASR transcripts to imbue video LLMs with real-time commentary ability and improves general video QA. The paper is commendably thorough in its data pipeline design, ablations, and resource release; the ablations in Tables 1 and 2 are informative, and the OVOBench gain (59.8 vs. 50.4 for Qwen2-VL-7B) is a concrete, falsifiable improvement. However, the headline commentary-superiority claim rests entirely on a new benchmark whose metric is aligned with the training objective, which undermines the claim unless the metric is independently validated.

major comments (2)
  1. [Sections 4.2, 5.1, Supplementary 8.2] The LiveSports-3K-CC win-rate metric is not a validated measure of commentary quality. GPT-4o serves both as the fixed baseline opponent and as the judge, and the judge is instructed to prefer the response that best matches the ground-truth ASR transcript in semantic and stylistic alignment. LiveCC is explicitly trained to predict dense timestamp-aligned ASR word sequences and to emit an ellipsis for silent frames (Section 3.2), so the metric largely measures fidelity to the ASR reference rather than an independent notion of commentary quality. The abstract's claim that LiveCC-7B-Instruct surpasses 72B models in commentary quality therefore rests on a metric that is aligned with the training objective. Without a human-preference study or a correlation analysis showing that the judge's choices track human judgments of commentary quality, this central claim is not established. I recommend adding such validation (e.g., a 100-example human pairwise study with agreement statistics) or reframing the claim as 'better ASR-transcript alignment' rather than 'better commentary quality.'
  2. [Table 4 and Supplementary Section 8.2] Because GPT-4o is both the judge and the fixed opponent, the win rates may also be affected by self-preference bias. The paper states that each prompt is evaluated twice with positions swapped, but it does not report the judge's agreement rate or how ties are resolved. I ask the authors to report these statistics, and to include at least one independent judge (e.g., a different LLM or human annotators) on a subset to confirm that the ranking is not an artifact of judge self-preference. This is particularly important because the win rates are the only evidence for the headline commentary claim.
minor comments (4)
  1. [Throughout] Several typos and inconsistencies should be fixed: 'escepically' in Section 1, 'Implemetation' in Sections 5.1 and 8, 'removel' in Figure 4b, and the inconsistent spelling 'Live-Whisper-526K' versus 'Live-WhisperX-526K'.
  2. [Figure 2] The data pipeline diagram is dense and the flow is hard to follow; consider splitting it into two sub-figures (pretraining and SFT) or annotating the filtering thresholds directly on the edges.
  3. [Section 5.1] The statement that 'for all models, we calculate the logits of multiple choices to select answers' is ambiguous for API-based models like GPT-4o, for which logits may not be accessible; please clarify whether generation-based evaluation is used for those models and how the logit method is applied consistently.
  4. [Table 3] The OVOBench column abbreviations (RTVP, BT, FAR) are not expanded in the caption or in the text; a footnote or definition would improve readability.

Circularity Check

1 steps flagged · score 6.0 of 10

LiveSports-3K-CC defines commentary quality as alignment with the ASR transcript, which is exactly LiveCC's training target; the headline commentary-superiority claim is therefore partially circular, while the QA results remain independent.

  1. fitted input called prediction [Sections 3.2, 4.2, 5.1; Supplementary 8.2; Table 4]
    "we developed this track to assess video comprehension by evaluating the alignment between model-generated and groundtruth CCs from ASR. ... We use GPT-4o as a judge to select the better prediction based on the ground-truth CCs. The selection criteria include both stylistic and semantic consistency. (Sec. 4.2) The model is trained to generate frame assigned ASR words in an autoregressive manner. (Sec. 3.2) ... select the one that better aligns with the human commentary. ... Semantic Alignment: The commentary should convey the same meaning, details, and key points as the human commentary. ..."

    The benchmark's ground truth is the ASR transcript, and the judge is instructed to prefer the commentary that best matches that transcript semantically and stylistically. LiveCC is trained to autoregressively predict exactly those frame-aligned ASR words, predicting an ellipsis token for silence. The paper itself describes the evaluation task as 'to complete the ASR text based on the given video clip' (Sec. 5.1), and notes that Qwen models fail because they 'tend to simply caption the video rather than follow the preceding ASR context to continue the video commentary' (Table 4 caption). Thus the win-rate metric measures proximity to the model's own training target; a fluent descriptive caption loses by construction to ASR-mimicking output.

full rationale

LiveCC is an empirical systems paper, not a formal derivation. Most of its pipeline (data curation, filtering, training, streaming inference, latency) is self-contained and externally checkable. The QA gains on VideoMME, MVBench, OVOBench, and LiveSports-QA are evaluated on external or independently constructed benchmarks with held-out videos removed from training, so those results support the generalizability claim independently. The commentary-superiority claim, however, is not independently supported. The only evidence for 'surpass advanced 72B models ... in commentary quality' is the LiveSports-3K-CC win rate. That benchmark's ground truth is the ASR closed caption; the judge is explicitly instructed to prefer the response that best matches the ASR transcript semantically and stylistically; and LiveCC is trained to autoregressively predict exactly those frame-aligned ASR words on millions of YouTube closed captions. The evaluation task is described by the paper itself as 'to complete the ASR text based on the given video clip.' Hence the metric operationalizes 'quality' as success at the training objective: a model that imitates ASR fragmentary style is rewarded even if a human would prefer a fluent, informative caption. The Table 4 note that Qwen models 'tend to simply caption the video rather than follow the preceding ASR context' confirms that the benchmark rewards transcript continuation, not an independent notion of commentary quality. This is a partial, metric-level circularity: the central headline claim reduces, via the benchmark definition, to the training target. It is not a self-citation chain or a formal derivation, and the QA results remain valid external evidence.

Assumptions & free parameters 15 free parameters · 5 assumptions · 0 invented entities

The ledger lists the hand-chosen filtering thresholds and training hyperparameters that define the dataset and model; none are fitted to optimize a benchmark, but they shape the data distribution. The axioms capture the unproven dependencies: visual grounding of ASR, timestamp accuracy from the uniform-distribution heuristic, the validity of the GPT-4o judge, assumed absence of benchmark contamination, and the suitability of the base model.

free parameters (15)
  • frame_rate_FPS = 2
    Default frame sampling rate for interleaving (Section 3.2).
  • interleaving_interval_k_seconds = 1
    Time interval covered by each frame group and associated words (Section 3.2).
  • pretrain_clip_length = 30-240s
    Clipping range for pre-training; max 240s default, 60s in ablations (Section 3.1, A1).
  • max_word_gap_for_split = 3s
    If gap between ASR words exceeds 3 seconds, a new clip is generated (Section 3.1, A1).
  • word_speed_range = 1 to 4 words per second
    Clips with speech rate outside this range are discarded (Section 3.1, A1).
  • english_detection_confidence = 0.9
    XLM-R confidence threshold for English detection (Section 3.1).
  • min_distinct_words = 30
    Video must have at least 30 distinct words in CC (Section 3.1).
  • text_loss_range_pretrain = [1.5, 6.5]
    Perplexity filter using Qwen2-1.5B loss on transcripts, pre-training (Section 3.1, A2).
  • text_loss_range_sft = [1.5, 5.0]
    Perplexity filter range for SFT data (Section 3.1, B4).
  • talking_head_confidence_threshold = 0.3
    Keep videos if Qwen-VL confidence in talking-head content is below 0.3 (Section 3.1, A3).
  • asd_ratio_threshold = 0.05
    Remove clips with active-speaker detection ratio above 0.05 (Section 3.1, B5).
  • max_frames_training = 480 formal, 120 ablation
    Maximum number of visual frames during training and evaluation (Section 5.1, Table 4 note).
  • learning_rate = 2e-5 pretraining, 1e-5 SFT
    Training hyperparameters (Section 5.1).
  • batch_size = 512 on 128 GPUs
    Training hyperparameter (Section 5.1).
  • visual_context_tokens = 16K ablation, 24K formal
    Visual context length limit (Section 5.1).
assumptions (5)
  • domain assumption ASR transcripts are visually grounded in the retained videos.
    The pipeline filters by text perplexity and removes talking-head clips, but the model relies on the remaining transcripts describing visual content (Section 3.1).
  • domain assumption Heuristic word-level timestamps from YouTube CC are accurate enough for frame-word alignment.
    For pre-training, each CC segment's duration is uniformly distributed across its words (Section 3.2), an approximation that may misalign words and frames.
  • domain assumption GPT-4o is a reliable judge of commentary quality in the LiveSports-3K-CC setup.
    The win-rate metric asks GPT-4o to compare outputs against ASR ground truth in style and semantics (Sections 4.2, 5.1).
  • domain assumption The training pool does not overlap with the evaluation benchmarks.
    The paper only removes LiveSports-3K videos from training; no dedup is reported against VideoMME, MVBench, or OVOBench, and training sources include large YouTube archives (Sections 3.1, 5.3).
  • domain assumption Qwen2-VL-7B is a suitable backbone that can be extended to streaming text prediction.
    The method builds on Qwen2-VL and inherits its vision and language capacities (Section 3.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale." pith.science (2026). https://pith.science/paper/X345NTZD

@misc{pith2026250416030,
  author       = {Pith},
  title        = {Pith review of: LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X345NTZD}},
  note         = {Machine review of arXiv:2504.16030}
}
read the original abstract

Recent video large language models (Video LLMs) often depend on costly human annotations or proprietary model APIs (e.g., GPT-4o) to produce training data, which limits their training at scale. In this paper, we explore large-scale training for Video LLM with cheap automatic speech recognition (ASR) transcripts. Specifically, we propose a novel streaming training approach that densely interleaves the ASR words and video frames according to their timestamps. Compared to previous studies in vision-language representation with ASR, our method naturally fits the streaming characteristics of ASR, thus enabling the model to learn temporally-aligned, fine-grained vision-language modeling. To support the training algorithm, we introduce a data production pipeline to process YouTube videos and their closed captions (CC, same as ASR), resulting in Live-CC-5M dataset for pre-training and Live-WhisperX-526K dataset for high-quality supervised fine-tuning (SFT). Remarkably, even without SFT, the ASR-only pre-trained LiveCC-7B-Base model demonstrates competitive general video QA performance and exhibits a new capability in real-time video commentary. To evaluate this, we carefully design a new LiveSports-3K benchmark, using LLM-as-a-judge to measure the free-form commentary. Experiments show our final LiveCC-7B-Instruct model can surpass advanced 72B models (Qwen2.5-VL-72B-Instruct, LLaVA-Video-72B) in commentary quality even working in a real-time mode. Meanwhile, it achieves state-of-the-art results at the 7B/8B scale on popular video QA benchmarks such as VideoMME and OVOBench, demonstrating the broad generalizability of our approach. All resources of this paper have been released at https://showlab.github.io/livecc.

Figures

Figures reproduced from arXiv: 2504.16030 by the authors.

Figure 1
Figure 1. LiveCC provides real-time commentary for streaming video, emulating a human commentator. This example is drawn from the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. LiveCC data production pipeline. We begin by inte [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Overview of the Live-WhisperX-526K dataset. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Modeling Overview of LiveCC. The model processes streaming video frames through a visual encoder to produce visual tokens while assigning ASR text from corresponding frame intervals as text tokens. The LLM autoregressively predicts text tokens within this densely inter…
Figure 6
Figure 6. Figure 6: (a) Category Distribution of the LiveSports Benchmark: The benchmark includes 3k live CCs and MCQs, split into two tracks: CC and QA. (b) Event Duration and ASR Word Count in the CC track: For CC, event duration (left y-axis) and ASR word count (right y-axis) are analy…
Figure 7
Figure 7. Figure 7: Comparison of pre-trained and instruction tuning enhanced model’s predictions on the same video. This ex￾ample is sourced from Video-MME [23], with the YouTube ID whksDmTR9YE featuring animal fights. our LiveSports-3K benchmarks, including advanced pro￾prietary models,…
Figure 8
Figure 8. Figure 8: The prompts used during the pre-training instruction-tuning (aka. SFT) stages. CM represents commentary, QA denotes question [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: The comparison between the commentary generated by [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Real-time video commentary demo on unseen YouTube video ( [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Real-time video commentary demo on unseen YouTube video ( [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Real-time video commentary demo on unseen YouTube video ( [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Real-time video commentary demo on unseen YouTube video ( [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark

    cs.CV 2025-04 conditional novelty 6.0 of 10

    Video-MMLU: a 1,065-video lecture benchmark where most AI video models score 10-50%, but text-only models answer 40% of quiz questions without video.

  2. Aero Realtime: Fully Aligned Input-Output Streams for Low-Latency Streaming Multimodal Generation

    cs.AI 2026-08 conditional novelty 5.0 of 10

    Aero Realtime aligns continuous video, audio, and text output on one 80ms grid, letting a 4B model generate lexical tokens or silence in a single stream while reusing the KV cache.

  3. Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Codec-guided sparse patch selection plus a lightweight speak/silent gate yields a 4B streaming VLM that is competitive on static tasks, stronger on video/spatial benchmarks, and much cheaper at inference.

Reference graph

Works this paper leans on

116 extracted references · 23 canonical work pages · cited by 3 Pith papers

  1. [1]

    Marks, Chiori Hori, Peter Anderson, Stefan Lee, and Devi Parikh

    Huda AlAmri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K. Marks, Chiori Hori, Peter Anderson, Stefan Lee, and Devi Parikh. Audio visual scene-aware dialog. In CVPR, pages 7558– 7567, 2019. 3

    2019

  2. [2]

    Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Kar ´en Simonyan

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Men- sch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj B...

    2022

  3. [3]

    Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalk- wyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Fi- rat, James Molloy, Micha...

    arXiv 2023

  4. [4]

    Qwen tech- nical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Day- iheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfe...

    arXiv 2023

  5. [5]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv:2308.12966, 2023. 2

    arXiv 2023

  6. [6]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhao- hai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Jun- yang Lin. Qwen2.5-vl technical repor...

    arXiv

  7. [7]

    Whisperx: Time-accurate speech transcription of long-form audio

    Max Bain, Jaesung Huh, Tengda Han, and Andrew Zis- serman. Whisperx: Time-accurate speech transcription of long-form audio. arXiv preprint arXiv:2303.00747, 2023. 3, 4

    arXiv 2023

  8. [8]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, S...

    1901

Show all 116 references
  1. [9]

    Sst: Single-stream tem- poral action proposals

    Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles. Sst: Single-stream tem- poral action proposals. In Proceedings of the IEEE confer- ence on Computer Vision and Pattern Recognition , pages 2911–2920, 2017. 3

    2017

  2. [10]

    Quo vadis, action recognition? A new model and the kinetics dataset

    Jo ˜ao Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In CVPR, pages 4724–4733, 2017. 3

    2017

  3. [11]

    A short note about kinetics-

    Jo ˜ao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-

  4. [12]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv:2310.09478,

    arXiv

  5. [13]

    Videollm-online: Online video large language model for streaming video

    Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongx- ing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In CVPR,

  6. [14]

    Sharegpt4video: Improving video understanding and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understanding and generation with better captions. arXiv preprint arXiv:2406.04325, 2024. 2

    2024 arXiv

  7. [15]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Confere...

    2024

  8. [16]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhang- wei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui He, Botian Shi, Xingcheng Zhang, Han Lv, ...

    2024 arXiv

  9. [17]

    How far are we to gpt-4v? clos- ing the gap to commercial multimodal models with open- source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? clos- ing the gap to commercial multimodal models with open- source suites. arXiv preprint arXiv:2404.16821, 2024. 8

    2024 arXiv

  10. [18]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 8

    2024 arXiv

  11. [19]

    Unsupervised cross-lingual representation learning at scale

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm´an, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In ACL, 2020. 3

    2020

  12. [20]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C.H.Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. arXiv:2305.06500, 2023. 2

    2023 arXiv

  13. [21]

    An im- age is worth 16x16 words: Transformers for image recog- nition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An im- age is worth 16x16 words: Transformers for image recog- nitio...

    2021

  14. [22]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 arXiv

  15. [23]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024. 2, 7, 8, 9

    2024 arXiv

  16. [24]

    Vita: Towards open-source interactive omni multimodal llm

    Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, et al. Vita: Towards open-source interactive omni multimodal llm. arXiv:2408.05211, 2024. 2, 3

    2024 arXiv

  17. [25]

    Vita-1.5: Towards gpt-4o level real-time vision and speech interaction

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yun- hang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv:2501.01957, 2025. 2, 3

    2025 arXiv

  18. [26]

    Online action detection

    Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snoek, and Tinne Tuytelaars. Online action detection. In ECCV, pages 269–284, 2016. 3

    2016

  19. [27]

    Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context

    Gemini Team Google. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv:2403.05530, 2024. 2

    2024 arXiv

  20. [28]

    The ”something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fr ¨und, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The ”something something” video da...

    2017

  21. [29]

    Hello gpt-4o, 2024

    GPT-4o. Hello gpt-4o, 2024. 2, 4, 6, 7, 8, 15

    2024

  22. [30]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, pages 961–970, 2015. 3, 7

    2015

  23. [31]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  24. [32]

    Multimodal pretraining for dense video cap- tioning

    Gabriel Huang, Bo Pang, Zhenhai Zhu, Clara Rivera, and Radu Soricut. Multimodal pretraining for dense video cap- tioning. In IJCNLP-AACL, pages 470–490, 2020. 2

    2020

  25. [33]

    Online video understanding: A com- prehensive benchmark and memory-augmented method

    Zhenpeng Huang, Xinhao Li, Jiaqi Li, Jing Wang, Xi- angyu Zeng, Cheng Liang, Tao Wu, Xi Chen, Liang Li, and Limin Wang. Online video understanding: A com- prehensive benchmark and memory-augmented method. arXiv:2501.00584, 2025. 3

    2025 arXiv

  26. [34]

    Zamir, Yu-Gang Jiang, Alex Gor- ban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah

    Haroon Idrees, Amir R. Zamir, Yu-Gang Jiang, Alex Gor- ban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The THUMOS challenge on action recognition for videos ”in the wild”. Comput. Vis. Image Underst. , 155:1–23,

  27. [35]

    Cag-qil: Context-aware actionness grouping via q imitation learning for online temporal action localization

    Hyolim Kang, Kyungmin Kim, Yumin Ko, and Seon Joo Kim. Cag-qil: Context-aware actionness grouping via q imitation learning for online temporal action localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13729–13738, 2021. 3

    2021

  28. [36]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv:2001.08361, 2020. 2

    2001 arXiv

  29. [37]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV, pages 706–715, 2017. 2

    2017

  30. [38]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv:2308.00692, 2023. 2

    2023 arXiv

  31. [39]

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg. TVQA: localized, compositional video question answering. In EMNLP, pages 1369–1379, 2018. 3

    2018

  32. [40]

    Otter: A multi-modal model with in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv:2305.03726,

    arXiv

  33. [42]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv:2408.03326, 2024. 2

    2024 arXiv

  34. [43]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 2

    2023 arXiv

  35. [44]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wen- hai Wang, Ping Luo, Yali Wang, Limin Wang, and 11 Yu Qiao. Videochat: Chat-centric video understanding. arXiv:2305.06355, 2023. 2

    2023 arXiv

  36. [45]

    Mvbench: A comprehensive multi-modal video under- standing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video under- standing benchmark. In CVPR, pages 22195–22206, 2024. 2, 7, 8

    2024

  37. [46]

    Ovo-bench: How far is your video-llms from real-world online video understand- ing? arXiv:2501.05510, 2025

    Yifei Li, Junbo Niu, Ziyang Miao, Chunjiang Ge, Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, and Jiaqi Wang. Ovo-bench: How far is your video-llms from real-world online video understand- ing? arX...

    2025 arXiv

  38. [47]

    A light weight model for active speaker detection

    Junhua Liao, Haihan Duan, Kanghui Feng, Wanbing Zhao, Yanbing Yang, and Liangyin Chen. A light weight model for active speaker detection. InCVPR, pages 22932–22941,

  39. [48]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv:2311.10122, 2023. 2

    2023 arXiv

  40. [49]

    Streaming- bench: Assessing the gap for mllms to achieve stream- ing video understanding

    Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. Streaming- bench: Assessing the gap for mllms to achieve stream- ing video understanding. arXiv preprint arXiv:2411.03628,

    arXiv

  41. [50]

    VILA: on pre-training for vi- sual language models

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Moham- mad Shoeybi, and Song Han. VILA: on pre-training for vi- sual language models. InCVPR, pages 26679–26689, 2024. 2, 5

    2024

  42. [51]

    Egocentric video-language pretrain- ing

    Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, Hongfa Wang, Dima Damen, Bernard Ghanem, Wei Liu, and Mike Zheng Shou. Egocentric video-language pretrain- ing. arXi...

    2022 arXiv

  43. [52]

    Univtg: Towards unified video- language temporal grounding

    Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shra- man Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video- language temporal grounding. In ICCV, pages 2782–2792,

  44. [53]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv:2310.03744, 2023. 2

    2023 arXiv

  45. [54]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2023. 2, 5

    2023

  46. [55]

    Al- varez

    Jihao Liu, Zhiding Yu, Shiyi Lan, Shihao Wang, Rongyao Fang, Jan Kautz, Hongsheng Li, and Jose M. Al- varez. Streamchat: Chatting with streaming video. arXiv:2412.08646, 2024. 2, 3

    2024 arXiv

  47. [56]

    Oryx mllm: On-demand spatial- temporal understanding at arbitrary resolution

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On-demand spatial- temporal understanding at arbitrary resolution. arXiv preprint arXiv:2409.12961, 2024. 8

    2024 arXiv

  48. [57]

    Video-ChatGPT: Towards detailed video un- derstanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-ChatGPT: Towards detailed video un- derstanding via large vision and language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2024

  49. [58]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. In NeurIPS, pages 46212–46244, 2023. 3

    2023

  50. [59]

    Mm1: Meth- ods, analysis & insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Meth- ods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611, 2024. 2

    2024 arXiv

  51. [60]

    Howto100m: Learning a text-video embedding by watch- ing hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watch- ing hundred million narrated video clips. In ICCV, pages 2630–2640, 2019. 2, 3

    2019

  52. [61]

    Soccernet-caption: Dense video captioning for soccer broadcasts commentaries

    Hassan Mkhallati, Anthony Cioppa, Silvio Giancola, Bernard Ghanem, and Marc Van Droogenbroeck. Soccernet-caption: Dense video captioning for soccer broadcasts commentaries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5074–5085, 2023. 5

    2023

  53. [62]

    Introducing chatgpt

    OpenAI. Introducing chatgpt. https://openai.com/ blog/chatgpt/, 2023. 1

    2023

  54. [63]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. arXiv:2303.08774, 2023. 1

    2023 arXiv

  55. [64]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. https://cdn. openai.com/papers/GPTV_System_Card.pdf ,

  56. [65]

    Py- torch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, and Francisco et al Massa. Py- torch: An imperative style, high-performance deep learning library. In NeurIPS, pages 8026–8037, 2019. 7

    2019

  57. [66]

    Perception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Kop- pula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Advances in Neural Information Processing Sy...

    2024

  58. [67]

    Streaming long video understanding with large language models

    Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuan- grui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. In NeurIPS, 2024. 2, 3

    2024

  59. [68]

    Dispider: Enabling video llms with active real-time inter- action via disentangled perception, decision, and reaction

    Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video llms with active real-time inter- action via disentangled perception, decision, and reaction. arXiv:2501.03218, 2025. 2, 3

    2025 arXiv

  60. [69]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gen- erative pre-training. 2018. 2

    2018

  61. [70]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsu- pervised multitask learners. 2019. 2

    2019

  62. [71]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen 12 Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pages 87...

    2021

  63. [72]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML, pages 28492–28518, 2023. 3, 4

    2023

  64. [73]

    Matchtime: Towards automatic soccer game commentary generation

    Jiayuan Rao, Haoning Wu, Chang Liu, Yanfeng Wang, and Weidi Xie. Matchtime: Towards automatic soccer game commentary generation. arXiv preprint arXiv:2406.18530,

    arXiv

  65. [74]

    Timechat: A time-sensitive multimodal large lan- guage model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large lan- guage model for long video understanding. InCVPR, pages 14313–14323, 2024. 2

    2024

  66. [75]

    Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra. Longvu: Spatiote...

    arXiv

  67. [76]

    Online real-time multiple spa- tiotemporal action localisation and prediction

    Gurkirt Singh, Suman Saha, Michael Sapienza, Philip HS Torr, and Fabio Cuzzolin. Online real-time multiple spa- tiotemporal action localisation and prediction. In Proceed- ings of the IEEE international conference on computer vi- sion, pages 3637–3646, 2017. 3

    2017

  68. [77]

    Moviechat: From dense token to sparse memory for long video understand- ing

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understand- ing. arXiv:2307.16449, 2023. 2

    2023 arXiv

  69. [78]

    Llama: Open and efficient foundation language mod- els

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Bap- tiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language mod- els. arXiv:2302.13971, 2023. 1

    2023 arXiv

  70. [79]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fer- nandes, Jere...

    2023 arXiv

  71. [80]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi- hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution. arXiv:2409.12191, 2024. 2, 4, 5, 7, 8, 15

    2024 arXiv

  72. [81]

    Internvid: A large-scale video- text dataset for multimodal understanding and generation

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video- text dataset for multimodal understanding and generation. arXiv:2307.06942, 2023. 2

    2023 arXiv

  73. [82]

    Videollm knows when to speak: Enhancing time-sensitive video comprehension with video-text duet interaction for- mat, 2024

    Yueqian Wang, Xiaojun Meng, Yuxuan Wang, Jianxin Liang, Jiansheng Wei, Huishuai Zhang, and Dongyan Zhao. Videollm knows when to speak: Enhancing time-sensitive video comprehension with video-text duet interaction for- mat, 2024. 2, 3

    2024

  74. [83]

    Videollamb: Long-context video understanding with recur- rent memory bridges

    Yuxuan Wang, Cihang Xie, Yang Liu, and Zilong Zheng. Videollamb: Long-context video understanding with recur- rent memory bridges. arXiv:2409.01071, 2024. 2, 3

    2024 arXiv

  75. [84]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R ´emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, ...

    2020

  76. [85]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9777–9786, 2021. 3, 7

    2021

  77. [86]

    Streaming video understanding and multi-round interaction with memory- enhanced knowledge

    Haomiao Xiong, Zongxin Yang, Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Jiawen Zhu, and Huchuan Lu. Streaming video understanding and multi-round interaction with memory- enhanced knowledge. In ICLR, 2025. 2, 3

    2025

  78. [87]

    Qwen2.5-omni technical report

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215, 2025. 8, 15

    2025 arXiv

  79. [88]

    Ad- vancing high-resolution video-language representation with large-scale video transcriptions

    Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Ad- vancing high-resolution video-language representation with large-scale video transcriptions. In CVPR, pages 5026– 5035, 2022. 2, 3

    2022

  80. [89]

    Vidchapters-7m: Video chapters at scale

    Antoine Yang, Arsha Nagrani, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vidchapters-7m: Video chapters at scale. In NeurIPS, 2023. 3

    2023

  81. [90]

    Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, An- toine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning. In CVPR, pages 10714–10726, 2023. 2, 3 13

    2023

  82. [91]

    Qwen2 technical report

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. 1, 4

    2024 arXiv

  83. [92]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhi- hui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 8

    2024 arXiv

  84. [93]

    Deepspeed- visualchat: Multi-round multi-image interleave chat via multi-modal causal attention

    Zhewei Yao, Xiaoxia Wu, Conglong Li, Minjia Zhang, Heyang Qi, Olatunji Ruwase, Ammar Ahmad Awan, Samyam Rajbhandari, and Yuxiong He. Deepspeed- visualchat: Multi-round multi-image interleave chat via multi-modal causal attention. arXiv:2309.14327, 2023. 2

    2023 arXiv

  85. [94]

    mplug-owl3: Towards long image-sequence understand- ing in multi-modal large language models

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understand- ing in multi-modal large language models. arXiv preprint arXiv:2408.04840, 2024. 8

    2024 arXiv

  86. [95]

    Ferret: Refer and ground anything anywhere at any granularity

    Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv:2310.07704, 2023. 2

    2023 arXiv

  87. [96]

    MERLOT RESERVE: neural script knowledge through vision and language and sound

    Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yan- peng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. MERLOT RESERVE: neural script knowledge through vision and language and sound. In CVPR, pages 16354–16366, 2022. 2, 3

    2022

  88. [97]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11941–11952, 2023. 2

    2023

  89. [98]

    Videollama 3: Frontier multimodal foundation models for image and video under- standing

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video under- standing. ...

    arXiv

  90. [99]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv:2306.02858, 2023. 2

    2023 arXiv

  91. [100]

    Flash-vstream: Memory- based real-time understanding for long video streams

    Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory- based real-time understanding for long video streams. arXiv:2406.08085, 2024. 2

    2024 arXiv

  92. [101]

    Lmms-eval: Reality check on the evaluation of large multi- modal models

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuan- han Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multi- modal models. arXiv:2407.12772, 2024. 7

    2024 arXiv

  93. [102]

    Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual in- put and output. arXiv preprint arXiv:2407.03320, 2024. 8

    2024 arXiv

  94. [103]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 8

    2024 arXiv

  95. [104]

    Gpt4roi: Instruction tuning large language model on region-of- interest

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of- interest. arXiv:2307.03601, 2023. 2

    2023 arXiv

  96. [106]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024. 2, 8, 15, 16

    2024 arXiv

  97. [107]

    Real-time online video detection with temporal smoothing transformers

    Yue Zhao and Philipp Kr ¨ahenb¨uhl. Real-time online video detection with temporal smoothing transformers. In Eu- ropean Conference on Computer Vision , pages 485–502. Springer, 2022. 3

    2022

  98. [108]

    Streaming video model

    Yucheng Zhao, Chong Luo, Chuanxin Tang, Dongdong Chen, Noel Codella, and Zheng-Jun Zha. Streaming video model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14602– 14612, 2023. 3

    2023

  99. [109]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonza- lez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS, 2023. 2, 6, 7

    2023

  100. [110]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018. 2

    2018

  101. [111]

    Streaming dense video captioning

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. In CVPR, pages 18243–18252, 2024. 2, 3

    2024

  102. [112]

    Minigpt-4: Enhancing vision- language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv:2304.10592, 2023. 2 14 Live : Learning Video LLM with Streaming Speech Transcription at Scale Supplementary Material

    2023 arXiv

  103. [114]

    Demo This section showcases four demo videos to demonstrate the capability of our LiveCC-7B-Instruct to provide real- time commentary in real-world videos across different do- mains, including sports (football), science (astronomy), news (weather forecast), and instructional (...

  104. [115]

    Prompt Template In this section, we detail the prompt designs used during the pre-training, instruction tuning, and inference stages

    Implemetation Details 8.1. Prompt Template In this section, we detail the prompt designs used during the pre-training, instruction tuning, and inference stages. As shown in Figure 8(a) and (b), the video title and pre- viously transcribed ASR text are provided as contextual in...

  105. [116]

    If the above criteria is not enough to judge, then consider:

    Semantic Alignment: The commentary should convey the same meaning, details, and key points as the human commentary. If the above criteria is not enough to judge, then consider:

  106. [117]

    Commentary A is better aligned with the human commentary

    Stylistic Consistency: The commentary should maintain a tone, word choice, and structure similar to the human commentary. ---Commentary A--- {a_pred} ---------- ---Commentary B--- {b_pred} ---------- ---Human Commentary--- {gt_asr} ---------- Your response should be "Commentar...

    1973

  107. [118]

    Argentina v France: Full Penalty Shoot-out — 2022 #FIFAWorldCup Final

    Additional Experiments 9.1. Response Latency To highlight the efficiency of our streaming model, we present the response latency of LLaV A-Video-7B/72B alongside our model in Table 5. Response latency is de- fined as the time a user waits to see the model’s output, a critical ...

  108. [600]

    Arxiv, 1808.01340, 2018. 3

    2018 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.