Pith. sign in

REVIEW 11 cited by

VideoLLM: Modeling Video Sequence with Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.13292 v2 pith:OBXBUGYT submitted 2023-05-22 cs.CV

Guo Chen , Yin-Dong Zheng , Jiahao Wang , Jilan Xu , Yifei Huang , Junting Pan , Yi Wang , Yali Wang
show 3 more authors
Yu Qiao Tong Lu Limin Wang
This is my paper
classification cs.CV
keywords videovideollmsequenceunderstandingllmstaskslanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

With the exponential growth of video data, there is an urgent need for automated technology to analyze and comprehend video content. However, existing video understanding models are often task-specific and lack a comprehensive capability of handling diverse tasks. The success of large language models (LLMs) like GPT has demonstrated their impressive abilities in sequence causal reasoning. Building upon this insight, we propose a novel framework called VideoLLM that leverages the sequence reasoning capabilities of pre-trained LLMs from natural language processing (NLP) for video sequence understanding. VideoLLM incorporates a carefully designed Modality Encoder and Semantic Translator, which convert inputs from various modalities into a unified token sequence. This token sequence is then fed into a decoder-only LLM. Subsequently, with the aid of a simple task head, our VideoLLM yields an effective unified framework for different kinds of video understanding tasks. To evaluate the efficacy of VideoLLM, we conduct extensive experiments using multiple LLMs and fine-tuning methods. We evaluate our VideoLLM on eight tasks sourced from four different datasets. The experimental results demonstrate that the understanding and reasoning capabilities of LLMs can be effectively transferred to video understanding tasks. We release the code at https://github.com/cg1177/VideoLLM.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs

    cs.CV 2025-06 reject novelty 6.0 of 10

    A clue-grounded audio-visual counting benchmark over 497 long videos and an RL-trained counting model, whose headline result is undermined by training on the DVD-Counting evaluation benchmark.

  2. GMoT: Gated Motion-Aware Tokenization for Fine-Grained Micro-Gesture Video Reasoning with Multimodal LLMs

    cs.CV 2026-07 conditional novelty 5.0 of 10

    GMoT's gated motion tokens improve multimodal LLM micro-gesture recognition on iMiGUE and SMG, with limited support for reasoning-grounding claims.

  3. Bidirectional Action Sequence Learning for Long-term Action Anticipation with Large Language Models

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Adding a backward prediction task to LLM training improves long-term action anticipation on Ego4D, lowering edit distance for predicted action sequences.

  4. IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning

    cs.CV 2025-07 conditional novelty 5.0 of 10

    IntentVCNet uses per-frame object coordinates, red-box visual prompts, and a lightweight box adapter to make video captioning focus on a user-selected object, reporting 225.19 CIDEr on the IntentVC public test set.

  5. MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning

    cs.CV 2025-06 conditional novelty 5.0 of 10

    MINT-CoT-7B interleaves fine-grained visual tokens into each math reasoning step and reports 73.70 on MathVista-Math, 64.72 on GeoQA, and 69.6 on MMStar-Math.

  6. Time-Scaling State-Space Models for Dense Video Captioning

    cs.CV 2025-09 conditional novelty 4.0 of 10

    A state-space model with transfer state processes videos chunk by chunk, carries the hidden state forward, and performs online dense video captioning with the same state as full-sequence processing.

  7. NeuroVoxel-LM: Language-Aligned 3D Perception via Dynamic Voxelization and Meta-Embedding

    cs.CV 2025-07 conditional novelty 4.0 of 10

    NeuroVoxel-LM combines dynamic multi-resolution voxelization with attention-based pooling of NeRF weights, reporting faster 3D feature extraction and modestly better NeRF captioning than fixed-resolution and max-pooli...

  8. TriPSS: A Tri-Modal Keyframe Extraction Framework Using Perceptual, Structural, and Semantic Representations

    cs.CV 2025-06 conditional novelty 4.0 of 10

    TriPSS fuses CIELAB color, ResNet-50, and LLaMA caption embeddings through PCA and HDBSCAN to extract keyframes, reporting F1 of 0.6104 on TVSum20 and 0.5902 on SumMe.

  9. Technical Report for Ego4D Long-Term Action Anticipation Challenge 2025

    cs.CV 2025-06 conditional novelty 4.0 of 10

    A three-stage pipeline using the EgoVideo-V encoder, a verb-noun co-occurrence reranker, SAM2 hand-object features, and a fine-tuned Llama 2 model took first place in the Ego4D 2025 long-term action anticipation challenge.

  10. Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Review

    cs.LG 2025-06 conditional novelty 4.0 of 10

    Spatio-temporal foundation models are organized into a pipeline of data harmonization, model design, training, and adaptation, with a data property taxonomy for model selection.

  11. Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision

    cs.CV 2025-06 accept novelty 3.0 of 10

    A comprehensive review of cross-view video understanding that uses both first-person and third-person cameras, organized into a three-direction taxonomy with a dataset catalog and future research gaps.

Pith tools