Pith. sign in

REVIEW 8 cited by

VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.09418 v1 pith:CELOFLDB submitted 2024-06-13 cs.CV

Muhammad Maaz , Hanoona Rasheed , Salman Khan , Fahad Khan This is my paper
classification cs.CV
keywords videoencodersimageunderstandinglmmsspatialtemporalcontext
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Building on the advances of language models, Large Multimodal Models (LMMs) have contributed significant improvements in video understanding. While the current video LMMs utilize advanced Large Language Models (LLMs), they rely on either image or video encoders to process visual inputs, each of which has its own limitations. Image encoders excel at capturing rich spatial details from frame sequences but lack explicit temporal context, which can be important in videos with intricate action sequences. On the other hand, video encoders provide temporal context but are often limited by computational constraints that lead to processing only sparse frames at lower resolutions, resulting in reduced contextual and spatial understanding. To this end, we introduce VideoGPT+, which combines the complementary benefits of the image encoder (for detailed spatial understanding) and the video encoder (for global temporal context modeling). The model processes videos by dividing them into smaller segments and applies an adaptive pooling strategy on features extracted by both image and video encoders. Our architecture showcases improved performance across multiple video benchmarks, including VCGBench, MVBench and Zero-shot question-answering. Further, we develop 112K video-instruction set using a novel semi-automatic annotation pipeline which further improves the model performance. Additionally, to comprehensively evaluate video LMMs, we present VCGBench-Diverse, covering 18 broad video categories such as lifestyle, sports, science, gaming, and surveillance videos. This benchmark with 4,354 question-answer pairs evaluates the generalization of existing LMMs on dense video captioning, spatial and temporal understanding, and complex reasoning, ensuring comprehensive assessment across diverse video types and dynamics. Code: https://github.com/mbzuai-oryx/VideoGPT-plus.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RynnBrain 1.1: Towards More Capable and Generalizable Embodied Foundation Model

    cs.RO 2026-07 conditional novelty 6.0 of 10

    RynnBrain 1.1 reports state-of-the-art embodied cognition and localization scores with a 122B-A10B model and improved real-robot VLA policies via joint multi-embodiment training.

  2. VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An open 4B video MLLM with inflated-3D ViT tokenization and adaptive streaming perception outperforms comparable open models on general, long-video, and streaming benchmarks while using fewer visual tokens.

  3. Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors

    cs.CV 2025-08 conditional novelty 6.0 of 10

    LangDC compresses video tokens dynamically by converting clips into captions from a small language model, cutting compute by 49% with near-parity accuracy.

  4. DisCo: Towards Distinct and Coherent Visual Encapsulation in Video MLLMs

    cs.CV 2025-07 conditional novelty 6.0 of 10

    DisCo assigns each visual token to a unique concept from the caption and aligns its attention across frames, improving video MLLM accuracy and token efficiency.

  5. IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A new multi-shot video dataset and an instance-prompt video LLM report large gains, but the main benchmark is built by the same authors and the model is not released.

  6. When and What: Diffusion-Grounded VideoLLM with Entity Aware Segmentation for Long Video Understanding

    cs.CV 2025-08 reject novelty 4.0 of 10

    A video-LLM that uses diffusion features, segmentation-guided object tokens, and discrete time tokens to improve temporal grounding, but its claimed SOTA results are not supported by its own tables.

  7. The high-speed X-ray camera on AXIS: design and performance updates

    astro-ph.IM 2025-08 unverdicted novelty 4.0 of 10

    An X-ray camera design-update whose supporting full text is a different paper (RynnEC, an embodied AI model), leaving all camera performance claims unverified.

  8. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools