REVIEW 17 cited by
Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Long video understanding poses a significant challenge for current Multi-modal Large Language Models (MLLMs). Notably, the MLLMs are constrained by their limited context lengths and the substantial costs while processing long videos. Although several existing methods attempt to reduce visual tokens, their strategies encounter severe bottleneck, restricting MLLMs' ability to perceive fine-grained visual details. In this work, we propose Video-XL, a novel approach that leverages MLLMs' inherent key-value (KV) sparsification capacity to condense the visual input. Specifically, we introduce a new special token, the Visual Summarization Token (VST), for each interval of the video, which summarizes the visual information within the interval as its associated KV. The VST module is trained by instruction fine-tuning, where two optimizing strategies are offered. 1.Curriculum learning, where VST learns to make small (easy) and large compression (hard) progressively. 2. Composite data curation, which integrates single-image, multi-image, and synthetic data to overcome the scarcity of long-video instruction data. The compression quality is further improved by dynamic compression, which customizes compression granularity based on the information density of different video intervals. Video-XL's effectiveness is verified from three aspects. First, it achieves a superior long-video understanding capability, outperforming state-of-the-art models of comparable sizes across multiple popular benchmarks. Second, it effectively preserves video information, with minimal compression loss even at 16x compression ratio. Third, it realizes outstanding cost-effectiveness, enabling high-quality processing of thousands of frames on a single A100 GPU.
Forward citations
Cited by 17 Pith papers
-
Vid-SME: Membership Inference Attacks against Large Video Understanding Models
Vid-SME computes Sharma-Mittal entropy differences between natural and reversed video frame sequences to infer training membership in video understanding LLMs, but its effectiveness is confounded by member/non-member ...
-
CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models
CRAFT recursively merges video tokens with training-free similarity selection plus learnable gated fusion, retaining ~97% of average accuracy at 8x compression across six benchmarks.
-
Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understanding
AVP, an iterative plan-observe-reflect agent that samples query-relevant video segments directly, reports state-of-the-art accuracy on five long-video QA benchmarks with roughly 5.4x less inference time than the leadi...
-
MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models
MESH, a three-layer video hallucination benchmark, shows LVMs ace basic objects and coarse traits but slip badly on fine character details and multi-subject actions in longer clips.
-
VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understanding
The new VRU-Accident benchmark (1K videos, 6K QA pairs, 1K dense captions) shows the best evaluated MLLM reaches 66.9% on VRU-accident VQA versus 94.7% for human experts, with the weakest performance on causal and pre...
-
ExpStar: Towards Automatic Commentary Generation for Multi-discipline Scientific Experiments
ExpStar, with a new 7,714-sample ExpInstruct dataset, generates step-level scientific experiment commentary including procedures, principles, and safety guidelines.
-
AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.
-
LongAnimation: Long Animation Generation with Dynamic Global-Local Memory
LongAnimation uses a dynamic global-local memory, built from a long-video-understanding model's KV cache, to colorize animation sequences of about 500 frames with stable color consistency.
-
Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenders
Nar-KFC improves long-video question answering by interleaving query-relevant diverse keyframes with captions of non-keyframes, raising accuracy on Video-MME, LongVideoBench, and MLVU across 7B MLLMs.
-
VAGU & GtS: LLM-Based Benchmark and Framework for Joint Video Anomaly Grounding and Understanding
A new benchmark, a training-free framework, and a joint metric for video anomaly detection that combines temporal grounding with semantic understanding.
-
Iterative Zoom-In: Temporal Interval Exploration for Long Video Understanding
A confidence-guided iterative zoom-in framework improves long-video question answering for multimodal LLMs without any training.
-
MANTA: Cross-Modal Semantic Alignment and Information-Theoretic Optimization for Long-form Multimodal Understanding
A multimodal retrieval pipeline that projects video and audio into text and claims near-optimal context selection, with reported gains of up to 22.6% on Video-MME that rest on circular theory and unreleased data.
-
Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs
A CLIP-scored, Gumbel-Max frame sampler with per-frame multi-resolution allocation improves long-video question answering in Video-LLMs under a fixed token budget.
-
Task-Aware KV Compression For Cost-Effective Long Video Understanding
Video-X2L uses bi-level KV compression with task-aware selective reloading to improve long-video QA accuracy and reduce decode-time memory versus uniform KV compression.
-
CyberV: Cybernetics for Test-time Scaling in Video Understanding
A training-free test-time feedback loop, using attention drift to select key frames, improves video MLLM accuracy, with the largest gains on knowledge-heavy VideoMMMU.
-
FlexSelect: Flexible Token Selection for Efficient Long Video Understanding
FlexSelect selects a small fraction of query-relevant visual tokens using attention from an intermediate layer, improving long-video accuracy and inference speed across multiple VideoLLMs.
-
Infinite Video Understanding
The paper argues that video understanding research should aim at processing streams of arbitrary, unbounded duration and outlines the challenges, directions, and metrics needed.
Discussion (0). Sign in to comment.