REVIEW 17 cited by
Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Video-based large language models (Video-LLMs) have been recently introduced, targeting both fundamental improvements in perception and comprehension, and a diverse range of user inquiries. In pursuit of the ultimate goal of achieving artificial general intelligence, a truly intelligent Video-LLM model should not only see and understand the surroundings, but also possess human-level commonsense, and make well-informed decisions for the users. To guide the development of such a model, the establishment of a robust and comprehensive evaluation system becomes crucial. To this end, this paper proposes \textit{Video-Bench}, a new comprehensive benchmark along with a toolkit specifically designed for evaluating Video-LLMs. The benchmark comprises 10 meticulously crafted tasks, evaluating the capabilities of Video-LLMs across three distinct levels: Video-exclusive Understanding, Prior Knowledge-based Question-Answering, and Comprehension and Decision-making. In addition, we introduce an automatic toolkit tailored to process model outputs for various tasks, facilitating the calculation of metrics and generating convenient final scores. We evaluate 8 representative Video-LLMs using \textit{Video-Bench}. The findings reveal that current Video-LLMs still fall considerably short of achieving human-like comprehension and analysis of real-world videos, offering valuable insights for future research directions. The benchmark and toolkit are available at: \url{https://github.com/PKU-YuanGroup/Video-Bench}.
Forward citations
Cited by 17 Pith papers
-
SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing
SAVVY-Bench tests audio-visual LLMs on dynamic 3D spatial questions, and the SAVVY pipeline, combining visual tracks with spatial audio and global mapping, lifts Gemini-2.5-pro accuracy from 50.9% to 58.0%.
-
Out of Sight, Not Out of Context? Egocentric Spatial Reasoning in VLMs Across Disjoint Frames
A new egocentric benchmark shows vision-language models fail at spatial reasoning across disjoint frames, falling 28 points behind humans and only improving sharply when handed ground-truth 3D coordinates.
-
AdsQA: Towards Advertisement Video Understanding
AdsQA adds an ad-video question-answering benchmark and ReAd-R, a GRPO-trained model that beats 7B baselines but not larger closed models.
-
MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models
MESH, a three-layer video hallucination benchmark, shows LVMs ace basic objects and coarse traits but slip badly on fine character details and multi-subject actions in longer clips.
-
VideoForest: Person-Anchored Hierarchical Reasoning for Cross-Video Question Answering
A person-anchored tree plus multi-agent LLM pipeline lets a system answer cross-video queries about the same person, and it beats single-video models on the authors' new CrossVideoQA benchmark.
-
VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
VLM4D benchmarks spatiotemporal reasoning in VLMs and finds large gaps versus humans, with proposed methods showing partial improvement.
-
"Harmless to You, Hurtful to Me!": Investigating the Detection of Toxic Languages Grounded in the Perspective of Youth
The authors construct the first Chinese youth-toxicity dataset, show that youth and adult perceptions of toxic language diverge, and report that adding contextual meta information improves detection accuracy.
-
GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?
Introduces GLIMPSE, a video-QA benchmark whose questions cannot be answered from single frames; best model GPT-o3 scores 66.43% vs 94.82% human accuracy.
-
VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos
VRBench is a benchmark of 960 long narrative videos with 8,243 human-written multi-step questions, plus a two-level evaluation of answer accuracy and reasoning quality for 31 large models.
-
MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks
AVHaystacks is a new 3100-question benchmark for audio-visual QA across 500 videos, and the MAGNET multi-agent pipeline beats current baselines on it.
-
VideoConviction: A Multimodal Benchmark for Human Conviction and Stock Market Recommendations
VideoConviction provides the first expert-annotated multimodal benchmark of financial influencer video recommendations, showing MLLMs extract tickers better but struggle with actions and conviction, and that an invers...
-
VUDG: A Dataset for Video Understanding Domain Generalization
VUDG is a domain-generalization benchmark for video understanding with 11 domains and 36,388 QA pairs, and it shows that current large video-language models lose accuracy across visual domains.
-
TimeCausality: Evaluating the Causal Ability in Time Dimension for Vision Language Models
Introduces a 700-pair benchmark for temporal causal reasoning in VLMs, revealing large open-source vs. closed-source gaps and strong position bias.
-
NeMo: Needle in a Montage for Video-Language Understanding
NeMoBench, an automatically generated benchmark with 31,378 QA pairs, shows that video LLMs struggle with temporal grounding of relevant clips hidden in long montages.
-
Iterative Zoom-In: Temporal Interval Exploration for Long Video Understanding
A confidence-guided iterative zoom-in framework improves long-video question answering for multimodal LLMs without any training.
-
VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos
A new benchmark, VF-Eval, measures how well multimodal LLMs check, detect, and reason about errors in AI-generated videos, and shows frontier models remain far below human performance.
-
CoT-lized Diffusion: Let's Reinforce T2I Generation Step-by-step
CoT-Diff couples a multimodal LLM's step-by-step 3D layout reasoning into the diffusion denoising loop, claiming large gains in spatial alignment for text-to-image generation.
Discussion (0). Sign in to comment.