REVIEW 6 cited by
LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answering
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Despite remarkable recent progress, existing long-form VideoQA datasets fall short of meeting the criteria for genuine long-form video understanding. This is primarily due to the use of short videos for question curation, and the reliance on limited-length sub-clips as clues to answer those questions. Meanwhile, previous datasets have limited focus on question type and modality. To remedy this, we introduce LvBench, a Long-form video understanding benchmark for versatile multi-modal question-answering. Our LvBench stands out from existing long-form VideoQA datasets through three key characteristics: 1) Extended temporal durations: We consider videos ranging from 70 seconds to 4 hours, covering single-scene, multi-scene, and full-scene contexts. This design accounts for both video and clue lengths, capturing diverse contextual dynamics. 2) Diverse question types and modalities: LvBench introduces six distinct question types that evaluate various perceptual and cognitive capabilities, utilizing both video frames and subtitles. 3) High-quality annotations: We employ rigorous manual labeling by human annotators. Our dataset comprises 20,061 question-answer pairs sourced from 100 carefully selected movies across diverse genres, annotated collaboratively by multiple individuals. Analysis involving various baselines reveals a consistent trend: the performance of all existing methods significantly deteriorates when video and clue length increases. We expect LvBench to serve as a valuable resource for future works on long-form video understanding.
Forward citations
Cited by 6 Pith papers
-
TrajTok: Learning Trajectory Tokens enables better Video Understanding
TrajTok learns to tokenize video into object-trajectory tokens end-to-end, improving video CLIP, probing, and VLM performance over patch and token-merging baselines.
-
A Benchmark for Omni-Modal Reasoning in Long Videos
A new 45-minute-scale omni-modal video Q&A benchmark and a training-free retrieval-refine agent, whose reported agent score (66.64% in the abstract) is not supported by the paper's own main results (44.66%).
-
Movie Facts and Fibs (MF$^2$): A Benchmark for Long Movie Understanding
MF2 evaluates long-movie understanding by asking models to classify fact/fib claim pairs; the best model trails humans by 23.5 points in pairwise accuracy.
-
ScaleLong: A Multi-Timescale Benchmark for Long Video Understanding
ScaleLong embeds four timescale question types into the same long videos, and evaluation of 23 MLLMs reveals a U-shaped accuracy curve across timescales.
-
VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos
VRBench is a benchmark of 960 long narrative videos with 8,243 human-written multi-step questions, plus a two-level evaluation of answer accuracy and reasoning quality for 31 large models.
-
EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?
EOC-Bench evaluates MLLMs on egocentric object cognition across past, present, and future temporal dimensions, finding large gaps versus humans, especially in absolute time perception.
Discussion (0). Continue with ORCID to comment.