REVIEW 10 cited by
Lost in Time: A New Temporal Benchmark for VideoLLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models have demonstrated impressive performance when integrated with vision models even enabling video understanding. However, evaluating video models presents its own unique challenges, for which several benchmarks have been proposed. In this paper, we show that the currently most used video-language benchmarks can be solved without requiring much temporal reasoning. We identified three main issues in existing datasets: (i) static information from single frames is often sufficient to solve the tasks (ii) the text of the questions and candidate answers is overly informative, allowing models to answer correctly without relying on any visual input (iii) world knowledge alone can answer many of the questions, making the benchmarks a test of knowledge replication rather than video reasoning. In addition, we found that open-ended question-answering benchmarks for video understanding suffer from similar issues while the automatic evaluation process with LLMs is unreliable, making it an unsuitable alternative. As a solution, we propose TVBench, a novel open-source video multiple-choice question-answering benchmark, and demonstrate through extensive evaluations that it requires a high level of temporal understanding. Surprisingly, we find that most recent state-of-the-art video-language models perform similarly to random performance on TVBench, with only a few models such as Qwen2-VL, and Tarsier clearly surpassing this baseline.
Forward citations
Cited by 10 Pith papers
-
AdaThinkV: Adaptive Thinking for Token-Efficient Video Reasoning
A video reasoning model learns per question whether to reason aloud or answer directly, improving accuracy by about 3 points over the best adaptive baseline while using about 23% fewer output tokens.
-
EPFL-Smart-Kitchen-30: Densely annotated cooking dataset with 3D kinematics to challenge video and language models
EPFL-Smart-Kitchen-30 is a 29.7-hour multimodal cooking dataset with 60k action segments and four benchmarks, including a kinematic-focused VQA benchmark that shows current video-language models struggle with hand and...
-
Do Video-LLMs Actually Watch? Diagnosing Character-Tracking Failures in Long-Form Video
Video-LLMs scoring 37–38% on InfiniBench global appearance change answers only 4–31% under character-name swaps, so the score is not character tracking.
-
VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding
An open 4B video MLLM with inflated-3D ViT tokenization and adaptive streaming perception outperforms comparable open models on general, long-video, and streaming benchmarks while using fewer visual tokens.
-
Accuracy Without Grounding: Diagnosing Visual Dependency Dissociation in Video LLM Benchmarks
A benchmark audit shows that video LLM accuracy and true visual grounding are separable, and that most apparent video understanding comes from frame diversity, not temporal order.
-
EgoExo-Con: Exploring View-Invariant Video Temporal Understanding
Most Video-LLMs answer temporal questions far less consistently when the same event is shown from ego and exo views, and a GRPO variant with a reasoning-similarity reward partially closes the gap.
-
MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models
MESH, a three-layer video hallucination benchmark, shows LVMs ace basic objects and coarse traits but slip badly on fine character details and multi-subject actions in longer clips.
-
VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos
VRBench is a benchmark of 960 long narrative videos with 8,243 human-written multi-step questions, plus a two-level evaluation of answer accuracy and reasoning quality for 31 large models.
-
How Important are Videos for Training Video LLMs?
Training video LLMs on ordered static images with temporal questions nearly matches real video training on TVBench, suggesting video data is underused.
-
Fostering Video Reasoning via Next-Event Prediction
Next-event prediction, training video language models to caption unseen future frames, improves their scores on several temporal benchmarks while roughly preserving general video understanding.
Discussion (0). Continue with ORCID to comment.