REVIEW 17 cited by
MMVU: Measuring Expert-Level Multi-Discipline Video Understanding
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce MMVU, a comprehensive expert-level, multi-discipline benchmark for evaluating foundation models in video understanding. MMVU includes 3,000 expert-annotated questions spanning 27 subjects across four core disciplines: Science, Healthcare, Humanities & Social Sciences, and Engineering. Compared to prior benchmarks, MMVU features three key advancements. First, it challenges models to apply domain-specific knowledge and perform expert-level reasoning to analyze specialized-domain videos, moving beyond the basic visual perception typically assessed in current video benchmarks. Second, each example is annotated by human experts from scratch. We implement strict data quality controls to ensure the high quality of the dataset. Finally, each example is enriched with expert-annotated reasoning rationals and relevant domain knowledge, facilitating in-depth analysis. We conduct an extensive evaluation of 32 frontier multimodal foundation models on MMVU. The latest System-2-capable models, o1 and Gemini 2.0 Flash Thinking, achieve the highest performance among the tested models. However, they still fall short of matching human expertise. Through in-depth error analyses and case studies, we offer actionable insights for future advancements in expert-level, knowledge-intensive video understanding for specialized domains.
Forward citations
Cited by 17 Pith papers
-
Latent Visual Cache for Video Reasoning
A recurrent latent visual cache in the decoder, trained with contrastive key-frame alignment and a vision-grounded GRPO reward, improves video reasoning accuracy while shortening answers.
-
DELTAVID: Enhancing Fine-Grained Spatiotemporal Perception with Cross-Video Differences
Rule-reward training on controllable cross-video differences (Grounding + MCQ) improves Video MLLM local spatiotemporal evidence localization and transfers to general video QA benchmarks.
-
Video Reasoning without Training
An entropy-guided, inference-time value-cache controller improves video reasoning accuracy and cuts output tokens versus RL-trained baselines.
-
HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenes
A hierarchical benchmark for multimodal models on human-centric visual understanding finds frontier models average under 60% and miss question-uncued visual evidence, with test-time scaling helping only marginally.
-
VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
VLM4D benchmarks spatiotemporal reasoning in VLMs and finds large gaps versus humans, with proposed methods showing partial improvement.
-
"Harmless to You, Hurtful to Me!": Investigating the Detection of Toxic Languages Grounded in the Perspective of Youth
The authors construct the first Chinese youth-toxicity dataset, show that youth and adult perceptions of toxic language diverge, and report that adding contextual meta information improves detection accuracy.
-
ExpStar: Towards Automatic Commentary Generation for Multi-discipline Scientific Experiments
ExpStar, with a new 7,714-sample ExpInstruct dataset, generates step-level scientific experiment commentary including procedures, principles, and safety guidelines.
-
HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context
Requiring omni-modal models to summarize context before reasoning, with LLM-judged context and logical rewards, improves human-intent reasoning benchmarks.
-
GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning
GRPO-CARE improves answer accuracy and reasoning coherence over standard GRPO on a new video reasoning benchmark, with a 6.7 point gain on the hardest level and a 24.5 point higher consistency rate.
-
VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos
VRBench is a benchmark of 960 long narrative videos with 8,243 human-written multi-step questions, plus a two-level evaluation of answer accuracy and reasoning quality for 31 large models.
-
EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?
EOC-Bench evaluates MLLMs on egocentric object cognition across past, present, and future temporal dimensions, finding large gaps versus humans, especially in absolute time perception.
-
Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency
A GRPO-based post-training recipe for video LLMs using discrete QA rewards plus continuous temporal IoU rewards with variance-based data selection outperforms SFT and Video-R1.
-
VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinking
GRPO post-training with structured thinking and dual think/caption rewards improves Qwen2-VL-7B video captioning over the base model and SFT on DREAM-1K, VDC, and CAREBENCH.
-
ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding
ReAgent-V is an agentic video understanding framework whose critic agent generates real-time rewards to refine answers and filter training data, yielding gains of up to 6.9%, 2.1%, and 9.8% across three applications.
-
Reinforcing Video Reasoning with Focused Thinking
A GRPO variant with token-level KL weighting and partial-credit rewards improves video-QA on modified multi-answer benchmarks, but transfer to original single-answer benchmarks is not established.
-
VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos
A new benchmark, VF-Eval, measures how well multimodal LLMs check, detect, and reason about errors in AI-generated videos, and shows frontier models remain far below human performance.
-
Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiency
Suppressing "Wait"-like reflection tokens at decode time reduces reasoning token counts by 27-51% across five R1-style model families, with mixed accuracy effects.
Discussion (0). Sign in to comment.