REVIEW 3 major objections 6 minor 3 cited by
A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Current video-language models largely lack physical understanding, and a new 55,000-example benchmark shows it by forcing each question through a pair of visually similar videos with opposing answers.
desk verdict A genuinely useful benchmark resource: the largest minimal-change video pair set to date, with a strong but heuristic filtering step that deserves scrutiny before the headline gap is taken as gospel. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing device is the minimal-change video pair: two videos that look nearly identical, share identical question text, and have mutually exclusive answer choices, so the correct label flips between the two videos. Scoring is done with the Minimal Pair Score, which awards credit only when a model answers both members correctly and therefore converts any tendency to repeat the same answer from a harmless bias into a guaranteed error. The pairs are mined automatically from nine video sources: candidates are grouped by identical or semantically matched questions, pruned by metadata rules (shared objects, antonym actions), ranked by cosine similarity of ViCLIP video embeddings, and filtered with entailment rules so the two correct answers cannot both be true. Finally, a five-model ensemble of image-language models prompts for answers from single random frames and discards any pair where at least 30% of frames in both videos are answered correctly by four of five models, removing single-frame-solvable examples.
What would settle it
Take a random sample of MVP pairs and test them with a newer single-frame image-language model that was not part of the five-model filtering ensemble; if such a model answers more than a third of the filtered pairs correctly from one frame, the single-frame filter was under-powered and the measured gap is exaggerated.
Extended reading notes
Core claim
The paper's central claim is that state-of-the-art video-language models do not yet understand the physical world reliably, and that high scores on earlier spatio-temporal benchmarks are frequently attainable through shortcuts rather than through genuine video understanding. To make physical understanding measurable, MVP pairs each video with a visually similar twin that shares the same question but has the opposite correct answer, and the Minimal Pair Score requires both members of the pair to be answered correctly. Because a model latching onto superficial cues tends to give the same answer twice, such models collapse toward the 25% random floor. On the resulting 27,414 pairs, humans reach 92.9%, while the best evaluated open-source model, InternVL2.5-8B, reaches 40.2%, with close-to-random or sub-random scores on intuitive-physics and fine-grained robot-object-interaction splits. The paper concludes that current video-language models are far from human-level physical understanding, despite performing well on standard spatio-temporal reasoning benchmarks.
Load-bearing premise
The central bet is that the automatic pairing and filtering pipeline really produces minimal pairs, visually and textually near-identical aside from the physical event being tested and not answerable from any single frame, so the low model scores measure missing physical understanding rather than noisy or non-minimal data.
Editorial extensions
If this is right
- Reported accuracies on legacy video-QA suites such as MVBench should not be read as evidence of physical understanding, since the paper's text-only, video-only, single-frame, and caption-only baselines all solve substantial portions of those tasks.
- Under pair scoring, the best open-source model's 40.2% accuracy is only about 15 points above the 25% random floor, and several intuitive-physics splits fall below chance, meaning even top models do not reliably track object permanence, gravity, or collisions.
- Minimal video pairs can be produced automatically from existing datasets at scale, 27,414 pairs from nine sources, so shortcut-resistant evaluation does not require small hand-crafted contrast sets.
- Near-random scores on fine-grained robot-object interactions complicate the use of video-language models as large-scale visuomotor policy learners, a concern the paper raises explicitly.
Reading between the lines
- A likely consequence of the paper's frame-rate observation is that part of the 40.2% ceiling may be a sampling artifact: many video models represent a video as 16 uniformly sampled frames, so extending evaluation to high frame-rate sampling or temporally adaptive token selection would test how much of the gap is input-formatting rather than reasoning.
- The five-model single-frame filter is a moving target: as image-only models improve, some pairs currently classified as unsolvable from a single frame may become solvable, so the benchmark's difficulty should be re-audited with newer single-frame models after release.
- A side-by-side human experiment showing annotators both videos of a pair at once rather than one at a time would measure how much of the human 92.9% comes from explicitly contrasting the two scenes rather than from physical understanding of each scene alone.
- The same minimal-pair machinery could be applied to other dimensions of video understanding the benchmark does not cover, such as social causality, intention, or long-range narrative structure, by mining paired clips from narrative or multi-agent video sources.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MVP, a video question-answering benchmark built around minimal video pairs: each question is paired with two visually similar videos that share the same question but have opposing correct answers, and a model scores only if it answers both videos correctly. The benchmark contains 54,828 examples (27,414 pairs) drawn from nine sources spanning human-object interaction, robot-object interaction, intuitive physics, and coarse temporal reasoning. A three-stage curation pipeline first manually filters sources, then automatically mines minimal pairs using metadata, entailment rules, and ViCLIP embedding similarity, and finally removes pairs deemed solvable from a single frame by a five-VLM ensemble. The authors evaluate nine video LLMs, reporting best accuracy of 40.2% versus human performance of 92.9% and random chance of 25%. They also present a shortcut analysis of MVBench showing that language-only, video-only, single-frame, and Socratic-LLM baselines achieve non-trivial scores, and an ablation showing that minimal pairing reduces average model accuracy from 45.4% to 27.3%.
Significance. The benchmark addresses an important problem—score inflation in video QA due to shortcut solutions—and introduces a practical scoring scheme (minimal pair score) that penalizes models for relying on biases. The paper's strengths include a large-scale curated dataset, an automatic curation pipeline that goes beyond hand-crafted small sets, a clear ablation (Table 5) isolating the effect of minimal pairing and single-frame filtering, and evaluations across nine open and closed models. The MVBench shortcut analysis in Table 1 is a useful standalone contribution. If the validity concerns about single-frame filtering and the human baseline are resolved, MVP could become a standard robustness test for video-language models' physical understanding. The paper is honest about its limitations, including an explicit acknowledgement in Section F.3 that some shortcuts may remain.
major comments (3)
- [Section 3, 'Single-frame Bias Filtering'; Section F.3] The single-frame filter is not validated as an upper bound on single-frame solvability. The filter discards a pair only if at least 4/5 of five specified VLMs correctly answer the same randomly sampled frame for at least 30% of frames in both videos; consequently, a kept pair may contain up to 29% individually solvable frames, and a VideoLLM that is stronger than the proxy ensemble could exploit such frames. The authors' admission in Section F.3 that they 'probably missed some shortcuts or issues in MVP' makes this more than a hypothetical concern. Since the paper's central inference (near-random pair scores reflect a lack of physical understanding) depends on remaining pairs being free of single-frame shortcuts, the authors should either (a) measure the single-frame pair accuracy of the evaluated VideoLLMs on the final MVP, (b) calibrate the filter against human single-frame performance, or (c) report sensitivity to the 4/5 and 30% thresholds and temper the shortcut-robustness claim accordingly.
- [Section 4 (Table 4) and Section D] The human baseline of 92.9% is a headline result but is supported by insufficient detail: Section D states only that 6 researchers annotated 'a representative subset,' without reporting the number of pairs, the selection procedure, or inter-annotator agreement. If the subset is small or unrepresentative, the claimed gap between 92.9% and 40.2% could change materially. The authors should report the full annotation protocol, the number of pairs per annotator, and agreement statistics, and ideally release the human-annotated subset with the benchmark.
- [Section 3, 'Minimal-change Pair Mining' and Table 5] The benchmark's central construct is the 'minimal' video pair, but minimality is operationalized only through ViCLIP cosine similarity, metadata rules, and entailment checks, without any direct validation that the final pairs differ solely in the task-relevant physical property. The ablation in Table 5 demonstrates that minimal pairing lowers scores, which is encouraging, but it does not establish that the pairs are truly minimal; pairs could still differ on other visual dimensions that models might exploit. To support the claim that MVP isolates physical understanding, the authors should add a human-rated minimality study or a quantitative measure of task-relevant difference on a sample of pairs.
minor comments (6)
- [Section 2, Table 1] The text says Llama3-8B 'outperforms a random baseline by 6%' but the table shows a 6-percentage-point difference (0.36 vs 0.30); please use percentage points to avoid confusion with relative improvement.
- [Section B.2, Something Something v2] There is a typo in the QAified question: 'Which action is being performed in the video¿‘' should use a standard question mark and closing quotation mark.
- [Section F.3] The sentence 'it is a very exciting direction with enough enough dimensions' contains a duplicated 'enough'; please correct.
- [Section 3, 'Minimal-change Pair Mining'] The paper states that '16% of the videos in our final benchmark (∼ 8.8K examples) already possess explicit minimal visual pairs,' but if the final benchmark contains 54.8K examples or 27.4K videos, 16% corresponds to roughly 8.8K examples, not 8.8K videos; please clarify whether the count refers to examples or videos.
- [Section 3, 'Minimal-change Pair Scoring'] The term 'minimal-change' is used to mean visually similar videos, which is a weaker notion than the standard minimal-pair notion in NLP where inputs differ by a single atomic change; please state this explicitly to avoid confusion with prior work.
- [Table 4, footnote for 'Any text model †'] The row 'Any text model †' with 0.0 accuracy is a theoretical statement that assumes a deterministic text model; consider explaining this reasoning in the text rather than only in a footnote.
Circularity Check
No circularity: MVP scores are independent measurements on a fixed external benchmark; curation heuristics are not fitted predictions.
full rationale
MVP is a benchmark-construction paper, and its derivation chain is not circular. The benchmark is assembled from nine external video sources using fixed, a priori curation rules: symbolic and neural similarity matching via ViCLIP, entailment checks, and a single-frame filter requiring at least 4 of 5 fixed single-image VLMs to answer correctly on 30% of frames before a pair is discarded (Section 3). The headline result, VideoLLM scores of roughly 25-40% versus 92.9% human, is an independent measurement on the final fixed benchmark. No parameter is fitted to the evaluated models and then renamed as a prediction; the curation thresholds are not optimized against any VideoLLM's score, and the model performances are reported after the benchmark is frozen. The only feedback loop is that the single-frame filter uses VLMs to define 'single-frame solvable', and one filter member (LLaVA-OneVision) is later evaluated, but this is a design choice about benchmark construction, not a circular derivation: the filter decides by ensemble agreement on single frames, the evaluation feeds full videos with different prompts, and the main gap is reproduced by models outside the filter (GPT-4o, Gemini, Tarsier-34B, InternVL2.5). The paper explicitly flags the corresponding validity risk ('Despite our best efforts studying shortcuts, we probably missed some shortcuts or issues in MVP', Section F.3), which is a correctness concern, not a circularity. Self-citations (Winoground, Krojer et al., Maharaj et al.) are related-work attributions, not load-bearing: the minimal-pair idea is also credited to Levesque et al. and Sakaguchi et al., and no uniqueness claim rests on the authors' own prior work.
Assumptions & free parameters
free parameters (4)
- ViCLIP cosine similarity threshold for Language Table pairing =
0.9
- Top-k candidate pairs per question =
50 for Perception Test/STAR, 4000 for Something-Something v2
- Single-frame solvable threshold =
30% of frames in both videos with 4-of-5 VLM agreement
- Token difference threshold for Language Table answers =
0 < token_diff < 4
assumptions (4)
- domain assumption ViCLIP embedding cosine similarity is a valid proxy for visual minimality between two videos.
- domain assumption A sample is single-frame-solvable if 4 of 5 VLMs answer correctly from the same single frame.
- domain assumption Manual inspection of a subset is sufficient to keep noise low in the final benchmark.
- domain assumption The 6-researcher human baseline is representative of human performance.
Cite this review
Pith. "Pith review of A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs." pith.science (2026). https://pith.science/paper/LOCLVP4S
@misc{pith2026250609987,
author = {Pith},
title = {Pith review of: A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs},
year = {2026},
howpublished = {\url{https://pith.science/paper/LOCLVP4S}},
note = {Machine review of arXiv:2506.09987}
}
read the original abstract
Existing benchmarks for assessing the spatio-temporal understanding and reasoning abilities of video language models are susceptible to score inflation due to the presence of shortcut solutions based on superficial visual or textual cues. This paper mitigates the challenges in accurately assessing model performance by introducing the Minimal Video Pairs (MVP) benchmark, a simple shortcut-aware video QA benchmark for assessing the physical understanding of video language models. The benchmark is comprised of 55K high-quality multiple-choice video QA examples focusing on physical world understanding. Examples are curated from nine video data sources, spanning first-person egocentric and exocentric videos, robotic interaction data, and cognitive science intuitive physics benchmarks. To mitigate shortcut solutions that rely on superficial visual or textual cues and biases, each sample in MVP has a minimal-change pair -- a visually similar video accompanied by an identical question but an opposing answer. To answer a question correctly, a model must provide correct answers for both examples in the minimal-change pair; as such, models that solely rely on visual or textual biases would achieve below random performance. Human performance on MVP is 92.9\%, while the best open-source state-of-the-art video-language model achieves 40.2\% compared to random performance at 25\%.
Forward citations
Cited by 3 Pith papers
-
AdaThinkV: Adaptive Thinking for Token-Efficient Video Reasoning
A video reasoning model learns per question whether to reason aloud or answer directly, improving accuracy by about 3 points over the best adaptive baseline while using about 23% fewer output tokens.
-
FADE: From Passive Verification to Active Discovery in Counterfactual Video Understanding
FADE trains a video MLLM with evidence-internalized SFT plus fading-anchor RL, preserving counterfactual judgment accuracy when MCQ guidance is removed, with 90.4% and 67.4% retention on OQA and captioning on DualityV...
-
Embodied AI Agents: Modeling the World
Embodied AI agents should be built around physical world models plus a mental world model of the user, with virtual, wearable, and robotic agents sharing this core.
Reference graph
Works this paper leans on
-
[1]
Group QA examples into sets with the same ques- tion: P={X| ∀x i, xj ∈X, qi ≡q j}
-
[2]
For a given X, examples xi and xj are grouped into potential pairs if they have opposite (mutually exclusive) correct answers: P ′ ={(x i, xj)|a i ̸=a j}
-
[3]
From this set of potential pairsP ′, we choose the top-k for a given question based on visual similar- ity, measured via cosine similarity of embeddings from the video encoder ViCLIP-ViT-L (Wang et al., 2023b): Pk = (xi1 , xj1 ), . . . ,(xik , xjk ) sim(vim , vjm )≥sim(v im+1 , vjm+1 ) . In practice, we choosek= 50for each question. Additionally, we use d...
-
[4]
At the same time, Something Something v2 truly stood the test of time as a great video understanding dataset due to its low noise-ratio at a scale of 200K examples, and coverage of interesting yet simple phenomena
-
[5]
We group QA examples into sets such that a) both correct answers mention the same objects (e.g., both involve a “red triangle” and “green heart”) and b) the set of tokens inai and aj have a large enough overlap: P= (ai, aj) obj(ai)≡obj(a j)∧ 0<token_diff(a i, aj)<4 . Due to the finite number of attributes and objects in Language Table,obj(·)checks for the...
-
[6]
We narrow this set of potential pairsP with a visual similarity threshold, measured via cosine similarity of embeddings from the video encoder ViCLIP-ViT- L (Wang et al., 2023b): P ′ ={(x i, xj)|sim(v i, vj)>0.9}
-
[7]
Move yel- low triangle to blue heart
Finally, we ensure that answers are mutually exclu- sive, i.e., ai ̸ =⇒a j and aj ̸ =⇒a i. In practice, this involves several hand-crafted rules after inspect- ing failure cases: If the order of objects mentioned is different, there is no entailment (e.g., “Move yel- low triangle to blue heart” and “Move blue heart to yellow triangle”); if otherwise one a...
-
[8]
spinning something so it continues spinning
We group QA examplesai and aj into pairs such that the action inai is a well-defined antonym of the action inaj: P={(a i, aj)|antonym(a i, aj)}. In practice, we identify a subset of 82 action types (47% of all actions) that have a well-defined opposite, e.g., “spinning something so it continues spinning” and “spinning something that quickly stops spin- ning”
Show all 19 references
-
[9]
If no pairs fulfill this strict criterion, we relax it such that only one object must overlap: P ′ ={(x i, xj)|obj(v i)∩obj(v j)̸=∅}
We narrow down pairs further by selecting a pair xi and xj if the videos contain the same object(s) based on their captions: P ′ ={(x i, xj)|obj(v i)≡obj(v j)}. If no pairs fulfill this strict criterion, we relax it such that only one object must overlap: P ′ ={(x i, xj)|obj(v...
-
[10]
,(xik , xjk ) sim(vim, vjm)≥sim(v im+1 , vjm+1 )
From this set of potential pairsP ′, we choose the top-k based on visual similarity, measured via cosine similarity of embeddings from the video encoder ViCLIP-ViT-L (Wang et al., 2023b): Pk = (xi1 , xj1 ), . . . ,(xik , xjk ) sim(vim, vjm)≥sim(v im+1 , vjm+1 ) . In practice, ...
-
[11]
Group QA examples into sets with the same ques- tion: P={X| ∀x i, xj ∈X, qi ≡q j}. 2
-
[12]
How many objects are moving when the video ends? A) 2 B) 3
For a given X, examples xi and xj are grouped into potential pairs if they have opposite (mutually exclusive) correct answers: P ′ ={(x i, xj)|a i ̸=a j}. In the special case that the answers are both numer- ical, we require them both to be 1 apart, e.g., “How many objects are...
-
[13]
fuzzy subset
We further filter the set P ′ by requiring a large overlap of objects with the exact same attributes in both videos. Specifically, we keep a pair if the set of objects invi is a “fuzzy subset” of the objects in vj, or vice versa: P ′′ ={(x i, xj)|fuzzy_subset(obj(v i),obj(v j)...
-
[14]
There is too many edge cases to catch every single one at this scale of data curation
-
[15]
Paper writing is smooth when the story and con- tribution is clear from the beginning of the project (This was not the case in the first author’s last paper so it was nice to observe the contrast)
-
[16]
The intuitive physics datasets are (to the subjective taste of the first author) the cleanest and most fascinating sources inMVP
-
[18]
In order to push the field further we are now asking the models more and more nuanced questions, and the answer may lie only in a short span of a less than second
Frame rate plays a big role for solving many exam- ples inMVP. In order to push the field further we are now asking the models more and more nuanced questions, and the answer may lie only in a short span of a less than second. However many models may not have access to this sh...
-
[19]
Parsing outputs from LLMs into a structured for- mat such as answer options can feel like the wild west sometimes: Could models perform better if we prompt them better, or have more flexible ways of extracting the answer? F.3 Advice for others working on a similar direc- tion ...
2015
-
[2018]
Where is the person?
Association for Computational Linguistics. doi: 10. 18653/v1/D18-1167. https://aclanthology.org/D18-1167. Jie Lei, Tamara Berg, and Mohit Bansal. Revealing sin- gle frame bias for video-and-language learning. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Pro...
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.