{"id":"11ef2f67-eeb3-422c-a43e-928852411e2d","arxiv_id":"2506.09987","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MVP is a shortcut-resistant video QA benchmark of minimal video pairs; humans reach 92.9% while the best video-language model reaches 40.2%.","lead":"The authors built MVP, a 55,000-example video question benchmark where every question comes with a second, visually similar video that has the opposite correct answer. Humans score 92.9%, while the best tested video-language model scores 40.2%, showing current AI relies on shortcuts rather than physical understanding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-frame filtering may under-detect single-frame-solvable pairs, so near-random MVP scores could overstate the absence of physical understanding in VideoLLMs.","rationale":"The reader's weakest_assumption is the effectiveness of the automatic single-frame filtering and minimal-pair construction. I agree and sharpen the concern: the filtering uses 4/5 agreement of five specific open VLMs (LLaMA3.2-11B, Molmo-7B, Pixtral-12B, LLaVA-OneVision-7B, Idefics3-8B), a proxy that is itself a model class about which the paper later shows large performance differences (Tarsier-34B reaches 38.8% vs. 20.7% for LLaVA-OV-7B on the full task). Since the filters are comparable to or weaker than the evaluated models, single-frame-solvable pairs could survive. This is not merely a question of curation noise: the paper's own note that they 'probably missed some shortcuts or issues in MVP' (Section F.3) is an in-manuscript admission that the core shortcut-resistance claim is weaker than a formal guarantee. The strongest independent support is the ablation in Table 5, which shows the pairing step reduces accuracy substantially (45.4% to 27.3%), so the minimal-pair construction itself is working; but the remaining 25.1% near-random overall score requires the additional inference that the residual gap is due to physical understanding. My proposed test targets just that residual inference. I keep CONDITIONAL because the benchmark artifacts and code are promised and the test is feasible with the released data; the concern is load-bearing but not a demonstrated failure.","tokens_in":23860,"tokens_out":1388,"duration_ms":14745,"concrete_test":"Take the released MVP pair set and run the single-frame filtering with a stronger or diverse set of image-only models (e.g., InternVL2.5-8B, GPT-4o, Gemini-1.5-Pro) on one randomly sampled frame per video. Recompute the pair score for models evaluated in the paper after discarding any pair where a strong single-frame model achieves above-chance accuracy. If removing those pairs moves the best model from 40.2% toward or above 50%, the headline gap overstates the temporal/physical reasoning deficit.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The benchmark's central inference is that near-random pair scores on MVP reflect a lack of physical understanding, because the pipeline removed shortcuts including single-frame-solvable examples. That inference depends on the single-frame filtering stage being an upper bound on single-frame solvability: if a model could answer a video's question from one frame even though at most 4/5 of the five proxy VLMs did not do so, or if the proxy VLMs are weaker than the evaluated VideoLLMs at exploiting such cues, the 'minimal pair score' hypothesis is violated. The authors acknowledge they 'probably missed some shortcuts or issues in MVP' (Section F.3), and the 5-model ensemble threshold already flags 30% of frames as solvable in discarded pairs, so the fragility is not hypothetical.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MVP, a video question-answering benchmark built around minimal video pairs: each question is paired with two visually similar videos that share the same question but have opposing correct answers, and a model scores only if it answers both videos correctly. The benchmark contains 54,828 examples (27,414 pairs) drawn from nine sources spanning human-object interaction, robot-object interaction, intuitive physics, and coarse temporal reasoning. A three-stage curation pipeline first manually filters sources, then automatically mines minimal pairs using metadata, entailment rules, and ViCLIP embedding similarity, and finally removes pairs deemed solvable from a single frame by a five-VLM ensemble. The authors evaluate nine video LLMs, reporting best accuracy of 40.2% versus human performance of 92.9% and random chance of 25%. They also present a shortcut analysis of MVBench showing that language-only, video-only, single-frame, and Socratic-LLM baselines achieve non-trivial scores, and an ablation showing that minimal pairing reduces average model accuracy from 45.4% to 27.3%.","tokens_in":24121,"tokens_out":10492,"duration_ms":127337,"significance":"The benchmark addresses an important problem—score inflation in video QA due to shortcut solutions—and introduces a practical scoring scheme (minimal pair score) that penalizes models for relying on biases. The paper's strengths include a large-scale curated dataset, an automatic curation pipeline that goes beyond hand-crafted small sets, a clear ablation (Table 5) isolating the effect of minimal pairing and single-frame filtering, and evaluations across nine open and closed models. The MVBench shortcut analysis in Table 1 is a useful standalone contribution. If the validity concerns about single-frame filtering and the human baseline are resolved, MVP could become a standard robustness test for video-language models' physical understanding. The paper is honest about its limitations, including an explicit acknowledgement in Section F.3 that some shortcuts may remain.","major_comments":[{"comment":"The single-frame filter is not validated as an upper bound on single-frame solvability. The filter discards a pair only if at least 4/5 of five specified VLMs correctly answer the same randomly sampled frame for at least 30% of frames in both videos; consequently, a kept pair may contain up to 29% individually solvable frames, and a VideoLLM that is stronger than the proxy ensemble could exploit such frames. The authors' admission in Section F.3 that they 'probably missed some shortcuts or issues in MVP' makes this more than a hypothetical concern. Since the paper's central inference (near-random pair scores reflect a lack of physical understanding) depends on remaining pairs being free of single-frame shortcuts, the authors should either (a) measure the single-frame pair accuracy of the evaluated VideoLLMs on the final MVP, (b) calibrate the filter against human single-frame performance, or (c) report sensitivity to the 4/5 and 30% thresholds and temper the shortcut-robustness claim accordingly.","section":"Section 3, 'Single-frame Bias Filtering'; Section F.3"},{"comment":"The human baseline of 92.9% is a headline result but is supported by insufficient detail: Section D states only that 6 researchers annotated 'a representative subset,' without reporting the number of pairs, the selection procedure, or inter-annotator agreement. If the subset is small or unrepresentative, the claimed gap between 92.9% and 40.2% could change materially. The authors should report the full annotation protocol, the number of pairs per annotator, and agreement statistics, and ideally release the human-annotated subset with the benchmark.","section":"Section 4 (Table 4) and Section D"},{"comment":"The benchmark's central construct is the 'minimal' video pair, but minimality is operationalized only through ViCLIP cosine similarity, metadata rules, and entailment checks, without any direct validation that the final pairs differ solely in the task-relevant physical property. The ablation in Table 5 demonstrates that minimal pairing lowers scores, which is encouraging, but it does not establish that the pairs are truly minimal; pairs could still differ on other visual dimensions that models might exploit. To support the claim that MVP isolates physical understanding, the authors should add a human-rated minimality study or a quantitative measure of task-relevant difference on a sample of pairs.","section":"Section 3, 'Minimal-change Pair Mining' and Table 5"}],"minor_comments":[{"comment":"The text says Llama3-8B 'outperforms a random baseline by 6%' but the table shows a 6-percentage-point difference (0.36 vs 0.30); please use percentage points to avoid confusion with relative improvement.","section":"Section 2, Table 1"},{"comment":"There is a typo in the QAified question: 'Which action is being performed in the video¿‘' should use a standard question mark and closing quotation mark.","section":"Section B.2, Something Something v2"},{"comment":"The sentence 'it is a very exciting direction with enough enough dimensions' contains a duplicated 'enough'; please correct.","section":"Section F.3"},{"comment":"The paper states that '16% of the videos in our final benchmark (∼ 8.8K examples) already possess explicit minimal visual pairs,' but if the final benchmark contains 54.8K examples or 27.4K videos, 16% corresponds to roughly 8.8K examples, not 8.8K videos; please clarify whether the count refers to examples or videos.","section":"Section 3, 'Minimal-change Pair Mining'"},{"comment":"The term 'minimal-change' is used to mean visually similar videos, which is a weaker notion than the standard minimal-pair notion in NLP where inputs differ by a single atomic change; please state this explicitly to avoid confusion with prior work.","section":"Section 3, 'Minimal-change Pair Scoring'"},{"comment":"The row 'Any text model †' with 0.0 accuracy is a theoretical statement that assumes a deterministic text model; consider explaining this reasoning in the text rather than only in a footnote.","section":"Table 4, footnote for 'Any text model †'"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to have significant impact given the prominence of the authors and the scalability of the benchmark. The 'Behind the Scenes' section is unusual but honest; however, the admissions there (e.g., missed shortcuts, frame-rate dependence) underscore that the benchmark's robustness claims need to be backed by additional experiments rather than just acknowledged. The human evaluation is the most opaque part of the paper; as a referee, I would strongly encourage the authors to provide complete annotation details and ideally a larger, more rigorously sampled human study. The benchmark release should be checked for the exact numbers reported in the paper, especially the 16% explicit-pair statistic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper for one reason: it ships the largest minimal-change video-pair benchmark yet, 27,414 pairs mined automatically from nine sources, and then shows that nine current VideoLLMs land near random (best 40.2%) where humans hit 92.9%. That gap is the story, and it is plausible. The MVBench shortcut analysis in Section 2 is also a solid contribution, with clean baselines for language-only, video-only, single-frame, and Socratic-caption shortcuts across all 20 tasks. The ablation in Table 5 is the right kind of evidence: random pairing gives 45.4%, minimal pairs drop it to 27.3%, and single-frame filtering drops it another 2.2%. That controls the core mechanism.\n\nThe new thing here is scale and automation. Vinoground and TempCompass had a few hundred minimal pairs; MVP has tens of thousands, and the pipeline is described well enough that others can replicate the mining on new sources. I also want to credit the authors for writing an unusually honest limitations section, including the explicit admission in F.3 that they probably missed some shortcuts. That is rare and useful.\n\nWhere is the soft spot? The single-frame-bias filter is exactly what the stress-test note says: five VLMs, a 4/5 threshold, and a 30% frame-solvable cutoff. That is a proxy for single-frame solvability, not a guarantee. If those five VLMs are weaker than the evaluated VideoLLMs at exploiting static cues, some pairs pass the filter that should not. I read the stress-test concern and then checked whether it actually damages the central inference. It does not land as hard as it sounds. Under-filtering leaves pairs that are solvable from a single frame; a model with good static understanding should score above random on those. The fact that models still hover near random means they are failing on both temporal and static physical understanding. That does not overstate the absence of physical understanding; it broadens it. What it does weaken is the paper's phrasing that MVP is 'shortcut-aware' in a tight sense. The benchmarking is shortcut-resistant to the level of the proxy ensemble, not perfectly. That is a real caveat, but it is a moderate one.\n\nThe other limitation is that the data and code are not yet available, so the minimality of the pairs cannot be independently checked. The paper says they will release MVP-mini and the pipeline; until then, treat the headline gap as conditional on the artifacts confirming that the pairs are visually minimal and answer-opposite. The entailment rules and ViCLIP thresholds are reasonable, but they are heuristics.\n\nBottom line: this paper deserves a serious referee and a hearing. It is aimed at benchmark builders and video-model developers. I would cite it and bring it to a reading group. The central result is important and likely correct in direction, though the exact size of the gap may shift once the community audits the pairs. My main advice to an editor would be: send it to review, but ask the authors to release the full benchmark and the filtering pipeline before acceptance, and consider asking for a spot-check of pair minimality with a small human study.","headline":"A genuinely useful benchmark resource: the largest minimal-change video pair set to date, with a strong but heuristic filtering step that deserves scrutiny before the headline gap is taken as gospel.","tokens_in":24567,"tokens_out":4446,"would_cite":true,"duration_ms":57133,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current video-language models largely lack physical understanding, and a new 55,000-example benchmark shows it by forcing each question through a pair of visually similar videos with opposing answers.","keywords":["minimal video pairs","video question answering","shortcut robustness","physical understanding","intuitive physics","video-language models","single-frame bias","benchmark evaluation"],"falsifier":"Take a random sample of MVP pairs and test them with a newer single-frame image-language model that was not part of the five-model filtering ensemble; if such a model answers more than a third of the filtered pairs correctly from one frame, the single-frame filter was under-powered and the measured gap is exaggerated.","tokens_in":23657,"feed_emoji":"🎬","tokens_out":11834,"duration_ms":132634,"temperature":0.7,"pith_summary":"Existing video question-answering benchmarks can be gamed: simple baselines that ignore the video, ignore the question, look at one frame, or read only a short caption all score far above chance on the popular MVBench suite, so high accuracies do not establish spatio-temporal understanding. To close those loopholes, this paper introduces Minimal Video Pairs (MVP), a benchmark of 54,828 multiple-choice examples organized into 27,414 minimal-change pairs: two visually similar videos, an identical question, and answer choices that are mutually exclusive. A model receives credit only if it answers both videos in a pair correctly, which pushes shortcut-following systems toward the 25% random floor. Human annotators reach 92.9% pair accuracy, while the best open-source video-language model reaches 40.2%; the paper interprets this as evidence that current video-language models are far from robust physical understanding. The intended upshot is that video-QA progress should be measured against shortcut-resistant pair scoring rather than inflated legacy benchmarks.","feed_headline":"Video AI collapses to near chance on paired physics questions","feed_subtitle":"A 55K-video benchmark pairs each clip with a near-twin and opposite answer; humans score 92.9%, the best AI 40.2%.","key_machinery":"The load-bearing device is the minimal-change video pair: two videos that look nearly identical, share identical question text, and have mutually exclusive answer choices, so the correct label flips between the two videos. Scoring is done with the Minimal Pair Score, which awards credit only when a model answers both members correctly and therefore converts any tendency to repeat the same answer from a harmless bias into a guaranteed error. The pairs are mined automatically from nine video sources: candidates are grouped by identical or semantically matched questions, pruned by metadata rules (shared objects, antonym actions), ranked by cosine similarity of ViCLIP video embeddings, and filtered with entailment rules so the two correct answers cannot both be true. Finally, a five-model ensemble of image-language models prompts for answers from single random frames and discards any pair where at least 30% of frames in both videos are answered correctly by four of five models, removing single-frame-solvable examples.","core_discovery":"The paper's central claim is that state-of-the-art video-language models do not yet understand the physical world reliably, and that high scores on earlier spatio-temporal benchmarks are frequently attainable through shortcuts rather than through genuine video understanding. To make physical understanding measurable, MVP pairs each video with a visually similar twin that shares the same question but has the opposite correct answer, and the Minimal Pair Score requires both members of the pair to be answered correctly. Because a model latching onto superficial cues tends to give the same answer twice, such models collapse toward the 25% random floor. On the resulting 27,414 pairs, humans reach 92.9%, while the best evaluated open-source model, InternVL2.5-8B, reaches 40.2%, with close-to-random or sub-random scores on intuitive-physics and fine-grained robot-object-interaction splits. The paper concludes that current video-language models are far from human-level physical understanding, despite performing well on standard spatio-temporal reasoning benchmarks.","pith_inferences":["A likely consequence of the paper's frame-rate observation is that part of the 40.2% ceiling may be a sampling artifact: many video models represent a video as 16 uniformly sampled frames, so extending evaluation to high frame-rate sampling or temporally adaptive token selection would test how much of the gap is input-formatting rather than reasoning.","The five-model single-frame filter is a moving target: as image-only models improve, some pairs currently classified as unsolvable from a single frame may become solvable, so the benchmark's difficulty should be re-audited with newer single-frame models after release.","A side-by-side human experiment showing annotators both videos of a pair at once rather than one at a time would measure how much of the human 92.9% comes from explicitly contrasting the two scenes rather than from physical understanding of each scene alone.","The same minimal-pair machinery could be applied to other dimensions of video understanding the benchmark does not cover, such as social causality, intention, or long-range narrative structure, by mining paired clips from narrative or multi-agent video sources."],"forward_implications":["Reported accuracies on legacy video-QA suites such as MVBench should not be read as evidence of physical understanding, since the paper's text-only, video-only, single-frame, and caption-only baselines all solve substantial portions of those tasks.","Under pair scoring, the best open-source model's 40.2% accuracy is only about 15 points above the 25% random floor, and several intuitive-physics splits fall below chance, meaning even top models do not reliably track object permanence, gravity, or collisions.","Minimal video pairs can be produced automatically from existing datasets at scale, 27,414 pairs from nine sources, so shortcut-resistant evaluation does not require small hand-crafted contrast sets.","Near-random scores on fine-grained robot-object interactions complicate the use of video-language models as large-scale visuomotor policy learners, a concern the paper raises explicitly."],"supporting_citations":[{"why":"Supplies MVBench, the 20-task suite whose shortcut susceptibility motivates the benchmark, and the VideoChat2 model evaluated on MVP.","marker":"Li et al., 2024b"},{"why":"Provides the Winoground group-score idea from which the paper adapts the Minimal Pair Score.","marker":"Thrush et al., 2022"},{"why":"Establishes the minimal-change and adversarial question principle that the paired video design builds on.","marker":"Levesque et al., 2012"},{"why":"Supplies the ViCLIP video encoder whose cosine similarity ranks candidate minimal pairs during mining.","marker":"Wang et al., 2023b"},{"why":"Provides the Llama 3 language models used for language-only baselines and as members of the single-frame filtering ensemble.","marker":"Dubey et al., 2024"},{"why":"Provides the Idefics3 model used as a single-frame baseline and as a member of the single-frame filtering ensemble.","marker":"Laurençon et al., 2024"},{"why":"Supplies the Perception Test, one of the nine video sources contributing fine-grained human-object interaction examples.","marker":"Patraucean et al., 2023"},{"why":"Supplies CLEVRER, source of the intuitive-physics and counterfactual examples whose problematic counterfactual split is filtered out.","marker":"Yi* et al., 2020"}],"fun_headline_variants":["Video AI scores near random on twin-video physics test","55K paired videos reveal AI's missing physical understanding","Physical reasoning benchmark drops best video model to 40%","Human vs AI: 92.9% to 40.2% on video physics pairs","Shortcut-free video pairs push AI to 25% random floor"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central bet is that the automatic pairing and filtering pipeline really produces minimal pairs, visually and textually near-identical aside from the physical event being tested and not answerable from any single frame, so the low model scores measure missing physical understanding rather than noisy or non-minimal data.","fun_headline_variants_meta":{"raw":{"variants":["Video AI scores near random on twin-video physics test","55K paired videos reveal AI's missing physical understanding","Physical reasoning benchmark drops best video model to 40%","Human vs AI: 92.9% to 40.2% on video physics pairs","Shortcut-free video pairs push AI to 25% random floor"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1571,"prompt_tokens":969,"completion_tokens":602,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":512}},"tokens_in":585,"tokens_out":602,"duration_ms":7347,"temperature":1.0,"reasoning_tokens":512,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:35:12.761591+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of MVP pairs and test them with a newer single-frame image-language model that was not part of the five-model filtering ensemble; if such a model answers more than a third of the filtered pairs correctly from one frame, the single-frame filter was under-powered and the measured gap is exaggerated.","supporting_citations":[],"review_version":1}