Pith. sign in

REVIEW 3 major objections 4 minor 127 references

VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that current multimodal LLMs, despite strong object recognition, systematically lack 'visual knowledge'—the intuitive grasp of physical and social principles—and quantifies the deficit with a new video benchmark, VKnowU.

desk verdict Useful benchmark and a plausible training recipe, but the headline human-gap is not apples-to-apples and the main VKnowU gain is in-sample on lambda. read the letter →

arxiv 2511.20272 v2 pith:XVB2T5XC submitted 2025-11-25 cs.CV

Tianxiang Jiang , Sheng Xia , Yicheng Xu , Linquan Wu , Xiangyu Zeng , Limin Wang , Yu Qiao , Yi Wang This is my paper
classification cs.CV
keywords visualknowledgemultimodalLLMbenchmarkvideounderstandingintuitivephysicsspatialawarenesstheoryofmindreinforcementlearningVKnowU
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to measure, and then to improve, a specific blind spot in multimodal large language models: not what objects are in a video, but what the scene implies about physics, materials, spatial layout, intentions, and social relations. It introduces VKnowU, 1,680 multiple-choice questions over 1,249 videos in eight categories split into world-centric and human-centric knowledge. Across 23 state-of-the-art models, the strongest reach 74.6% accuracy versus 89.6% human performance, a 15-point gap driven mostly by intuitive physics and spatial awareness, where models barely exceed chance. To show the deficit is learnable, the paper contributes VKnowQA training data and VideoKnow+, a baseline that forces models to describe visual evidence before answering and trains with a visual-knowledge reward; it gains 3.7 points on VKnowU and transfers to other video benchmarks. The point is that 'seeing' in the pixel sense is not the same as understanding the invisible scaffolding that lets humans predict and interpret scenes.

What carries the argument

The central object is VKnowU, a video multiple-choice benchmark built to isolate visual knowledge from audio and language shortcuts. Its QA pipeline has four steps: transcribe audio with Whisper and drop questions answerable from subtitles; run three text-only LLMs in a blind-VQA setting and remove questions answered correctly by at least two; use DeepSeek-R1 to rewrite wrong options into plausible distractors; then manually verify and shuffle options. The companion mechanism is VideoKnow+, whose See-Think-Answer format forces a self-contained visual description before reasoning, and whose binary visual knowledge reward uses a frozen MLLM verifier to check that the answer follows from the de

What would settle it

Give the released VKnowU items to a fresh human panel under the same multiple-choice format and compute accuracy per task; if aggregate human accuracy no longer hovers near 89.6%, particularly if intuitive physics or spatial awareness scores drop, the central gap claim fails. A second check: if a text-only LLM, after the distractor rewriting, answers above the blind-VQA threshold on many items, the language-bias filtering is incomplete.

Watch

Extended reading notes

Core claim

On VKnowU, the paper finds that visual knowledge is a distinct, measurable capability in which current MLLMs are deficient: the best model scores 74.6% overall against 89.6% for humans, with the widest gaps in Intuitive Physics (59.0% vs 97.5%, a 38.5-point deficit) and Spatial Awareness (62.7% vs 95.4%, a 32.7-point deficit). Human-centric tasks are closer (best 81.9% vs 86.8%), which the authors attribute to abundant social supervision in pretraining. The paper further claims visual knowledge is learnable: VideoKnow+, trained on VKnowQA with a See-Think-Answer output format and a GRPO reward that verifies whether the model's visual description alone determines the answer, improves 3.7% on

Load-bearing premise

The claimed 15% human-performance gap rests on using human accuracy from the original annotation sources as the reference for final multiple-choice items whose wrong answers were rewritten by DeepSeek-R1 and whose options were reshuffled; if humans perform differently on those rewritten items, the headline gap is not apples-to-apples.

Editorial extensions

If this is right

  • Current MLLMs' object recognition does not entail understanding of physical laws: state-of-the-art models are near random on intuitive physics and spatial relations.
  • Model rankings on VKnowU cluster into independent world-centric and human-centric competencies; improving one does not automatically improve the other.
  • Visual knowledge is trainable: a relatively small RL stage on 30K questions yields consistent gains on VKnowU and transfers to several standard video benchmarks.
  • Long chain-of-thought can hurt: on straightforward visual-knowledge questions, reasoning-heavy models sometimes underperform due to repetition and ungrounded speculation.
  • Evaluation protocol matters: after filtering, text-only LLM blind accuracy drops to near chance on the QA pool, suggesting the remaining questions require visual evidence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reported 15-point human gap should be re-verified on the final rewritten items; humans were scored on original annotations, not on the DeepSeek-R1-distractor versions, so some of the gap may be an artifact of the rewriting rather than a pure model deficit.
  • Editorial inference: the polarization between proprietary and open-source models—physics-heavy strengths versus social-heavy strengths—suggests that visual-knowledge benchmarks could be used as reward data to rebalance pretraining curricula.
  • Editorial inference: the scarcity of world-centric supervision in standard image-text pretraining points to synthetic simulation and world models as a testable source of data for intuitive physics and spatial awareness.
  • Editorial inference: if future models are trained to maximize VKnowU with task-specific weights, the largest headroom is in world-centric tasks, so aggregated scores across the eight categories may hide uneven gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces VKnowU, a video multiple-choice benchmark of 1,680 questions across 1,249 videos, designed to evaluate eight types of visual knowledge grouped into world-centric (Intuitive Physics, Object Affordance, Object Material, Spatial Awareness) and human-centric (Event Anticipation, Mental State, Social Relation, Subjective Intention) categories. The benchmark is constructed by filtering six existing datasets through audio-reliance minimization, blind-LLM language-bias filtering, DeepSeek-R1 distractor rewriting, and human review. The authors evaluate 23 MLLMs, reporting a 15.0% overall gap relative to 'Human Performance' inherited from source annotations, with larger gaps on world-centric tasks. They also present VKnowQA (30K and 12K training sets) and a baseline model, VideoKnow+, built on Qwen2.5-VL-7B with an SFT cold start and GRPO with a visual-knowledge reward, reporting +3.7% on VKnowU and gains on MVBench, Video-MME, MMVU, and VSI-Bench.

Significance. If the benchmark and training claims hold, VKnowU would be a useful diagnostic for a relatively underexplored capability, and the See-Think-Answer + visual-knowledge-reward recipe would be a constructive baseline for future work. The paper's strengths include a thoughtful multi-stage filtering pipeline, a broad evaluation of current MLLMs, the inclusion of both open and proprietary models, and the public release of code and data. However, the headline claims rest on three load-bearing methodological decisions that currently compromise the conclusions: the human baseline is not measured on the final rewritten items, the main reward weight is tuned on the evaluation benchmark itself, and the training data overlap with the evaluation benchmark at the source-family level. These issues are fixable, but they must be addressed before the quantitative claims can be accepted.

major comments (3)
  1. [§3.3 (Steps III–IV), Table 1, §3.4] The human performance row in Table 1 is footnoted as 'sourced from original annotation or researchers' responses.' Because Step III rewrites distractors with DeepSeek-R1 to make them more plausible and Step IV shuffles options, the human scores were obtained on different, likely easier, items than the final VKnowU questions. The 15.0% overall gap and the per-task deficits (IP −38.5%, SA −32.7%) are therefore not apples-to-apples comparisons. No human-rater protocol, sample size, or inter-annotator agreement is reported. The authors should either re-measure human accuracy on the final VKnowU items or explicitly restrict the human-comparison claim to the original annotation conditions.
  2. [Table 4 vs. Table 2, Eq. (1)] The visual-knowledge reward weight λ is selected by training on VKnowU itself: Table 4 sweeps λ using 1K RL steps on VKnowU, the best value (0.1) is then used for the final VideoKnow+, and Table 2 reports the resulting +3.7% improvement on the same benchmark. This makes the headline gain in-sample with respect to hyperparameter selection. The authors should hold out a validation split of VKnowU (or a disjoint development set) for λ selection and report results on a separate test split, or otherwise justify a fixed λ chosen without reference to VKnowU performance.
  3. [§4.1, Table 9 vs. Table 6; §5.1] VKnowQA-30K is drawn from dataset families that overlap with VKnowU's sources: VKnowU uses IntPhys2, VSI-Bench, Social-IQ-2.0, PACS, VLEP, and RexTime, while VKnowQA-30K includes Intphys, VSI-100K, and Social-IQ, among others. Deduplication prevents exact item overlap but does not prevent the model from learning family-specific regularities that transfer to VKnowU. The claimed +3.7% improvement on VKnowU may therefore partly reflect training on the same distributions rather than general visual-knowledge learning. The authors should demonstrate transfer to evaluation tasks whose source families are entirely absent from VKnowQA, or ablate VKnowQA sources to show the gain survives when family-overlapping sources are removed.
minor comments (4)
  1. [Abstract vs. §3.4] The abstract says '28 SOTA MLLMs,' while Section 3.4 and Table 1 evaluate 23 models. Please reconcile.
  2. [§3.3 Step IV] The human verification step is described only as 'comprehensive human review.' Provide details: number of annotators, instructions, whether disagreements were adjudicated, and agreement statistics.
  3. [Tables 1, 2, 3] Some table cells are malformed in the text (e.g., 'LLaV A-OV-7B' spacing, Table 3 rows lacking labels). Please check the compiled tables for typographical errors.
  4. [§5.2] The ablation in Table 3 is informative, but the text states that GRPO alone repeats descriptions in SA; consider showing a quantitative measure (e.g., output length or repetition rate) to support this claim.

Circularity Check

1 steps flagged · score 4.0 of 10

VideoKnow+'s +3.7% VKnowU gain is selected on VKnowU itself; the human-gap baseline is carried over from pre-rewrite annotations, though the core benchmark construction is otherwise independent.

  1. fitted input called prediction [Section 5.2, Table 4; Table 5; Table 2; Eq. (1)]
    "Choice of Visual Knowledge Reward Ratio λ. We study the impact of varying λ, which controls the weight of the visual knowledge reward rv, by training 1K RL steps for rapid exploration. As shown in Table 4, VideoKnow+’s performance on VKnowU is sensitive to this parameter. The best performance 66.79 is achieved at λ=0.1..."

    λ is tuned by VKnowU accuracy, and the verifier model in Table 5 is also chosen by VKnowU accuracy; Table 2 then reports VideoKnow+’s VKnowU score (67.7, +3.7%) for that tuned configuration. The benchmark is thus used both to select the configuration and to claim the improvement, so the reported gain is not a held-out result; it is the same quantity that was optimized, not an independent prediction.

full rationale

The load-bearing VKnowU benchmark construction is not circular: the QA pipeline (Whisper audio filtering, blind-LLM language-bias filtering, DeepSeek-R1 distractor rewriting, human review) is a curation procedure, and the 23-model evaluation is an external measurement against curated items. Appendix A's Bayes decomposition is standard conditioning and is not used to derive any fitted number. The circular element is confined to the VideoKnow+ validation: λ in Eq. (1) and the verifier model are selected by accuracy on VKnowU itself (Tables 4 and 5), and the same VKnowU set is then used to report the final '+3.7%' gain (Table 2). That makes the gain an in-sample, selection-optimized figure, not an independent estimate of VKnowU performance. Separately, I flag a missing-measurement limitation in Table 1: the human-performance row is 'sourced from original annotation or researchers' responses', but Step III rewrote distractors and Step IV shuffled options before evaluation, so the 15.0% human gap is not re-measured on the final items; this is a validity risk for the headline comparison, not a construction-level circularity. No load-bearing self-citation chain appears; the authors' prior video-model citations are contextual.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The headline claims depend on a transferred human baseline and several hand-set filtering thresholds, while the training claim depends on lambda tuned on VKnowU. No new physical or conceptual entity is introduced beyond the benchmark and dataset artifacts, so the invented-entity ledger is empty.

free parameters (4)
  • lambda (visual knowledge reward weight) = 0.1
    Selected by sweeping on VKnowU (Table 4); the final model uses lambda=0.1, so the reported VKnowU gain is partly tuned on the evaluation set.
  • audio-similarity threshold = 0.3
    Hand-set cutoff for dropping QAs whose Whisper subtitles are semantically similar to the answer (Section 3.3.I).
  • blind-VQA answerability threshold = >5 of 10 correct, flagged by at least 2 of 3 LLMs
    Hand-set criterion for removing language-answerable questions (Section 3.3.II).
  • training frame count and resolution = 16 frames at 128x28x28 for training; 32 frames at 256x28x28 for evaluation
    Chosen for efficiency; frame count affects task accuracy non-monotonically (Figure 6), so this choice can influence reported results.
assumptions (5)
  • domain assumption Visual knowledge is a coherent, separable construct with eight categories.
    Defined in Section 3.1 from cognitive psychology; no independent operationalization beyond the benchmark itself.
  • domain assumption Human labels from original source datasets remain valid after DeepSeek-R1 rewrites options and shuffles order.
    Table 1 human performance is 'sourced from original annotation or researchers' responses,' but VKnowU options are rewritten in Section 3.3.III; human accuracy on the final items was not re-measured.
  • domain assumption If blind text-only LLMs cannot answer a question, it is not language-answerable.
    Section 3.3.II assumes the three chosen LLMs are a sufficient proxy for all language-prior shortcuts available to evaluated MLLMs.
  • domain assumption The frozen MLLM verifier's judgment that a description is sufficient to derive the answer is a valid reward signal.
    Section 4, Equation (1): r_v is generated by Qwen2.5-VL-7B, the same model family as the base policy; reward correctness is not independently measured.
  • standard math Bayes decomposition P(T_a | V, T_q) proportional to P(V | T_a, T_q) * P(T_a | T_q).
    Appendix A: mathematically fine as a proportionality, but the normative claim that increasing P(V | T_a, T_q) reduces hallucination is an additional behavioral assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs." pith.science (2026). https://pith.science/paper/XVB2T5XC

@misc{pith2026251120272,
  author       = {Pith},
  title        = {Pith review of: VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XVB2T5XC}},
  note         = {Machine review of arXiv:2511.20272}
}
read the original abstract

While Multimodal Large Language Models (MLLMs) have become adept at recognizing objects, they often lack the intuitive, human-like understanding of the world's underlying physical and social principles. This high-level vision-grounded semantics, which we term visual knowledge, forms a bridge between perception and reasoning, yet remains an underexplored area in current MLLMs. To systematically evaluate this capability, we present VKnowU, a comprehensive benchmark featuring 1,680 questions in 1,249 videos, covering 8 core types of visual knowledge spanning both world-centric (e.g., intuitive physics) and human-centric (e.g., subjective intentions). Evaluation of 28 SOTA MLLMs reveals that leading models still fall short of human performance, with particularly notable gaps in the world-centric. To bridge this gap, we introduce a new dataset, VKnowQA, and VideoKnow+, a baseline model that explicitly incorporates visual knowledge into MLLMs. VideoKnow+ follows a structured See-Think-Answer paradigm and adopts reinforcement learning with visual knowledge reward, achieving a +3.7% improvement on VKnowU and consistent gains on MVBench (+5.4%), Video-MME (+7.0%), and MMVU (+5.7%). Our work highlights visual knowledge as a missing cornerstone for developing more generalizable MLLMs that can not only see but also truly understand our worlds.

Figures

Figures reproduced from arXiv: 2511.20272 by the authors.

Figure 1
Figure 1. VKnowU systematically evaluates visual knowledge understanding of MLLMs across world-centric and human-centric tasks, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of the VKnowU. Representative videos and QA pairs are shown for each of the 8 tasks. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. QA filtering pipeline used to construct VKnowU, removing non-visual shortcuts and ensuring each QA requires visual knowledge. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (22 more)
Figure 4
Figure 4. Figure 4: Radar chart of MLLM accuracy on VKnowU. in physical perception, which is difficult to capture ex￾haustively through textual descriptions alone. The results suggest that the standard next-word prediction training objective may not provide sufficient supervisory signals …
Figure 5
Figure 5. Figure 5: Pearson correlation among the 8 visual knowledge tasks. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: Text-only LLM accuracy on VKnowU QA pool. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Average lengths of question and options (in words) in [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Average video duration (in seconds) in VKnowU. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Correlation between visual knowledge in VKnowU and existing MLLM benchmarks. [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 12
Figure 12. Figure 12: Word length distribution visualizations of VKnowQA [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Word cloud visualization of VKnowQA-30K. [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]
Figure 14
Figure 14. Figure 14: Word cloud visualization of VKnowQA-CS-12K [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 11
Figure 11. Figure 11: Word length distribution visualizations of VKnowQA [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 16
Figure 16. Figure 16: Prompt for VKnowU evaluation for vanilla models. [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: Prompt for MLLMs to generate to See-Think-Answer output format. See-Think-Answer Prompt You are provided a text description of a problem and a question. Determine the answer to the question based on the text description. Provide only the single option letter (e.g., A,…
Figure 18
Figure 18. Figure 18: Prompt for verifier model to calculate visual knowledge [PITH_FULL_IMAGE:figures/full_fig_p018_18.png]
Figure 19
Figure 19. Figure 19: Prompt for DeepSeek-R1 [34] to enhance options. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_19.png]
Figure 20
Figure 20. Figure 20: Qualitative comparisons on Event Anticipation of VKnowU. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]
Figure 21
Figure 21. Figure 21: Qualitative comparisons on Spatial Awareness of VKnowU. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_21.png]
Figure 22
Figure 22. Figure 22: Qualitative comparisons on Object Affordance of VKnowU. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_22.png]
Figure 23
Figure 23. Figure 23: Qualitative comparisons on Intuitive Physics of VKnowU. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_23.png]
Figure 24
Figure 24. Figure 24: Qualitative comparisons on Subjective Intention of VKnowU. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_24.png]
Figure 25
Figure 25. Figure 25: Failure case of VideoKnow+: Inaccurate visual knowledge I. [PITH_FULL_IMAGE:figures/full_fig_p027_25.png]
Figure 26
Figure 26. Figure 26: Failure case of VideoKnow+: Inaccurate visual knowledge II. [PITH_FULL_IMAGE:figures/full_fig_p027_26.png]
Figure 27
Figure 27. Figure 27: Failure case of VideoKnow+: Insufficient visual knowledge. [PITH_FULL_IMAGE:figures/full_fig_p028_27.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

127 extracted references · 46 linked inside Pith

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 2

    arXiv 2023

  2. [2]

    On seeing stuff: The perception of materials by humans and machines

    Edward H Adelson. On seeing stuff: The perception of materials by humans and machines. InHuman vision and electronic imaging VI, pages 1–12. SPIE, 2001. 20

    2001

  3. [3]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InProceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015. 2

    2015

  4. [4]

    Eye-contact, distance and affiliation.Sociometry, pages 289–304, 1965

    Michael Argyle and Janet Dean. Eye-contact, distance and affiliation.Sociometry, pages 289–304, 1965. 20

    1965

  5. [5]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025. 2, 5, 7, 21

    arXiv 2025

  6. [6]

    Representing the existence and the lo- cation of hidden objects: Object permanence in 6-and 8- month-old infants.Cognition, 23(1):21–41, 1986

    Ren ´ee Baillargeon. Representing the existence and the lo- cation of hidden objects: Object permanence in 6-and 8- month-old infants.Cognition, 23(1):21–41, 1986. 20

    1986

  7. [7]

    theory of mind

    Simon Baron-Cohen, Alan M Leslie, and Uta Frith. Does the autistic child have a “theory of mind”?Cognition, 21 (1):37–46, 1985. 20

    1985

  8. [8]

    Arkitscenes: A di- verse real-world dataset for 3d indoor scene understanding using mobile rgb-d data.arXiv preprint arXiv:2111.08897,

    Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, et al. Arkitscenes: A di- verse real-world dataset for 3d indoor scene understanding using mobile rgb-d data.arXiv preprint arXiv:2111.08897,

    arXiv

Show all 127 references
  1. [9]

    Intphys 2: Benchmarking intuitive physics understand- ing in complex synthetic environments.arXiv preprint arXiv:2506.09849, 2025

    Florian Bordes, Quentin Garrido, Justine T Kao, Ad- ina Williams, Michael Rabbat, and Emmanuel Dupoux. Intphys 2: Benchmarking intuitive physics understand- ing in complex synthetic environments.arXiv preprint arXiv:2506.09849, 2025. 4, 14

    2025 arXiv

  2. [10]

    Routledge, 1995

    Franz Brentano.Psychology from an empirical standpoint. Routledge, 1995. Original work published 1874. 21

    1995

  3. [11]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. InProceed- ings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015. 14

    2015

  4. [12]

    Rextime: A benchmark suite for reasoning-across-time in videos.Advances in Neural In- formation Processing Systems, 37:28662–28673, 2024

    Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin, Yu-Chu Yu, Yen- Chun Chen, and Frank Wang. Rextime: A benchmark suite for reasoning-across-time in videos.Advances in Neural In- formation Processing Systems, 37:28662–28673, 2024. 4, 14

    2024

  5. [13]

    Are we on the right way for evaluating large vision-language models?Advances in Neural Infor- mation Processing Systems, 37:27056–27087, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?Advances in Neural Infor- mation Processing Systems, 37:27056–27087, 2024. 15

    2024

  6. [14]

    Video-holmes: Can mllm think like holmes for complex video reasoning?, 2025

    Junhao Cheng, Yuying Ge, Teng Wang, Yixiao Ge, Jing Liao, and Ying Shan. Video-holmes: Can mllm think like holmes for complex video reasoning?, 2025. 2

    2025

  7. [15]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 5

    2024 arXiv

  8. [16]

    Le, Sergey Levine, and Yi Ma

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V . Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025. 8

    2025

  9. [17]

    Whatever next? predictive brains, situated agents, and the future of cognitive science.Behavioral and brain sciences, 36(3):181–204, 2013

    Andy Clark. Whatever next? predictive brains, situated agents, and the future of cognitive science.Behavioral and brain sciences, 36(3):181–204, 2013. 20

    2013

  10. [18]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blis- tein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodal- ity, long context, and next generation agentic cap...

    2025 arXiv

  11. [19]

    Instructblip: Towards general- purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning. Advances in neural information processing systems, 36: 49250–49267, 2023. 2

    2023

  12. [20]

    MIT press, 1987

    Daniel C Dennett.The intentional stance. MIT press, 1987. 20

    1987

  13. [21]

    The llama 3 herd of models.arXiv e-prints, pages arXiv–2407,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv e-prints, pages arXiv–2407,

  14. [22]

    An argument for basic emotions.Cognition & emotion, 6(3-4):169–200, 1992

    Paul Ekman. An argument for basic emotions.Cognition & emotion, 6(3-4):169–200, 1992. 20

    1992

  15. [23]

    Unreal engine.https : / / www

    Epic Games. Unreal engine.https : / / www . unrealengine.com, 2019. Version 4.22.1. 14

    2019

  16. [24]

    Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025. 2, 8, 14, 17, 18, 21

    2025 arXiv

  17. [25]

    Material perception.Annual review of vision science, 3:365–388, 2017

    Roland W Fleming. Material perception.Annual review of vision science, 3:365–388, 2017. 20

    2017

  18. [26]

    The free-energy principle: a unified brain the- ory?Nature reviews neuroscience, 11(2):127–138, 2010

    Karl Friston. The free-energy principle: a unified brain the- ory?Nature reviews neuroscience, 11(2):127–138, 2010. 20 9

    2010

  19. [27]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InProceedings of the Computer Vision and Pat...

  20. [28]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. InEuropean Conference on Computer Vision, pages 148–166. Springer, 2024. 2

    2024

  21. [29]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2309.05656, 2023

    Yiyang Fu, Yusheng Dai, Zilin Zhang, Zihao Chen, Xin Zhang, Yike Wang, Zhen Zhang, Jun Zhu, and Yongxin Wu. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2309.05656, 2023. 2, 15

    2023 arXiv

  22. [30]

    Houghton Mifflin, 1979

    James J Gibson.The ecological approach to visual percep- tion. Houghton Mifflin, 1979. 20

    1979

  23. [31]

    The” something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The” something something” video database for learning and evaluating visual common sense. InP...

    2017

  24. [32]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024. 4

    2024 arXiv

  25. [33]

    Hallusionbench: an advanced diagnostic suite for entangled language halluci- nation and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language halluci- nation and visual illusion in large vision-language models. InProc...

    2024

  26. [34]

    Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. 2, 4, 8, 19

    2025 arXiv

  27. [35]

    Doubleday, 1966

    Edward T Hall.The hidden dimension. Doubleday, 1966. 20

    1966

  28. [36]

    Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning

    Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guob- ing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Li- hang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv e-prints, pages arXiv–2507, 2025. 5

    2025

  29. [37]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749,

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749,

    arXiv

  30. [38]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 5

    2024 arXiv

  31. [39]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, L ´elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Tho...

    2023

  32. [40]

    From clip to dino: Visual encoders shout in multi-modal large language models.arXiv preprint arXiv:2310.08825, 2023

    Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin’e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, and Hongkai Xiong. From clip to dino: Visual encoders shout in multi-modal large language models.arXiv preprint arXiv:2310.08825, 2023. 2

    2023 arXiv

  33. [41]

    Towards social ai: A survey on understand- ing social interactions.arXiv preprint arXiv:2409.15316,

    Sangmin Lee, Minzhi Li, Bolin Lai, Wenqi Jia, Fiona Ryan, Xu Cao, Ozgur Kara, Bikram Boote, Weiyan Shi, Diyi Yang, et al. Towards social ai: A survey on understand- ing social interactions.arXiv preprint arXiv:2409.15316,

    arXiv

  34. [42]

    What is more likely to happen next? video-and-language future event prediction.arXiv preprint arXiv:2010.07999,

    Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. What is more likely to happen next? video-and-language future event prediction.arXiv preprint arXiv:2010.07999,

    2010 arXiv

  35. [43]

    Detecting mo- ments and highlights in videos via natural language queries

    Jie Lei, Tamara L Berg, and Mohit Bansal. Detecting mo- ments and highlights in videos via natural language queries. Advances in Neural Information Processing Systems, 34: 11846–11858, 2021. 14

    2021

  36. [44]

    Seed-bench: Evaluating multimodal llms with generative and discriminative tasks.arXiv preprint arXiv:2401.15536, 2024

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Liu, Rui Wang, Ziyang Zhang, Yutong Zhou, Ji Zhang, Hang Gao, Jun Han, et al. Seed-bench: Evaluating multimodal llms with generative and discriminative tasks.arXiv preprint arXiv:2401.15536, 2024. 2

    2024 arXiv

  37. [45]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 5, 8

    2024 arXiv

  38. [46]

    From represen- tation to reasoning: Towards both evidence and common- sense reasoning for video question-answering

    Jiangtong Li, Li Niu, and Liqing Zhang. From represen- tation to reasoning: Towards both evidence and common- sense reasoning for video question-answering. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3, 17

    2022

  39. [47]

    Mvbench: A comprehensive multi-modal video under- standing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video under- standing benchmark. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22195...

    2024

  40. [48]

    Pope: A simple method to hallucination eval- uation in visual question answering

    Xinyu Li, Xiaofeng Zhang, Zhen Wang, Jing Liu, and Dahua Li. Pope: A simple method to hallucination eval- uation in visual question answering. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2023. 2

    2023

  41. [49]

    Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning.arXiv preprint arXiv:2504.06958, 2025

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning.arXiv preprint arXiv:2504.06958, 2025. 2 10

    2025 arXiv

  42. [50]

    Core knowledge deficits in multi-modal language models.arXiv preprint arXiv:2410.10855, 2024

    Yijiang Li, Qingying Gao, Tianwei Zhao, Bingyang Wang, Haoran Sun, Haiyun Lyu, Robert D Hawkins, Nuno Vas- concelos, Tal Golan, Dezhi Luo, et al. Core knowledge deficits in multi-modal language models.arXiv preprint arXiv:2410.10855, 2024. 3

    2024 arXiv

  43. [51]

    Self-rewarding vision- language model via reasoning decomposition.arXiv preprint arXiv:2508.19652, 2025

    Zongxia Li, Wenhao Yu, Chengsong Huang, Rui Liu, Zhenwen Liang, Fuxiao Liu, Jingxi Che, Dian Yu, Jor- dan Boyd-Graber, Haitao Mi, et al. Self-rewarding vision- language model via reasoning decomposition.arXiv preprint arXiv:2508.19652, 2025. 2

    2025 arXiv

  44. [52]

    Explainable multimodal emotion recognition

    Zheng Lian, Haiyang Sun, Licai Sun, Hao Gu, Zhuofan Wen, Siyuan Zhang, Shun Chen, Mingyu Xu, Ke Xu, Kang Chen, et al. Explainable multimodal emotion recognition. arXiv preprint arXiv:2306.15401, 2023. 17

    2023 arXiv

  45. [53]

    Improved visual-spatial reasoning via r1-zero-like training.arXiv preprint arXiv:2504.00883, 2025

    Zhenyi Liao, Qingsong Xie, Yanhao Zhang, Zijian Kong, Haonan Lu, Zhenyu Yang, and Zhijie Deng. Improved visual-spatial reasoning via r1-zero-like training.arXiv preprint arXiv:2504.00883, 2025. 17

    2025 arXiv

  46. [54]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InEuropean conference on computer vision, pages 740–755. Springer, 2014. 2

    2014

  47. [55]

    Sphinx-x: Scaling data and parameters for a family of multi-modal large language models.arXiv preprint arXiv:2402.05935, 2024

    Dongyang Liu, Renrui Zhang, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, Kaipeng Zhang, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models.arXiv preprint arXiv:2402.05935, 2024. 2

    2024 arXiv

  48. [56]

    Generative physical ai in vision: A survey.arXiv preprint arXiv:2501.10928, 2025

    Daochang Liu, Junyu Zhang, Anh-Dung Dinh, Eunbyung Park, Shichao Zhang, Ajmal Mian, Mubarak Shah, and Chang Xu. Generative physical ai in vision: A survey.arXiv preprint arXiv:2501.10928, 2025. 2

    2025 arXiv

  49. [57]

    Visual instruction tuning.Advances in neural infor- mation processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural infor- mation processing systems, 36:34892–34916, 2023. 2

    2023

  50. [58]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024. 2

    2024

  51. [59]

    Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild

    Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan. Mafw: A large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild. In Proceedings of the 30th ACM international conference on multi...

    2022

  52. [60]

    Mmbench: Is your multi- modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi- modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024. 15

    2024

  53. [61]

    Temp- compass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024

    Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Temp- compass: Do video llms really understand videos?arXiv preprint arXiv:2403.00476, 2024. 2

    2024 arXiv

  54. [62]

    Videoreasonbench: Can mllms perform vision-centric complex video reasoning?arXiv preprint arXiv:2505.23359, 2025

    Yuanxin Liu, Kun Ouyang, Haoning Wu, Yi Liu, Lin Sui, Xinhao Li, Yan Zhong, Y Charles, Xinyu Zhou, and Xu Sun. Videoreasonbench: Can mllms perform vision-centric complex video reasoning?arXiv preprint arXiv:2505.23359, 2025. 2

    2025

  55. [63]

    When thinking drifts: Evidential grounding for robust video reasoning.arXiv preprint arXiv:2510.06077, 2025

    Mi Luo, Zihui Xue, Alex Dimakis, and Kristen Grauman. When thinking drifts: Evidential grounding for robust video reasoning.arXiv preprint arXiv:2510.06077, 2025. 6

    2025

  56. [64]

    MIT press, 2010

    David Marr.Vision: A computational investigation into the human representation and processing of visual information. MIT press, 2010. 2

    2010

  57. [65]

    Ba- sic books, 1988

    Donald A Norman.The psychology of everyday things. Ba- sic books, 1988. 20

    1988

  58. [66]

    Clarendon Press, 1978

    John O’keefe and Lynn Nadel.The hippocampus as a cog- nitive map. Clarendon Press, 1978. 20

    1978

  59. [67]

    On visual knowledge.Frontiers of Information Technology & Electronic Engineering, 20(8):1021–1025,

    Yun-he Pan. On visual knowledge.Frontiers of Information Technology & Electronic Engineering, 20(8):1021–1025,

  60. [68]

    Basic Books, 1954

    Jean Piaget.The construction of reality in the child. Basic Books, 1954. 20

    1954

  61. [69]

    Free Press,

    Jean Piaget.The moral judgment of the child. Free Press,

  62. [70]

    Does the chimpanzee have a theory of mind?Behavioral and brain sciences, 1 (4):515–526, 1978

    David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind?Behavioral and brain sciences, 1 (4):515–526, 1978. 20

    1978

  63. [71]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InInternational conference on machine learning,...

    2021

  64. [72]

    Robust speech recognition via large-scale weak supervision, 2022

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. 4

    2022

  65. [73]

    Reka flash 3, 2025

    RekaAI. Reka flash 3, 2025. https://huggingface.co/RekaAI/reka-flash-3. 4

    2025

  66. [74]

    Intphys: A framework and benchmark for visual intuitive physics reasoning.arXiv preprint arXiv:1803.07616, 2018

    Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, V ´eronique Izard, and Em- manuel Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning.arXiv preprint arXiv:1803.07616, 2018. 17

    2018 arXiv

  67. [75]

    Wellness Insti- tute, Inc., 2001

    Andrew Salter.Conditioned reflex therapy. Wellness Insti- tute, Inc., 2001. 3

    2001

  68. [76]

    Lawrence Erlbaum Associates, 1977

    Roger C Schank and Robert P Abelson.Scripts, plans, goals, and understanding: An inquiry into human knowl- edge structures. Lawrence Erlbaum Associates, 1977. 20

    1977

  69. [77]

    Core dimensions of human material perception.Proceedings of the National Academy of Sci- ences, 2025

    Filipp Schmidt, Martin N Hebart, Alexandra C Schmid, and Roland W Fleming. Core dimensions of human material perception.Proceedings of the National Academy of Sci- ences, 2025. Note: Year based on available publication data which may be prospective. 20

    2025

  70. [78]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junx- iao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. 14

    2024 arXiv

  71. [79]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 2 11

    2019

  72. [80]

    A cognitive evaluation benchmark of image reasoning and description for large vision-language models.arXiv preprint arXiv:2402.18409, 2024

    Xiujie Song, Mengyue Wu, Kenny Q Zhu, Chunhao Zhang, and Yanyi Chen. A cognitive evaluation benchmark of image reasoning and description for large vision-language models.arXiv preprint arXiv:2402.18409, 2024. 2

    2024 arXiv

  73. [81]

    Origins of knowledge.Psychologi- cal review, 99(4):605, 1992

    Elizabeth S Spelke, Karen Breinlinger, John Macomber, and Kristen Jacobson. Origins of knowledge.Psychologi- cal review, 99(4):605, 1992. 20

    1992

  74. [82]

    Mimo-vl technical report, 2025

    Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, Kainan Bao, Hao Tian, Hailin Zhang, Gang Wang, Dawei Zhu, Cici, Chenhong He, Bowen Ye, Bowen Shen, Zihan Zhang, Zihan Jiang, Zhix- ian Zheng, Zhichao...

    2025

  75. [83]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text.arXiv preprint arXiv:2403.05530, 2024. 2

    2024 arXiv

  76. [84]

    Qwen2.5: A party of foundation models,

    Qwen Team. Qwen2.5: A party of foundation models,

  77. [85]

    Trl: Trans- former reinforcement learning.https : / / github

    Leandro von Werra, Younes Belkada, Lewis Tunstall, Ed- ward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallou´edec. Trl: Trans- former reinforcement learning.https : / / github . com/huggingface/trl, 2020. 18

    2020

  78. [86]

    Make your training flex- ible: Towards deployment-efficient video models.arXiv preprint arXiv:2503.14237, 2025

    Chenting Wang, Kunchang Li, Tianxiang Jiang, Xiangyu Zeng, Yi Wang, and Limin Wang. Make your training flex- ible: Towards deployment-efficient video models.arXiv preprint arXiv:2503.14237, 2025. 2

    2025 arXiv

  79. [87]

    Videoclip-xl: Advancing long descrip- tion understanding for video clip models.arXiv preprint arXiv:2410.00741, 2024

    Jiapeng Wang, Chengyu Wang, Kunzhe Huang, Jun Huang, and Lianwen Jin. Videoclip-xl: Advancing long descrip- tion understanding for video clip models.arXiv preprint arXiv:2410.00741, 2024. 2

    2024 arXiv

  80. [88]

    Videorft: Incentivizing video reasoning capabil- ity in mllms via reinforced fine-tuning.arXiv preprint arXiv:2505.12434, 2025

    Qi Wang, Yanrui Yu, Ye Yuan, Rui Mao, and Tianfei Zhou. Videorft: Incentivizing video reasoning capabil- ity in mllms via reinforced fine-tuning.arXiv preprint arXiv:2505.12434, 2025. 2, 8, 18

    2025

  81. [89]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025. 5

    2025 arXiv

  82. [90]

    Visual knowl- edge in the big model era: Retrospect and prospect.Fron- tiers of Information Technology & Electronic Engineering, 26(1):1–19, 2025

    Wenguan Wang, Yi Yang, and Yunhe Pan. Visual knowl- edge in the big model era: Retrospect and prospect.Fron- tiers of Information Technology & Electronic Engineering, 26(1):1–19, 2025. 1, 3

    2025

  83. [91]

    Internvideo2: Scaling foundation models for multimodal video understanding, 2024

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Chenting Wang, Guo Chen, Baoqi Pei, Ziang Yan, Rongkun Zheng, Jilan Xu, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Hongjie Zhang, Yifei Huang, Yu Qiao, Yali Wang, and Limin Wang. Internvideo2: Scaling foundation mo...

    2024

  84. [92]

    Social-iq 2.0 challenge: Benchmarking multimodal social understanding.https : / / github

    Alex Wilf, Leena Mathur, Sheryl Mathew, Claire Ko, Yous- souf Kebe, Paul Pu Liang, and Louis-Philippe Morency. Social-iq 2.0 challenge: Benchmarking multimodal social understanding.https : / / github . com / abwilf / Social-IQ-2.0-Challenge, 2023. 4, 14

    2023

  85. [93]

    Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children’s understanding of deception.Cognition, 13(1):103–128, 1983

    Heinz Wimmer and Josef Perner. Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children’s understanding of deception.Cognition, 13(1):103–128, 1983. 20

    1983

  86. [94]

    Star: A benchmark for situated reason- ing in real-world videos.arXiv preprint arXiv:2405.09711,

    Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reason- ing in real-world videos.arXiv preprint arXiv:2405.09711,

    arXiv

  87. [95]

    Visionary-r1: Mitigating shortcuts in vi- sual reasoning with reinforcement learning.arXiv preprint arXiv:2505.14677, 2025

    Jiaer Xia, Yuhang Zang, Peng Gao, Yixuan Li, and Kaiyang Zhou. Visionary-r1: Mitigating shortcuts in vi- sual reasoning with reinforcement learning.arXiv preprint arXiv:2505.14677, 2025. 2, 7

    2025

  88. [96]

    Next-qa: Next phase of question-answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 9777–9786, 2021. 2, 17

    2021

  89. [97]

    Advancing multi- modal reasoning capabilities of multimodal large language models via visual perception reward.arXiv preprint arXiv:2506.07218, 2025

    Tong Xiao, Xin Xu, Zhenya Huang, Hongyu Gao, Quan Liu, Qi Liu, and Enhong Chen. Advancing multi- modal reasoning capabilities of multimodal large language models via visual perception reward.arXiv preprint arXiv:2506.07218, 2025. 7

    2025

  90. [98]

    Expvid: A benchmark for experi- ment video understanding & reasoning.arXiv preprint arXiv:2510.11606, 2025

    Yicheng Xu, Yue Wu, Jiashuo Yu, Ziang Yan, Tianxiang Jiang, Yinan He, Qingsong Zhao, Kai Chen, Yu Qiao, Limin Wang, et al. Expvid: A benchmark for experi- ment video understanding & reasoning.arXiv preprint arXiv:2510.11606, 2025. 2

    2025

  91. [99]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. 4

    2025 arXiv

  92. [100]

    Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie

    Jihan Yang, Shusheng Yang, Anjali W. Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in space: How mul- timodal large language models see, remember, and recall spaces, 2025. 4, 14

    2025

  93. [101]

    R1-onevision: Advancing gener- alized multimodal reasoning through cross-modal formal- ization.arXiv preprint arXiv:2503.10615, 2025

    Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing gener- alized multimodal reasoning through cross-modal formal- ization.arXiv preprint arXiv:2503.10615, 2025. 2

    2025 arXiv

  94. [102]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhi- hui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024. 5

    2024 arXiv

  95. [103]

    12 mplug-owl3: Towards long image-sequence understand- ing in multi-modal large language models.arXiv preprint arXiv:2408.04840, 2024

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. 12 mplug-owl3: Towards long image-sequence understand- ing in multi-modal large language models.arXiv preprint arXiv:2408.04840, 2024. 5

    2024 arXiv

  96. [104]

    mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023. 2

    2023 arXiv

  97. [105]

    Clevrer: Collision events for video representation and reasoning

    Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019. 17

    1910 arXiv

  98. [106]

    Vrbench: A benchmark for multi-step reasoning in long narrative videos.arXiv preprint arXiv:2506.10857,

    Jiashuo Yu, Yue Wu, Meng Chu, Zhifei Ren, Zizheng Huang, Pei Chu, Ruijie Zhang, Yinan He, Qirui Li, Songze Li, et al. Vrbench: A benchmark for multi-step reasoning in long narrative videos.arXiv preprint arXiv:2506.10857,

    arXiv

  99. [107]

    Modeling context in referring expres- sions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expres- sions. InEuropean conference on computer vision, pages 69–85. Springer, 2016. 16

    2016

  100. [108]

    Pacs: A dataset for physical audiovisual commonsense reasoning

    Samuel Yu, Peter Wu, Paul Pu Liang, Ruslan Salakhut- dinov, and Louis-Philippe Morency. Pacs: A dataset for physical audiovisual commonsense reasoning. InEuropean Conference on Computer Vision, pages 292–309. Springer,

  101. [109]

    Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi.arXiv preprint arXiv:2311.12550, 2023

    Xing Yue, Bofang Qu, Ge Zhang, Yujia Yao, Qiang Zhang, Zhihao Zhao, Yupeng Su, Bowen Zhang, Ting Yu, Yushi Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi.arXiv preprint arXiv:2311.12550, 2023. 2

    2023 arXiv

  102. [110]

    Mmmu: A massive multi- discipline multimodal understanding and reasoning bench- mark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi- discipline multimodal understanding and reasoning bench- mark for expert agi. InProceedings of the IEEE/CVF Con- ference ...

    2024

  103. [111]

    Social-iq: A question answer- ing benchmark for artificial social intelligence

    Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-iq: A question answer- ing benchmark for artificial social intelligence. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8807–8817, 2019. 17

    2019

  104. [112]

    From recognition to cognition: Visual commonsense rea- soning

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense rea- soning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720–6731,

  105. [113]

    Timesuite: Improving mllms for long video understanding via grounded tuning.arXiv preprint arXiv:2410.19702, 2024

    Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhen- grong Yue, Yi Wang, et al. Timesuite: Improving mllms for long video understanding via grounded tuning.arXiv preprint arXiv:2410.19702, 2024. 2

    2024 arXiv

  106. [114]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 2

    2023

  107. [115]

    Long-clip: Unlocking the long-text capa- bility of clip

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capa- bility of clip. InEuropean conference on computer vision, pages 310–325. Springer, 2024. 2

    2024

  108. [116]

    Humaneval-v: Benchmarking high-level visual reasoning with complex diagrams in coding tasks.arXiv preprint arXiv:2410.12381, 2024

    Fengji Zhang, Linquan Wu, Huiyu Bai, Guancheng Lin, Xiao Li, Xiao Yu, Yue Wang, Bei Chen, and Jacky Keung. Humaneval-v: Benchmarking high-level visual reasoning with complex diagrams in coding tasks.arXiv preprint arXiv:2410.12381, 2024. 2

    2024 arXiv

  109. [117]

    Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024. 5, 17

    2024 arXiv

  110. [118]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176, 2025

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176, 2025. 4

    2025 arXiv

  111. [119]

    Mmvu: Measuring expert-level multi- discipline video understanding

    Yilun Zhao, Haowei Zhang, Lujing Xie, Tongyan Hu, Guo Gan, Yitao Long, Zhiyuan Hu, Weiyuan Chen, Chuhan Li, Zhijian Xu, et al. Mmvu: Measuring expert-level multi- discipline video understanding. InProceedings of the Com- puter Vision and Pattern Recognition Conference, pages 8...

    2025

  112. [120]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shen- glong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 2

    2025 arXiv

  113. [121]

    Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense.Engineer- ing, 6(3):310–345, 2020

    Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Ed- monds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, et al. Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense.Engineer- ing, 6(3):310–345, 2020. 1, 2 13 VKnowU: Evaluating Visual ...

    2020

  114. [123]

    Rate the options' quality (1 = good, 0 = needs improvement) based on: - Distinctiveness of each option - Plausibility of distractors - Relevance to question - Avoidance of redundancy/nonsense - Appropriate difficulty level

  115. [124]

    rating": 0,

    If rating is 0, provide improved options following above guidelines and take into account the following: - Preserve the correct answer's validity - Ensure all distractors are plausible but incorrect **Output Requirements:** You MUST provide your evaluation in the following str...

  116. [125]

    Output must be valid JSON (check for proper quotes, commas, etc.)

  117. [126]

    cognitive map

    Do not output any other explanation or comment. Rewriting Options Prompt Figure 19. Prompt for DeepSeek-R1 [34] to enhance options. 19 F. Scope of the Eight Visual Knowledge Tasks Intuitive Physics.Intuitive Physics refers to the cogni- tive ability to judge the physical plaus...

  118. [127]

    Rafting Valcartier

    Fireplace: The fireplace is located in the living room area, and it's quite far from the TV. It's not the closest object to the TV. 2. Stove: The stove is in the kitchen area, and it's also quite far from the TV. It's not the closest object to the TV either. 3. Table: The tabl...

  119. [1965]

    Original work published 1932. 21

    1932

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.