Pith. sign in

REVIEW 3 major objections 5 minor 93 references

ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that current multimodal large language models, despite strong general video understanding, remain far below human performance at qualitative spatial-temporal reasoning from continuous visual cues, with the best model scorin

desk verdict A carefully built dynamic-spatial-reasoning benchmark whose useful diagnostic numbers are undercut by an undocumented human baseline. read the letter →

arxiv 2607.20868 v1 pith:EKWVLUF5 submitted 2026-07-23 cs.CV

Han Li , Si Liu , Zehao Huang , Dongxin Lyu , Longfei Xu , Jiahui Fu , Daxin Tian , Yuliang Xiu
show 1 more author
Naiyan Wang
This is my paper · ORCID
classification cs.CV
keywords multimodallargelanguagemodelsspatial-temporalreasoningvideoquestionansweringbenchmarkdynamicscenesqualitativeevaluationoutcomepredictionphysicaldynamics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ViSTR-Bench asks whether multimodal large language models (MLLMs) can answer simple binary questions about dynamic scenes—whether a vehicle is moving faster, whether a basketball shot will go in, whether a Jenga tower will fall—using only the visual evidence available before the outcome is shown. The paper's central empirical claim is that current models largely cannot: the best system reaches 62.0% overall accuracy against a 50% random baseline and 57.9% frequency baseline, while human annotators score 91.0%. The paper argues this gap is not about recognizing objects or reading static geometry but about tracking temporal evidence, estimating motion states, extrapolating outcomes, and inferring latent physical dependencies. A sympathetic reader would care because these are the exact abilities needed for embodied AI, autonomous driving, and robotics, and because the benchmark is designed to be diagnostic rather than just competitive.

What carries the argument

The load-bearing instrument is the benchmark's construction pipeline: event localization to isolate single reasoning episodes, visual prompting (bounding boxes) to ground targets, outcome truncation at manually chosen decision points, and a four-criteria human quality control (visibility, temporal sufficiency, objectivity, non-triviality). This pipeline converts raw videos into binary qualitative questions that can only be answered by aggregating temporal evidence, and it defines the error taxonomy used to diagnose failures.

What would settle it

Re-annotate a random sample of the outcome-prediction items by showing only the truncated pre-outcome clips to a fresh set of annotators and measuring agreement with the published labels; if agreement is below roughly 95% or if annotators cannot confidently infer the answers, the benchmark's 29-point human-model gap would be an artifact of label construction rather than a measure of reasoning ability.

Watch

Extended reading notes

Core claim

ViSTR-Bench is a 1,340-item video question-answer benchmark organized into four reasoning dimensions—Motion Perception, Spatial Relations, Outcome Prediction, and Physical Dynamics—spanning 15 binary-choice subtasks. All items are qualitative (Yes/No or two-option choices), and for outcome tasks the videos are truncated at a manually chosen decision point so the answer cannot be read off the final frame. Evaluations across proprietary, open-source, and specialized spatial MLLMs show that the best model, GPT-5.4-thinking, reaches 62.0%, only 4.1 points above the frequency-based chance baseline and 29.0 points below the 91.0% human score. Error analysis of 600 incorrect predictions attributes

Load-bearing premise

The benchmark's central gap claim rests on the assumption that every truncated outcome-prediction and physical-dynamics clip has an unambiguous ground-truth answer that can be inferred from the retained pre-outcome footage; the paper does not report inter-annotator agreement, discarded-sample counts, or independent checks against label leakage (Sec. III-B, 'Outcome Truncation' and 'Human Quality Control').

Editorial extensions

If this is right

  • If the reported gap is accurate, current MLLMs cannot be relied upon for tasks that require anticipating physical outcomes from partial observations, such as driving or manipulation planning.
  • The failure distribution implies that improving target tracking and motion-state estimation is a more urgent bottleneck than improving object recognition.
  • Because specialized spatial MLLMs do not outperform general-purpose ones on ViSTR-Bench, static or geometry-centric spatial training does not transfer to dynamic reasoning.
  • Text-based chain-of-thought prompting is not sufficient to close the gap; gains are small and task-dependent.
  • Providing explicit task-relevant evidence (novel views, optical flow summaries) improves accuracy on specific tasks, suggesting input-centric and tool-augmented directions are promising.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 29-point gap is real, then a model that couples a low-level tracker (optical flow, object permanence) with a language model might approach human performance on this benchmark without any new reasoning architecture—implying the bottleneck is perceptual evidence, not inference.
  • The benchmark's binary format may underestimate models that have partial knowledge; converting it to graded confidence or open-ended justification could separate 'knows the gist' from 'commits to the right answer.'
  • The truncation protocol suggests a natural training scheme: sample videos, truncate before outcomes, and supervise the model to predict the truncated outcome—this could produce a scalable self-supervised objective for temporal reasoning.
  • Because human accuracy reaches 100% on several subtasks (e.g., Rotation Direction, Interaction Direction, Fall Direction), those items may be easier than the average; a per-subtask analysis of where humans are also imperfect could refine the benchmark's difficulty calibration.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces ViSTR-Bench, a video question-answering benchmark with 1,340 binary-choice items across 15 subtasks organized into four dimensions: Motion Perception, Spatial Relations, Outcome Prediction, and Physical Dynamics. The authors collect videos from public datasets, web sources, and self-recorded clips; apply event localization, visual prompting, and outcome truncation; and then evaluate a broad set of proprietary, open-source, and specialized spatial MLLMs. The headline result is that the best model, GPT-5.4-thinking, achieves 62.0% overall accuracy versus 57.9% for the frequency-based chance baseline and 91.0% for human performance, yielding a 29.0-point human-model gap. The paper also studies text-based chain-of-thought prompting, visual input formats, a six-way error taxonomy, and pilot input/tool augmentation strategies.

Significance. If the benchmark and the human baseline are valid, ViSTR-Bench fills a real gap by focusing on qualitative, reasoning-oriented spatial-temporal understanding rather than static spatial attributes or quantitative prediction. The paper has several concrete strengths: the subtask counts sum exactly to 1,340; the reported weighted human mean matches the stated 91.0%; the inclusion of both random and frequency baselines is helpful; the benchmark is externally constructed with no fitted parameters; and the error analysis and pilot studies provide useful diagnostic signal. The central claim, however, depends critically on two pieces of evidence that are currently under-reported: the human evaluation protocol and the reliability of the outcome-truncation labels. Without those, the 29-point gap could be an artifact of information leakage rather than a measure of spatial-temporal reasoning ability.

major comments (3)
  1. [Sec. IV-A and Sec. IV-B (Table II)] The human evaluation is described in a single sentence: 'we conduct human evaluation to estimate human-level performance.' There is no annotator count, no description of instructions, no statement about whether evaluators saw the same truncated clips as the models, no blinding procedure, and no inter-annotator agreement. The headline 29.0-point gap (62.0% vs. 91.0%) is load-bearing for the paper's main claim. If the human evaluators saw full videos, or were the same expert annotators who selected the truncation points with knowledge of the outcome, the 91.0% figure is inflated and the gap is not a fair measure. The authors should report a full protocol and inter-annotator agreement, and ideally run a blind evaluation on the identical truncated inputs.
  2. [Sec. III-B, 'Outcome Truncation' and 'Human Quality Control'] The manual decision point for truncation is chosen by annotators who know the full-video outcome, and the quality-control stage reports no count of discarded samples, no inter-annotator agreement, and no independent check that the retained pre-outcome clip makes the ground-truth answer unambiguous. For outcome-prediction and physical-dynamics tasks, subtle outcome-revealing cues (e.g., a player's reaction, a ball's curve, a Jenga tower's tilt) may leak into the retained prefix. This is not circularity, but it is a label-validity risk. The authors should report the discard rate, have a separate group of annotators label the truncated clips blind to the outcome, and quantify agreement.
  3. [Table II and Sec. IV-B] No confidence intervals or significance tests are reported. Several subtasks have very small sample sizes (Knot Type n=29, Golf Shot n=53, Fall Direction n=46, Passage Feasibility n=55), and many per-subtask model accuracies are within a few points of 50%, so the per-subtask ranking claims are not statistically distinguishable. Even the overall claim that 'only three evaluated models outperform the frequency-based baseline' needs interval estimates; with n=1,340 the top-model difference of 4.1 points is likely significant, but the paper should demonstrate this rather than assert it. At minimum, report Clopper-Pearson intervals or bootstrap CIs for overall and per-subtask accuracy, and test against the frequency baseline.
minor comments (5)
  1. [Fig. 1] The figure panel labels 'Human Gap: 36.0%' and 'X' are confusing, and the 36.0% value does not match the 29.0-point gap quoted in Sec. IV-B. Please clarify whether the figure refers to a per-example illustration and correct the inconsistency.
  2. [Sec. III-B] The term 'expert annotators' is used without defining who they are, how many participated, or what expertise they had. Please provide this information, especially given the emphasis on 'objectivity' and 'sufficiency.'
  3. [Sec. IV-D] The diagnostic study on 20 Basketball Shot samples is labeled 'small,' which is appropriate, but the numbers should be presented with uncertainty (e.g., exact binomial CIs) and framed as anecdotal rather than as evidence of a general pattern.
  4. [Sec. III-B, 'Release and Leaderboard Protocol'] The paper reports results on the complete 1,340-item set while planning to release only 50% publicly. Please clarify that no model selection or hyperparameter tuning was performed on the private held-out portion, so that the leaderboard protocol cannot be gamed by information leakage from the public split.
  5. [Abstract] Minor grammar: 'a comprehensive four-dimensional evaluations' should be 'a comprehensive four-dimensional evaluation framework'; 'VisualSpatial-TemporalReasoningBenchmark' also needs spacing in the introduction.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ViSTR-Bench is an external benchmark with no fitted parameters or derivation chain; the 62.0% vs. 91.0% gap is an empirical measurement, not a result reduced from its own inputs.

full rationale

This paper introduces a new benchmark and reports measured performance of many third-party MLLMs against a human baseline. There is no derivation chain, no fitted parameter subsequently called a prediction, and no uniqueness theorem imported from the authors' prior work. The central claim (best model 62.0% vs. human 91.0%) is an empirical measurement over 1,340 manually annotated QA pairs and a large set of external models; the claim is not used to construct the labels. The only author-overlap items, GeoThinker and SpatialLadder, are evaluated as test subjects and perform poorly (52.8% and 47.8%), which is the opposite of what a self-serving construction would produce; they provide no evidence for the benchmark's validity. The legitimate concerns raised by the paper — undocumented human evaluation protocol, no inter-annotator agreement, no count of discarded samples, truncation decision points chosen by annotators who knew the full-video outcome, and no explicit leakage check for 'Outcome Truncation' samples — are measurement-validity and correctness risks, not circularity as defined here. To flag circularity I would need to exhibit a specific reduction (e.g., the human figure being derived from the construction criterion or the same annotators being re-used as the human baseline), which the paper's text does not establish. The paper even reports a small diagnostic study in Sec. IV-D where human accuracy rises monotonically with more video exposure while the tested model does not, which is consistent with the benchmark measuring information usage rather than a self-confirming loop. Under the hard rule that non-findings are expected and appropriate for benchmark/evaluation papers, I assign score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

ViSTR-Bench is an empirical benchmark, so the ledger contains no fitted parameters and no invented physical entities. The load-bearing assumptions are about label validity, human annotation reliability, and the comparability of input formats across models.

assumptions (4)
  • domain assumption Ground-truth labels for truncated outcome-prediction and physical-dynamics tasks are unambiguous and inferable from the retained pre-outcome clip.
    Invoked in Sec. III-B 'Outcome Truncation' and 'Human Quality Control'; no inter-annotator agreement or label-leakage check is reported.
  • domain assumption Expert annotators' non-triviality filtering reliably removes samples solvable from a single static frame or language priors.
    Invoked in Sec. III-B 'Human Quality Control'; the criteria are subjective and no discarded-sample statistics are given.
  • ad hoc to paper Uniform 16-frame sampling for image-only models preserves enough temporal evidence for fair comparison with native video models.
    Introduced in Sec. IV-A 'Input Protocol'; the paper later shows ordered frames barely outperform shuffled frames, so this assumption is active and possibly optimistic.
  • domain assumption The reported human evaluation (91.0%) is a representative upper bound on the benchmark.
    Stated in Sec. IV-A 'Baselines and Human Evaluation'; annotator count, expertise, and instructions are not specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?." pith.science (2026). https://pith.science/paper/EKWVLUF5

@misc{pith2026260720868,
  author       = {Pith},
  title        = {Pith review of: ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EKWVLUF5}},
  note         = {Machine review of arXiv:2607.20868}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse expert-level tasks, but they still struggle with fundamental abilities that humans naturally develop through continuous observation of the real world, such as spatial perception and dynamic reasoning. Recent studies have recognized this gap and introduced dedicated benchmarks to evaluate the spatial-temporal capabilities of MLLMs. However, existing benchmarks mostly focus on static scenes or require exact quantitative predictions, leaving intuitive reasoning from temporal cues largely underexplored. In this paper, we introduce the Visual Spatial-Temporal Reasoning Benchmark (ViSTR-Bench), a novel evaluation suite designed to systematically assess whether MLLMs can perform qualitative reasoning from continuous visual cues in dynamic scenes. Guided by the principles of temporal emphasis, reasoning orientation, and qualitative evaluation, ViSTR-Bench establishes a comprehensive four-dimensional evaluations covering Motion Perception, Spatial Relations, Outcome Prediction, and Physical Dynamics. The benchmark comprises 15 distinct subtasks and 1,340 high-quality video question-answer pairs spanning diverse tabletop, indoor, and outdoor scenarios. Extensive evaluations of a broad spectrum of state-of-the-art proprietary, open-source, and specialized spatial MLLMs reveal that, despite their strong general video understanding capabilities, current models still face substantial bottlenecks in complex spatial-temporal reasoning and remain far below human performance.

Figures

Figures reproduced from arXiv: 2607.20868 by the authors.

Figure 1
Figure 1. Motivating examples and benchmark results. (a) Two representative examples from ViSTR-Bench, where underlined options indicate the correct answers. Although these questions are visually straightforward for humans, Claude [1] still produces incorrect answers, highlighting the difficulty of reasoning from continuous visual cues in dynamic scenes. (b) Quantitative evaluation results on ViSTR-Bench, showing that current… view at source ↗
Figure 2
Figure 2. Representative task examples from ViSTR-Bench. Underlined options indicate the correct answers. Each task is formulated as a qualitative binary￾choice question that requires models to aggregate temporal evidence and reason about dynamic visual scenes. encoder initialized with geometric priors to capture 3D struc￾tural information. Spatial-SSRL [50] proposes a novel self￾supervised reinforcement learning paradigm aim… view at source ↗
Figure 3
Figure 3. Benchmark statistics. ViSTR-Bench consists of four task dimensions and 15 subtasks. TABLE I TASK-LEVEL DATA SOURCE STATISTICS OF VISTR-BENCH. FOR EACH SUBTASK, WE REPORT THE TOTAL NUMBER OF QA PAIRS AND THE NUMBER OF SAMPLES COLLECTED FROM PUBLIC DATASETS, CURATED WEB VIDEOS, AND SELF-COLLECTED RECORDINGS. Task Total Public Web Collected Motion Perception Vehicle Movement 112 112 0 0 Relative Velocity 84 84 0 0 Rota… view at source ↗
Figures from the paper (34 more)
Figure 4
Figure 4. Figure 4: Benchmark construction pipeline. We construct ViSTR-Bench through a systematic multi-stage process, including data collection, video preprocessing, QA pair generation, and human quality control. This pipeline ensures that each sample contains sufficient temporal eviden…
Figure 5
Figure 5. Figure 5: Radar comparison across model groups. We visualize representative models from proprietary general-purpose MLLMs, open-source general-purpose MLLMs, and specialized spatial MLLMs across the 15 subtasks in ViSTR-Bench. Human performance is shown as a reference, highlight…
Figure 6
Figure 6. Figure 6: Representative examples for each primary error type. remain modest. Manual CoT achieves the best overall per￾formance, improving Direct Prompting by 1.6% and reaching 55.2%. By decomposing each task into task-specific reasoning steps, Manual CoT provides stronger guida…
Figure 7
Figure 7. Figure 7: Preliminary exploration on model improvement. We examine two complementary strategies for making task-relevant visual evidence more explicit. (a) For input-centric augmentation on Ego Motion, we reconstruct the scene using VGGT-Ω [90] and render auxiliary novel views t…
Figure 8
Figure 8. Figure 8: Manual CoT prompt template for Vehicle Movement. Relative Velocity You are an expert in driving video analysis and motion comparison. I will provide you with a driving video. Your task is to determine which vehicle is moving faster: the vehicle in the green box or the …
Figure 9
Figure 9. Figure 9: Manual CoT prompt template for Relative Velocity [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Manual CoT prompt template for Rotation Direction. Ego Motion You are an expert in video-based spatial reasoning. I will provide you with a video recorded by a moving camera in an indoor scene. The target object is the {TARGET}. Your task is to determine the final pos…
Figure 11
Figure 11. Figure 11: Manual CoT prompt template for Ego Motion [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Manual CoT prompt template for Passage Feasibility. Interaction Direction You are an expert in person-object interaction analysis. I will provide you with a video of an interaction between a person and {TARGET}. Your task is to determine the person’s movement directio…
Figure 13
Figure 13. Figure 13: Manual CoT prompt template for Interaction Direction [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Manual CoT prompt template for Basketball Shot. Soccer Shot You are an expert in sports video analysis and physics trajectory prediction. I will provide you with a video of a soccer free kick. Your task is to predict whether the ball will go into the goal based on its…
Figure 15
Figure 15. Figure 15: Manual CoT prompt template for Soccer Shot [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Manual CoT prompt template for Golf Shot. Billiards Shot You are an expert in sports video analysis and trajectory-based motion prediction. I will provide you with a video of billiards. Your task is to predict whether the target ball will go into the pocket based on i…
Figure 17
Figure 17. Figure 17: Manual CoT prompt template for Billiards Shot [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: Manual CoT prompt template for Swimming Race. Fall Direction You are an expert in video-based human motion analysis. I will provide you with a video of a person falling. Your task is to determine the person’s motion direction or motion category. The two candidate answ…
Figure 19
Figure 19. Figure 19: Manual CoT prompt template for Fall Direction [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: Manual CoT prompt template for Jenga Stability. Mikado Dependency You are an expert in physical interaction analysis and video-based contact reasoning. I will provide you with a video of a Mikado game. Your task is to predict whether the stick indicated by the pointin…
Figure 21
Figure 21. Figure 21: Manual CoT prompt template for Mikado Dependency [PITH_FULL_IMAGE:figures/full_fig_p023_21.png]
Figure 22
Figure 22. Figure 22: Manual CoT prompt template for Knot Type. id deface: 83 gpt 5.4 thinking gemini 3.1 pro preview claude ops 4.6 thinking : This is a driving video. Please determine whether the vehicle in the green box shows subtle movement during the video. Answer Yes or No. : The veh…
Figure 23
Figure 23. Figure 23: Qualitative visualization for Vehicle Movement. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude￾Opus-4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p024_23.png]
Figure 24
Figure 24. Figure 24: Qualitative visualization for Relative Velocity. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus￾4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p025_24.png]
Figure 25
Figure 25. Figure 25: Qualitative visualization for Rotation Direction. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude￾Opus-4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p026_25.png]
Figure 26
Figure 26. Figure 26: Qualitative visualization for Ego Motion. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus-4.6- thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p027_26.png]
Figure 27
Figure 27. Figure 27: Qualitative visualization for Passage Feasibility. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude￾Opus-4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p028_27.png]
Figure 28
Figure 28. Figure 28: Qualitative visualization for Interaction Direction. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude￾Opus-4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p029_28.png]
Figure 29
Figure 29. Figure 29: Qualitative visualization for Basketball Shot. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus￾4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p030_29.png]
Figure 30
Figure 30. Figure 30: Qualitative visualization for Soccer Shot. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus-4.6- thinking [1]. Gray text denotes built-in thinking content. id deface: 395 gpt 5.4 thinking gemini 3.1 pro preview claude ops 4…
Figure 31
Figure 31. Figure 31: Qualitative visualization for Golf Shot. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus-4.6- thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p031_31.png]
Figure 32
Figure 32. Figure 32: Qualitative visualization for Billiards Shot. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus￾4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p032_32.png]
Figure 33
Figure 33. Figure 33: Qualitative visualization for Swimming Race. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus￾4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p033_33.png]
Figure 34
Figure 34. Figure 34: Qualitative visualization for Fall Direction. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus￾4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p034_34.png]
Figure 35
Figure 35. Figure 35: Qualitative visualization for Jenga Stability. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus￾4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p035_35.png]
Figure 36
Figure 36. Figure 36: Qualitative visualization for Mikado Dependency. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude￾Opus-4.6-thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p036_36.png]
Figure 37
Figure 37. Figure 37: Qualitative visualization for Knot Type. We compare the outputs of GPT-5.4-thinking [40], Gemini-3.1-Pro-Preview [41], and Claude-Opus-4.6- thinking [1]. Gray text denotes built-in thinking content [PITH_FULL_IMAGE:figures/full_fig_p037_37.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

93 extracted references · 21 linked inside Pith

  1. [1]

    Anthropic Model System Cards,

    Anthropic, “Anthropic Model System Cards,” https://www.anthropic.co m/system-cards, 2026

    2026

  2. [2]

    MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts,

    P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao, “MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 23 439– 23 554

    2024

  3. [3]

    Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset,

    K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li, “Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset,”Advances in Neural Information Processing Systems, vol. 37, pp. 95 095–95 169, 2024

    2024

  4. [4]

    MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?

    R. Zhang, D. Jiang, Y . Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K.-W. Chang, Y . Qiaoet al., “MATHVERSE: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?” inEuropean Conference on Computer Vision, 2024, pp. 169–186

    2024

  5. [5]

    MMCode: Bench- marking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems,

    K. Li, Y . Tian, Q. Hu, Z. Luo, Z. Huang, and J. Ma, “MMCode: Bench- marking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems,” inFindings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 736–783

    2024

  6. [6]

    Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering,

    C. Si, Y . Zhang, R. Li, Z. Yang, R. Liu, and D. Yang, “Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2025, pp. 3956–3974

    2025

  7. [7]

    Emu3: Next-Token Prediction is All You Need,

    X. Wang, X. Zhang, Z. Luo, Q. Sun, Y . Cui, J. Wang, F. Zhang, Y . Wang, Z. Li, Q. Yuet al., “Emu3: Next-Token Prediction is All You Need,” arXiv preprint arXiv:2409.18869, 2024

    arXiv 2024

  8. [8]

    NExT-GPT: Any-to-Any Multimodal LLM,

    S. Wu, H. Fei, L. Qu, W. Ji, and T.-S. Chua, “NExT-GPT: Any-to-Any Multimodal LLM,” inForty-first International Conference on Machine Learning, 2024

    2024

Show all 93 references
  1. [9]

    MiniGPT-5: Interleaved Vision- and-Language Generation via Generative V okens,

    K. Zheng, X. He, and X. E. Wang, “MiniGPT-5: Interleaved Vision- and-Language Generation via Generative V okens,”arXiv preprint arXiv:2310.02239, 2023

    2023

  2. [10]

    Introducing GPT-5,

    OpenAI, “Introducing GPT-5,” https://openai.com/index/introducing-g pt-5, 2025

    2025

  3. [11]

    GPQA: A Graduate-Level Google- Proof Q&A Benchmark,

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y . Pang, J. Dirani, J. Michael, and S. R. Bowman, “GPQA: A Graduate-Level Google- Proof Q&A Benchmark,”arXiv preprint arXiv:2311.12022, 2023

    2023 arXiv

  4. [12]

    Gemini Achieves Gold-Medal Level at the Interna- tional Collegiate Programming Contest World Finals,

    Google DeepMind, “Gemini Achieves Gold-Medal Level at the Interna- tional Collegiate Programming Contest World Finals,” https://deepmind .google/blog/gemini-achieves-gold-medal-level-at-the-international-col legiate-programming-contest-world-finals, 2025

    2025

  5. [13]

    LMDrive: Closed-Loop End-to-End Driving with Large Language Models,

    H. Shao, Y . Hu, L. Wang, G. Song, S. L. Waslander, Y . Liu, and H. Li, “LMDrive: Closed-Loop End-to-End Driving with Large Language Models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 15 120–15 130

    2024

  6. [14]

    DriveLM: Driving with Graph Visual Question Answering,

    C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, and H. Li, “DriveLM: Driving with Graph Visual Question Answering,” inEuropean Conference on Computer Vision, 2024, pp. 256–274

    2024

  7. [15]

    DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model,

    Z. Xu, Y . Zhang, E. Xie, Z. Zhao, Y . Guo, K.-Y . K. Wong, Z. Li, and H. Zhao, “DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model,”IEEE Robotics and Automation Letters, vol. 9, no. 10, pp. 8186–8193, 2024

    2024

  8. [16]

    Open- VLA: An Open-Source Vision-Language-Action Model,

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketiet al., “Open- VLA: An Open-Source Vision-Language-Action Model,”arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  9. [17]

    RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control,

    B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahidet al., “RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control,” inConference on Robot Learning, 2023, pp. 2165–2183

    2023

  10. [18]

    Do As I Can, Not As I Say: Grounding Language in Robotic Affordances,

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausmanet al., “Do As I Can, Not As I Say: Grounding Language in Robotic Affordances,”arXiv preprint arXiv:2204.01691, 2022

    2022 arXiv

  11. [19]

    PaLM-E: An Embodied Multimodal Language Model,

    D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yuet al., “PaLM-E: An Embodied Multimodal Language Model,”arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  12. [20]

    OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models,

    M. Jia, Z. Qi, S. Zhang, W. Zhang, X. Yu, J. He, H. Wang, and L. Yi, “OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models,”arXiv preprint arXiv:2506.03135, 2025

    2025

  13. [21]

    MMSI-Video-Bench: A Holistic Benchmark for Video- Based Spatial Intelligence,

    J. Lin, R. Xu, S. Zhu, S. Yang, P. Cao, Y . Ran, M. Hu, C. Zhu, Y . Xie, Y . Longet al., “MMSI-Video-Bench: A Holistic Benchmark for Video- Based Spatial Intelligence,”arXiv preprint arXiv:2512.10863, 2025

    2025

  14. [22]

    SpaceR: Reinforcing MLLMs in Video Spatial Reasoning,

    K. Ouyang, Y . Liu, H. Wu, Y . Liu, H. Zhou, J. Zhou, F. Meng, and X. Sun, “SpaceR: Reinforcing MLLMs in Video Spatial Reasoning,” arXiv preprint arXiv:2504.01805, 2025

    2025 arXiv

  15. [23]

    Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces,

    J. Yang, S. Yang, A. W. Gupta, R. Han, L. Fei-Fei, and S. Xie, “Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 10 632–10 643

    2025

  16. [24]

    Cambrian-S: Towards Spatial Supersensing in Video,

    S. Yang, J. Yang, P. Huang, E. L. Brown II, Z. Yang, Y . Yu, S. Tong, Z. Zheng, Y . Xu, M. Wanget al., “Cambrian-S: Towards Spatial Supersensing in Video,” inThe Fourteenth International Conference on Learning Representations, 2025. 13

    2025

  17. [25]

    From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D,

    J. Zhang, Y . Chen, Y . Zhou, Y . Xu, Z. Huang, J. Mei, J. Chen, Y .- J. Yuan, X. Cai, G. Huanget al., “From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D,”arXiv preprint arXiv:2503.22976, 2025

    2025

  18. [26]

    Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World,

    Y . Huang, K. Wen, R. Gao, D. Liu, Y . Lou, J. Wu, J. Xu, J. Zhang, Z. Yang, Y . Linet al., “Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World,”arXiv preprint arXiv:2603.12746, 2026

    2026

  19. [27]

    STI- Bench: Are MLLMs Ready for Precise Spatial-Temporal World Under- standing?

    Y . Li, Y . Zhang, T. Lin, X. Liu, W. Cai, Z. Liu, and B. Zhao, “STI- Bench: Are MLLMs Ready for Precise Spatial-Temporal World Under- standing?” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 5622–5632

    2025

  20. [28]

    OST- Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding,

    J. Lin, C. Zhu, R. Xu, X. Mao, X. Liu, T. Wang, and J. Pang, “OST- Bench: Evaluating the Capabilities of MLLMs in Online Spatio-temporal Scene Understanding,”arXiv preprint arXiv:2507.07984, 2025

    2025

  21. [29]

    VideoLoom: A Video Large Language Model for Joint Spatial-Temporal Understanding,

    J. Shi, J. Wang, Z. You, B. He, and Z. Wu, “VideoLoom: A Video Large Language Model for Joint Spatial-Temporal Understanding,” arXiv preprint arXiv:2601.07290, 2026

    2026

  22. [30]

    MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence,

    X. Yin, C. Li, J. Chang, C.-M. Pun, and X. Cun, “MLLM-4D: Towards Visual-based Spatial-Temporal Intelligence,”arXiv preprint arXiv:2603.00515, 2026

    2026

  23. [31]

    DSI-Bench: A Benchmark for Dynamic Spatial Intelligence,

    Z. Zhang, Z. Wang, G. Zhang, W. Dai, Y . Xia, Z. Yan, M. Hong, and Z. Zhao, “DSI-Bench: A Benchmark for Dynamic Spatial Intelligence,” arXiv preprint arXiv:2510.18873, 2025

    2025

  24. [32]

    Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models,

    S. Zhou, Y . Chen, Y . Ge, W. Huang, J. Lin, Y . Shan, and X. Qi, “Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models,”arXiv preprint arXiv:2512.20557, 2025

    2025

  25. [33]

    VLM4D: Towards Spatiotemporal Awareness in Vision Language Models,

    S. Zhou, A. Vilesov, X. He, Z. Wan, S. Zhang, A. Nagachandra, D. Chang, D. Chen, X. E. Wang, and A. Kadambi, “VLM4D: Towards Spatiotemporal Awareness in Vision Language Models,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 8600–8612

    2025

  26. [34]

    Ego4D: Around the World in 3,000 Hours of Egocentric Video,

    K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liuet al., “Ego4D: Around the World in 3,000 Hours of Egocentric Video,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18 9...

    2022

  27. [35]

    ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes,

    A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017, pp. 5828–5839

    2017

  28. [36]

    ScanNet++: A High- Fidelity Dataset of 3D Indoor Scenes,

    C. Yeshwanth, Y .-C. Liu, M. Nießner, and A. Dai, “ScanNet++: A High- Fidelity Dataset of 3D Indoor Scenes,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 12–22

    2023

  29. [37]

    ARKitScenes: A Diverse Real- World Dataset For 3D Indoor Scene Understanding Using Mobile RGB- D Data,

    G. Baruch, Z. Chen, A. Dehghan, T. Dimry, Y . Feigin, P. Fu, T. Gebauer, B. Joffe, D. Kurz, A. Schwartzet al., “ARKitScenes: A Diverse Real- World Dataset For 3D Indoor Scene Understanding Using Mobile RGB- D Data,”arXiv preprint arXiv:2111.08897, 2021

    2021 arXiv

  30. [38]

    Scalability in Perception for Autonomous Driving: Waymo Open Dataset,

    P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V . Patnaik, P. Tsui, J. Guo, Y . Zhou, Y . Chai, B. Caineet al., “Scalability in Perception for Autonomous Driving: Waymo Open Dataset,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020...

    2020

  31. [39]

    Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset,

    J. Lin, A. Zeng, S. Lu, Y . Cai, R. Zhang, H. Wang, and L. Zhang, “Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset,”Advances in Neural Information Processing Systems, vol. 36, pp. 25 268–25 280, 2023

    2023

  32. [40]

    OpenAI GPT-5 System Card,

    A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthramet al., “OpenAI GPT-5 System Card,”arXiv preprint arXiv:2601.03267, 2025

    2025 arXiv

  33. [41]

    Google DeepMind Model Cards,

    Google DeepMind, “Google DeepMind Model Cards,” https://deepmi nd.google/models/model-cards, 2026

    2026

  34. [42]

    Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexity,

    ByteDance Seed Team, “Seed2.0 Model Card: Towards Intelligence Frontier for Real-World Complexity,” https://seed.bytedance.com/en/ seed2, 2026

    2026

  35. [43]

    Xiaomi MiMo-V2.5: A Leap in Agency and Multimodality,

    Xiaomi MiMo Team, “Xiaomi MiMo-V2.5: A Leap in Agency and Multimodality,” https://mimo.xiaomi.com/mimo-v2-5, 2026

    2026

  36. [44]

    LLaV A-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training,

    X. An, Y . Xie, K. Yang, W. Zhang, X. Zhao, Z. Cheng, Y . Wang, S. Xu, C. Chen, D. Zhuet al., “LLaV A-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training,”arXiv preprint arXiv:2509.23661, 2025

    2025 arXiv

  37. [45]

    Qwen3.5: Towards Native Multimodal Agents,

    Qwen Team, “Qwen3.5: Towards Native Multimodal Agents,” https: //qwen.ai/blog?id=qwen3.5, 2026

    2026

  38. [46]

    InternVL3.5: Advancing Open-Source Multi- modal Models in Versatility, Reasoning, and Efficiency,

    W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shaoet al., “InternVL3.5: Advancing Open-Source Multi- modal Models in Versatility, Reasoning, and Efficiency,”arXiv preprint arXiv:2508.18265, 2025

    2025 arXiv

  39. [47]

    GLM-4.6V: Open Source Multimodal Models with Native Multi- modal Tool Use,

    Z.ai, “GLM-4.6V: Open Source Multimodal Models with Native Multi- modal Tool Use,” https://docs.z.ai/guides/vlm/glm-4.6v, 2025

    2025

  40. [48]

    Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors,

    D. Zheng, Y . Li, L. Wanget al., “Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors,”Advances in Neural Information Processing Systems, vol. 38, pp. 20 560–20 586, 2026

    2026

  41. [49]

    Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence,

    D. Wu, F. Liu, Y .-H. Hung, and Y . Duan, “Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence,”arXiv preprint arXiv:2505.23747, 2025

    2025 arXiv

  42. [50]

    Spatial-SSRL: Enhancing Spatial Understanding via Self- Supervised Reinforcement Learning,

    Y . Liu, B. Zhang, Y . Zang, Y . Cao, L. Xing, X. Dong, H. Duan, D. Lin, and J. Wang, “Spatial-SSRL: Enhancing Spatial Understanding via Self- Supervised Reinforcement Learning,”arXiv preprint arXiv:2510.27606, 2025

    2025

  43. [51]

    Thinking with Geometry: Active Geometry Integration for Spatial Reasoning,

    H. Li, Q. Cao, T. Tang, K. Xiang, Z. Guo, J. Han, H. Xu, and X. Liang, “Thinking with Geometry: Active Geometry Integration for Spatial Reasoning,”arXiv preprint arXiv:2602.06037, 2026

    2026 arXiv

  44. [52]

    SEED-Bench: Benchmarking Multimodal Large Language Models,

    B. Li, Y . Ge, Y . Ge, G. Wang, R. Wang, R. Zhang, and Y . Shan, “SEED-Bench: Benchmarking Multimodal Large Language Models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 299–13 308

    2024

  45. [53]

    MMBench: Is Your Multi-modal Model an All- around Player?

    Y . Liu, H. Duan, Y . Zhang, B. Li, S. Zhang, W. Zhao, Y . Yuan, J. Wang, C. He, Z. Liuet al., “MMBench: Is Your Multi-modal Model an All- around Player?” inEuropean Conference on Computer Vision, 2024, pp. 216–233

    2024

  46. [54]

    MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities,

    W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang, “MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities,” inProceedings of the 41st International Conference on Machine Learning, 2024, pp. 57 730–57 754

    2024

  47. [55]

    MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sunet al., “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2024

  48. [56]

    MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding,

    X. Fang, K. Mao, H. Duan, X. Zhao, Y . Li, D. Lin, and K. Chen, “MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding,”Advances in Neural Information Processing Systems, vol. 37, pp. 89 098–89 124, 2024

    2024

  49. [57]

    Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis,

    C. Fu, Y . Dai, Y . Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y . Shen, M. Zhanget al., “Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2025

  50. [58]

    MVBench: A Comprehensive Multi-modal Video Understanding Benchmark,

    K. Li, Y . Wang, Y . He, Y . Li, Y . Wang, Y . Liu, Z. Wang, J. Xu, G. Chen, P. Luoet al., “MVBench: A Comprehensive Multi-modal Video Understanding Benchmark,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 22 195–22 206

    2024

  51. [59]

    VI- TATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Models,

    S. Li, L. Li, Y . Liu, S. Ren, Y . Liu, R. Gao, X. Sun, and L. Hou, “VI- TATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Models,” inEuropean Conference on Computer Vision, 2024, pp. 331–348

    2024

  52. [60]

    Multi-modal Situated Reasoning in 3D Scenes,

    X. Linghu, J. Huang, X. Niu, X. Ma, B. Jia, and S. Huang, “Multi-modal Situated Reasoning in 3D Scenes,”Advances in Neural Information Processing Systems, vol. 37, pp. 140 903–140 936, 2024

    2024

  53. [61]

    TempCompass: Do Video LLMs Really Understand Videos?

    Y . Liu, S. Li, Y . Liu, Y . Wang, S. Ren, L. Li, S. Chen, X. Sun, and L. Hou, “TempCompass: Do Video LLMs Really Understand Videos?” inFindings of the Association for Computational Linguistics: ACL 2024, 2024, pp. 8731–8772

    2024

  54. [62]

    OpenEQA: Embodied Question Answering in the Era of Foundation Models,

    A. Majumdar, A. Ajay, X. Zhang, P. Putta, S. Yenamandra, M. Henaff, S. Silwal, P. Mcvay, O. Maksymets, S. Arnaudet al., “OpenEQA: Embodied Question Answering in the Era of Foundation Models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,...

    2024

  55. [63]

    EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding,

    K. Mangalam, R. Akshulakov, and J. Malik, “EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding,”Ad- vances in Neural Information Processing Systems, vol. 36, pp. 46 212– 46 244, 2023

    2023

  56. [64]

    Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models,

    M. Ning, B. Zhu, Y . Xie, B. Lin, J. Cui, L. Yuan, D. Chen, and L. Yuan, “Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models,”Computational Visual Media, 2025

    2025

  57. [65]

    TOMATO: Assessing Visual Temporal Reasoning Capabili- ties in Multimodal Foundation Models,

    Z. Shangguan, C. Li, Y . Ding, Y . Zheng, Y . Zhao, T. Fitzgerald, and A. Cohan, “TOMATO: Assessing Visual Temporal Reasoning Capabili- ties in Multimodal Foundation Models,” inThe Thirteenth International Conference on Learning Representations, 2024

    2024

  58. [66]

    Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for 14 Vision Language Models,

    J. Wang, Y . Ming, Z. Shi, V . Vineet, X. Wang, Y . Li, and N. Joshi, “Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for 14 Vision Language Models,”Advances in Neural Information Processing Systems, vol. 37, pp. 75 392–75 421, 2024

    2024

  59. [67]

    MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA,

    H. Ye, H. Zhang, E. Daxberger, L. Chen, Z. Lin, Y . Li, B. Zhang, H. You, D. Xu, Z. Ganet al., “MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA,” inThe Thirteenth International Conference on Learning Representations, 2024

    2024

  60. [68]

    LongVILA: Scaling Long-Context Visual Language Models for Long Videos,

    Y . Chen, F. Xue, D. Li, Q. Hu, L. Zhu, X. Li, Y . Fang, H. Tang, S. Yang, Z. Liuet al., “LongVILA: Scaling Long-Context Visual Language Models for Long Videos,”arXiv preprint arXiv:2408.10188, 2024

    2024 arXiv

  61. [69]

    LLaV A-OneVision: Easy Visual Task Transfer,

    B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y . Li, Z. Liuet al., “LLaV A-OneVision: Easy Visual Task Transfer,” arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  62. [70]

    VILA: On Pre-training for Visual Language Models,

    J. Lin, H. Yin, W. Ping, P. Molchanov, M. Shoeybi, and S. Han, “VILA: On Pre-training for Visual Language Models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 689–26 699

    2024

  63. [71]

    Long Context Transfer from Language to Vision,

    P. Zhang, K. Zhang, B. Li, G. Zeng, J. Yang, Y . Zhang, Z. Wang, H. Tan, C. Li, and Z. Liu, “Long Context Transfer from Language to Vision,” arXiv preprint arXiv:2406.16852, 2024

    2024 arXiv

  64. [72]

    LLaV A- Video: Video Instruction Tuning With Synthetic Data,

    Y . Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li, “LLaV A- Video: Video Instruction Tuning With Synthetic Data,”arXiv preprint arXiv:2410.02713, 2024

    2024 arXiv

  65. [73]

    SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities,

    B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Sadigh, L. Guibas, and F. Xia, “SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 14 455–14 465

    2024

  66. [74]

    VLM-3R: Vision-Language Models Aug- mented with Instruction-Aligned 3D Reconstruction,

    Z. Fan, J. Zhang, R. Li, J. Zhang, R. Chen, H. Hu, K. Wang, H. Qu, S. Zhou, D. Wanget al., “VLM-3R: Vision-Language Models Aug- mented with Instruction-Aligned 3D Reconstruction,”arXiv preprint arXiv:2505.20279, 2025

    2025 arXiv

  67. [75]

    3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding,

    X. Huang, J. Wu, Q. Xie, and K. Han, “3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding,”arXiv preprint arXiv:2506.01946, 2025

    2025

  68. [76]

    SpatialLadder: Progressive Training for Spatial Rea- soning in Vision-Language Models,

    H. Li, D. Li, Z. Wang, Y . Yan, H. Wu, W. Zhang, Y . Shen, W. Lu, J. Xiao, and Y . Zhuang, “SpatialLadder: Progressive Training for Spatial Rea- soning in Vision-Language Models,”arXiv preprint arXiv:2510.08531, 2025

    2025

  69. [77]

    SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planning,

    Y . Liu, D. Chi, S. Wu, Z. Zhang, Y . Hu, L. Zhang, Y . Zhang, S. Wu, T. Cao, G. Huanget al., “SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planning,”arXiv preprint arXiv:2501.10074, 2025

    2025 arXiv

  70. [78]

    SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation,

    Z. Qi, W. Zhang, Y . Ding, R. Dong, X. Yu, J. Li, L. Xu, B. Li, X. He, G. Fanet al., “SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation,”arXiv preprint arXiv:2502.13143, 2025

    2025

  71. [79]

    Visual Spatial Tuning,

    R. Yang, Z. Zhu, Y . Li, J. Huang, S. Yan, S. Zhou, Z. Liu, X. Li, S. Li, W. Wanget al., “Visual Spatial Tuning,”arXiv preprint arXiv:2511.05491, 2025

    2025

  72. [80]

    Make Geometry Matter for Spatial Reasoning,

    S. Zhang, Q. Shen, S. Wang, T. Pan, and X. Wang, “Make Geometry Matter for Spatial Reasoning,”arXiv preprint arXiv:2603.26639, 2026

    2026

  73. [81]

    Think3D: Thinking with Space for Spatial Reasoning,

    Z. Zhang, Y . Wu, L. Jia, Y . Wang, Z. Zhang, Y . Li, B. Ran, F. Zhang, Z. Sun, Z. Yinet al., “Think3D: Thinking with Space for Spatial Reasoning,”arXiv preprint arXiv:2601.13029, 2026

    2026

  74. [82]

    LLaV A-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness,

    C. Zhu, T. Wang, W. Zhang, J. Pang, and X. Liu, “LLaV A-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness,”arXiv preprint arXiv:2409.18125, 2024

    2024 arXiv

  75. [83]

    LLaV A-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding,

    H. Zhou and G. H. Lee, “LLaV A-4D: Embedding SpatioTemporal Prompt into LMMs for 4D Scene Understanding,” inThe Fourteenth International Conference on Learning Representations, 2025

    2025

  76. [84]

    Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generation,

    ——, “Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generation,”arXiv preprint arXiv:2509.23828, 2025

    2025

  77. [85]

    Intern-S1: A Scientific Multimodal Foundation Model,

    L. Bai, Z. Cai, M. Cao, W. Cao, C. Chenet al., “Intern-S1: A Scientific Multimodal Foundation Model,”arXiv preprint arXiv:2508.15763, 2025

    2025

  78. [86]

    Intern-S1-Pro: Sci- entific Multimodal Foundation Model at Trillion Scale,

    Y . Zou, D. Zhu, L. Zhu, T. Zhu, Y . Zhouet al., “Intern-S1-Pro: Sci- entific Multimodal Foundation Model at Trillion Scale,”arXiv preprint arXiv:2603.25040, 2026

    2026

  79. [87]

    Large Language Models are Zero-Shot Reasoners,

    T. Kojima, S. S. Gu, M. Reid, Y . Matsuo, and Y . Iwasawa, “Large Language Models are Zero-Shot Reasoners,”Advances in Neural Infor- mation Processing Systems, vol. 35, pp. 22 199–22 213, 2022

    2022

  80. [88]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowd- hery, and D. Zhou, “Self-Consistency Improves Chain of Thought Reasoning in Language Models,”arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  81. [89]

    Plan-and-Solve Prompting: Improving Zero-Shot Chain-of- Thought Reasoning by Large Language Models,

    L. Wang, W. Xu, Y . Lan, Z. Hu, Y . Lan, R. K.-W. Lee, and E.- P. Lim, “Plan-and-Solve Prompting: Improving Zero-Shot Chain-of- Thought Reasoning by Large Language Models,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

    2023

  82. [90]

    J. Wang, M. Chen, S. Zhang, N. Karaev, J. Sch ¨onberger, P. Labatut, P. Bojanowski, D. Novotny, A. Vedaldi, and C. Rupprecht, “VGGT-Ω,” arXiv preprint arXiv:2605.15195, 2026

    2026 arXiv

  83. [91]

    W AFT: Warping-Alone Field Transforms for Optical Flow,

    Y . Wang and J. Deng, “W AFT: Warping-Alone Field Transforms for Optical Flow,”arXiv preprint arXiv:2506.21526, 2025

    2025

  84. [92]

    Don’t Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs,

    M. K. Janjua, H. Silva, D. Niu, and B. Rashidi, “Don’t Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026, pp. 5165–5174

    2026

  85. [93]

    Expressive Body Capture: 3D Hands, Face, and Body From a Single Image,

    G. Pavlakos, V . Choutas, N. Ghorbani, T. Bolkart, A. A. Osman, D. Tzionas, and M. J. Black, “Expressive Body Capture: 3D Hands, Face, and Body From a Single Image,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 10 975–10 985. 1...

    2019

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.