pith. sign in

arxiv: 2606.27999 · v2 · pith:XYQABNKDnew · submitted 2026-06-26 · 💻 cs.CV

HumanMoveVQA: Can Video MLLMs reason about human movement in videos?

Pulkit Gera , Faegheh Sardari , Asmar Nadeem , Valentina Bono , Padraig Boulton , Adrian Hilton , Armin Mustafa This is my paper

Pith reviewed 2026-06-30 09:44 UTC · model grok-4.3

classification 💻 cs.CV
keywords HumanMoveVQAvideo MLLMshuman motion understandingtrajectory reasoningorientation reasoning3D motion trackingworld coordinate systembenchmark
0
0 comments X

The pith

State-of-the-art video MLLMs show a clear gap in global human trajectory and orientation reasoning, but this gap shrinks after fine-tuning on world-consistent 3D supervision.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents HumanMoveVQA, a benchmark that tests multimodal large language models on their ability to reason about human movement using a first-frame anchored world coordinate system that preserves translation and rotation. It builds more than 10,000 question-answer pairs across seven categories that target motion aggregation, sequential ordering, and trajectory inference rather than local joint motions or scene events. Evaluations find that leading proprietary models perform poorly on these questions. The same paper shows that the shortfall is addressable: fine-tuning an open-source model with the generated world-consistent supervision produces clear gains. The work therefore frames deep human motion understanding as a solvable problem once models receive training signals that respect 3D geometry over time.

Core claim

The authors claim that current video MLLMs collapse complex human motion into coarse labels and fail at exocentric trajectory and orientation reasoning, yet demonstrate that the capability is learnable by constructing a benchmark from 3D motion tracks lifted from 2D video and showing measurable improvement after targeted fine-tuning.

What carries the argument

A multi-stage pipeline that lifts 2D video observations into world-consistent 3D motion tracks anchored to the first frame, then generates structured question-answer pairs across seven reasoning categories.

If this is right

  • Video MLLMs can be trained to maintain global spatial awareness of human movement when given world-consistent supervision.
  • Benchmarks that only test local actions or scene events will miss the trajectory-level failures revealed here.
  • Open-source models can reach closer performance to proprietary ones on motion reasoning once exposed to the same 3D-derived data.
  • Future model development should incorporate explicit geometric constraints rather than relying solely on semantic labels.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same lifting approach could be applied to other motion domains such as animal locomotion or robotic manipulation to create analogous benchmarks.
  • If the 3D tracks prove reliable, the benchmark could serve as a diagnostic tool to isolate whether failures stem from perception or from reasoning layers.
  • Models that succeed on this benchmark may also generalize better to downstream tasks that require predicting future human positions in space.

Load-bearing premise

The pipeline that converts 2D observations into 3D tracks produces trajectories and orientations accurate enough to create valid question-answer pairs.

What would settle it

Direct comparison of the generated 3D tracks against synchronized motion-capture ground truth, or a controlled experiment showing no accuracy gain after fine-tuning on the benchmark data.

Figures

Figures reproduced from arXiv: 2606.27999 by Adrian Hilton, Armin Mustafa, Asmar Nadeem, Faegheh Sardari, Padraig Boulton, Pulkit Gera, Valentina Bono.

Figure 1
Figure 1. Figure 1: Overview of HumanMoveVQA. We evaluate the ability of VideoMLLMs to reason about [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of pipeline generating HumanMoveVQA. Given an input video, we use [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Motion characteristics across the three full datasets before train/test splitting. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative results on the EMDB dataset. The world-view depicts extracted 3D SMPL-X poses and is shown for illustration only (not provided to the models). Green denotes the correct option. We compare predictions from multiple MLLMs across seven reasoning categories, where our model demonstrates more accurate and consistent responses. More results in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Heatmap visualizations of training strategies and cross-dataset generalization. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative results on the EMDB dataset. The world-view visualization depicts extracted 3D SMPL-X poses and is shown for illustration only (not provided to the models). Green denotes the correct option. We compare predictions from multiple MLLMs across seven reasoning categories, where our model demonstrates more accurate and consistent responses. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative results on the RICH dataset. The world-view visualization depicts extracted 3D SMPL-X poses and is shown for illustration only (not provided to the models). Green denotes the correct option. We compare predictions from multiple MLLMs across seven reasoning categories, where our model demonstrates more accurate and consistent responses. EgoBody – The EgoBody consists of static multi-view recordi… view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative results on the EgoBody dataset. The world-view visualization depicts extracted 3D SMPL-X poses and is shown for illustration only (not provided to the models). Green denotes the correct option. We compare predictions from multiple MLLMs across seven reasoning categories, where our model demonstrates more accurate and consistent responses. Effect of Frame Count – We evaluated models trained with… view at source ↗
Figure 9
Figure 9. Figure 9: Performance comparison across reasoning categories on the HumanMoveVQAbenchmark. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Effect of Frame Count. Qwen3-VL 8B is fine-tuned (SFT) with varying numbers of input [PITH_FULL_IMAGE:figures/full_fig_p018_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Effect of input resolution. Qwen3-VL 8B is fine-tuned (SFT) at different input resolutions [PITH_FULL_IMAGE:figures/full_fig_p018_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: System prompt used for evaluation of models. We use the same system prompt for training [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Prompt used for clothing caption generation. [PITH_FULL_IMAGE:figures/full_fig_p022_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Examples of category-wise reasoning traces for a sample EMDB video. For each category, [PITH_FULL_IMAGE:figures/full_fig_p023_14.png] view at source ↗
read the original abstract

Despite the rapid advance of Multimodal Large Language Models (MLLMs) in high-level video understanding, a fundamental bottleneck remains: these models collapse complex human motion into coarse semantic labels. Existing benchmarks mostly focus on scene-centric events or local joint articulations, failing to probe global human motion in space over time (trajectory and orientation changes). We introduce HumanMoveVQA, the first comprehensive benchmark designed to evaluate global trajectory and orientation reasoning from an exocentric perspective. Our benchmark utilizes a first-frame anchored world coordinate system, preserving translation and rotation relative to a fixed starting point. We propose a scalable, multi-stage pipeline that lifts 2D video observations into world-consistent 3D motion tracks to generate over 10K structured question-answer pairs across seven reasoning categories, including motion aggregation, sequential ordering, and trajectory-level inference. Our extensive evaluation reveals a critical capability gap in state-of-the-art proprietary models on deep human motion understanding. However, we demonstrate that this is a learnable problem; by fine-tuning an open-source baseline with our targeted, world-consistent supervision, we achieve a significant improvement. HumanMoveVQA establishes a rigorous geometric foundation for developing next-generation, movement-aware video understanding models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces HumanMoveVQA, a benchmark for video MLLMs focused on global human trajectory and orientation reasoning from an exocentric view using a first-frame-anchored world coordinate system. It describes a multi-stage pipeline that lifts 2D video to world-consistent 3D motion tracks to automatically generate over 10K QA pairs across seven reasoning categories (motion aggregation, sequential ordering, trajectory-level inference, etc.). Extensive evaluations on proprietary and open-source models reveal capability gaps in deep motion understanding, while fine-tuning an open-source baseline on the generated data yields significant gains. The work positions the benchmark as establishing a geometric foundation for movement-aware video models.

Significance. If the 3D lifting produces sufficiently accurate tracks, the benchmark would usefully expose limitations in current MLLMs' geometric motion reasoning that existing scene-centric or local-joint benchmarks miss, and the fine-tuning result would demonstrate that targeted world-consistent supervision is effective. The scalable pipeline construction and the explicit focus on trajectory/orientation changes are strengths that could guide future model development in video understanding.

major comments (2)
  1. [Abstract / pipeline description] Abstract and pipeline description: the central claims (capability gap in proprietary models and learnability via fine-tuning) rest on the validity of the >10K QA pairs. No quantitative validation of the 2D-to-3D lifting is reported (e.g., trajectory RMSE or orientation error against MoCap ground truth, multi-view reconstruction, or even aggregate manual inspection metrics on a held-out set). Without such evidence, systematic errors in translation, rotation, or occlusion handling could render questions in the motion-aggregation, sequential-ordering, and trajectory-inference categories noisy, making both the reported failures and the fine-tuning gains potentially artifactual.
  2. [Evaluation section] Evaluation section: the manuscript supplies no details on the number of source videos, the exact models and versions tested, the train/test split sizes for the fine-tuning experiment, or any error analysis / inter-annotator agreement on the generated QA pairs. These omissions prevent assessment of whether the observed gaps are robust or sensitive to pipeline artifacts.
minor comments (1)
  1. [Abstract] The seven reasoning categories are listed but their precise question templates and answer formats are not exemplified in the abstract; adding one concrete example per category would improve clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below and will revise the manuscript accordingly to provide the requested validation and experimental details.

read point-by-point responses

  1. Referee: [Abstract / pipeline description] Abstract and pipeline description: the central claims (capability gap in proprietary models and learnability via fine-tuning) rest on the validity of the >10K QA pairs. No quantitative validation of the 2D-to-3D lifting is reported (e.g., trajectory RMSE or orientation error against MoCap ground truth, multi-view reconstruction, or even aggregate manual inspection metrics on a held-out set). Without such evidence, systematic errors in translation, rotation, or occlusion handling could render questions in the motion-aggregation, sequential-ordering, and trajectory-inference categories noisy, making both the reported failures and the fine-tuning gains potentially artifactual.

    Authors: We agree that the absence of quantitative validation for the 2D-to-3D lifting is a limitation that affects confidence in the QA pair quality. The manuscript does not currently report trajectory RMSE, orientation error, or manual inspection metrics. In the revision we will add a validation subsection reporting these metrics on a held-out set (using available ground truth where possible and aggregate manual checks otherwise), along with discussion of occlusion and error handling. This will directly address whether the reported model gaps and fine-tuning gains could be artifactual. revision: yes

  2. Referee: [Evaluation section] Evaluation section: the manuscript supplies no details on the number of source videos, the exact models and versions tested, the train/test split sizes for the fine-tuning experiment, or any error analysis / inter-annotator agreement on the generated QA pairs. These omissions prevent assessment of whether the observed gaps are robust or sensitive to pipeline artifacts.

    Authors: We acknowledge these omissions in the current manuscript. The revised version will explicitly state the number of source videos, the precise model names and versions evaluated, the train/test split sizes used for fine-tuning, and will include error analysis together with inter-annotator agreement statistics for the generated QA pairs. These additions will enable readers to evaluate the robustness of the capability gaps and fine-tuning results. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical benchmark and fine-tuning results are independent of internal fits

full rationale

The paper presents an empirical benchmark constructed via a multi-stage 2D-to-3D lifting pipeline and reports model evaluations plus a fine-tuning experiment on the resulting >10K QA pairs. No equations, parameters, or predictions are defined in terms of the target quantities (e.g., no fitted trajectory parameters renamed as 'predictions' of motion reasoning). The central claims rest on external model performance numbers and a supervised fine-tuning run, which are falsifiable against the generated data rather than reducing to self-definition or self-citation chains. The unverified accuracy of the lifting pipeline is a validity concern, not a circularity reduction. This is a standard self-contained empirical contribution.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The benchmark rests on the accuracy of an external 2D-to-3D lifting pipeline whose error characteristics are not quantified in the abstract; no new physical constants or fitted parameters are introduced.

axioms (1)
  • domain assumption The multi-stage pipeline produces world-consistent 3D motion tracks from 2D observations that are accurate enough for the intended reasoning tasks.
    Invoked when the abstract states the pipeline is used to generate the QA pairs.

pith-pipeline@v0.9.1-grok · 5764 in / 1311 out tokens · 29333 ms · 2026-06-30T09:44:38.298452+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

48 extracted references · 10 canonical work pages · 4 internal anchors

  1. [1]

    GPT-4o System Card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  2. [2]

    Qwen3-vl technical report, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    2025

  3. [3]

    InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang, Yuchen Duan, Xuehui Wang, Haoran H...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    OpenReview

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: a dataset for understanding complex web videos via question answering. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educ...

    work page doi:10.1609/aaai.v33i01 2019

  5. [5]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InCVPR, 2025

    2025

  6. [6]

    Next-qa: Next phase of question- answering to explaining temporal actions.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021

    2021

  7. [7]

    arXiv preprint arXiv:2405.09711 (2024) 5

    Bo Wu and Shoubin Yu. Star: A benchmark for situated reasoning in real-world videos.ArXiv, abs/2405.09711, 2024

    work page arXiv 2024

  8. [8]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  9. [9]

    LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding

    Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long- context interleaved video-language understanding, 2024. URL https://arxiv.org/abs/ 2407.15754

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    Lvbench: An extreme long video understanding benchmark, 2024

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Lvbench: An extreme long video understanding benchmark, 2024

    2024

  11. [11]

    Sqa3d: Situated question answering in 3d scenes

    Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. InInternational Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=IDJx97BC38. 10

    2023

  12. [12]

    Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment, 2023

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Wang HongFa, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Cai Wan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment, 2023

    2023

  13. [13]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. InProceedings of the IEEE/CVF international conference on computer vision, pages 1728–1738, 2021

    2021

  14. [14]

    Actionart: Advancing multimodal large models for fine-grained human-centric video understanding.arXiv preprint arXiv:2504.18152, 2025

    Yi-Xing Peng, Qize Yang, Yu-Ming Tang, Shenghao Fu, Kun-Yu Lin, Xihan Wei, and Wei-Shi Zheng. Actionart: Advancing multimodal large models for fine-grained human-centric video understanding.arXiv preprint arXiv:2504.18152, 2025

    work page arXiv 2025

  15. [15]

    Mmhu: A massive-scale multimodal benchmark for human behavior understanding, 2025

    Renjie Li, Ruijie Ye, Mingyang Wu, Hao Frank Yang, Zhiwen Fan, Hezhen Hu, and Zhengzhong Tu. Mmhu: A massive-scale multimodal benchmark for human behavior understanding, 2025

    2025

  16. [16]

    Motionllm: Understanding human behaviors from human motions and videos.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    Ling-Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. Motionllm: Understanding human behaviors from human motions and videos.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  17. [17]

    Motionbench: Benchmarking and improving fine-grained video motion understanding for vision language models, 2025

    Wenyi Hong, Yean Cheng, Zhuoyi Yang, Weihan Wang, Lefan Wang, Xiaotao Gu, Shiyu Huang, Yuxiao Dong, and Jie Tang. Motionbench: Benchmarking and improving fine-grained video motion understanding for vision language models, 2025

    2025

  18. [18]

    Spatialreasoner: Towards explicit and generalizable 3d spatial reasoning

    Wufei Ma, Yu-Cheng Chou, Qihao Liu, Xingrui Wang, Celso M de Melo, Jianwen Xie, and Alan Yuille. Spatialreasoner: Towards explicit and generalizable 3d spatial reasoning. In Advances in Neural Information Processing Systems, volume 38, 2025

    2025

  19. [19]

    Spatialvlm: Endowing vision-language models with spatial reasoning capabilities

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14455–14465, June 2024

    2024

  20. [20]

    Spatial- thinker: Reinforcing 3d reasoning in multimodal llms via spatial rewards, 2025

    Hunar Batra, Haoqin Tu, Hardy Chen, Yuanze Lin, Cihang Xie, and Ronald Clark. Spatial- thinker: Reinforcing 3d reasoning in multimodal llms via spatial rewards, 2025

    2025

  21. [21]

    EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild

    Manuel Kaufmann, Jie Song, Chen Guo, Kaiyue Shen, Tianjian Jiang, Chengcheng Tang, Juan José Zárate, and Otmar Hilliges. EMDB: The Electromagnetic Database of Global 3D Human Pose and Shape in the Wild. InInternational Conference on Computer Vision (ICCV), 2023

    2023

  22. [22]

    Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J

    Chun-Hao P. Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J. Black. Capturing and inferring dense full-body human-scene contact. InProceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  23. [23]

    Egobody: Human body shape and motion of interacting people from head-mounted devices

    Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Marc Pollefeys, Federica Bogo, and Siyu Tang. Egobody: Human body shape and motion of interacting people from head-mounted devices. InEuropean Conference on Computer Vision, 2022

    2022

  24. [24]

    Mvbench: A comprehensive multi-modal video understanding benchmark.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understanding benchmark.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2024

  25. [25]

    Spatialrgpt: Grounded spatial reasoning in vision-language models

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. In NeurIPS, 2024

    2024

  26. [26]

    Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Fei-Fei Li, and Saining Xie. Thinking in space: How multimodal large language models see, remember, and recall spaces.2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10632– 10643, 2024. 11

    2025

  27. [27]

    4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation

    Chiao-An Yang, Ryo Hachiuma, Sifei Liu, Subhashree Radhakrishnan, Raymond A Yeh, Yu- Chiang Frank Wang, and Min-Hung Chen. 4d-rgpt: Toward region-level 4d understanding via perceptual distillation.arXiv preprint arXiv:2512.17012, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  28. [28]

    Learning situated awareness in the real world, 2026

    Chuhan Li, Ruilin Han, Joy Hsu, Yongyuan Liang, Rajiv Dhawan, Jiajun Wu, Ming-Hsuan Yang, and Xin Eric Wang. Learning situated awareness in the real world, 2026

    2026

  29. [29]

    Video-LLaV A: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaV A: Learning united visual representation by alignment before projection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors,Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, November 2024. Association fo...

    2024

  30. [30]

    Videogpt+: Integrating image and video encoders for enhanced video understanding,

    Muhammad Maaz, Hanoona Abdul Rasheed, Salman H. Khan, and Fahad Shahbaz Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding.ArXiv, abs/2406.09418, 2024

    work page arXiv 2024

  31. [31]

    Llava- video: Video instruction tuning with synthetic data.Trans

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Llava- video: Video instruction tuning with synthetic data.Trans. Mach. Learn. Res., 2025, 2024

    2025

  32. [32]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024

    2024

  33. [33]

    Motionbank: A large-scale video motion benchmark with disentangled rule-based annotations.arXiv preprint arXiv:2410.13790, 2024

    Liang Xu, Shaoyang Hua, Zili Lin, Yifan Liu, Feipeng Ma, Yichao Yan, Xin Jin, Xiaokang Yang, and Wenjun Zeng. Motionbank: A large-scale video motion benchmark with disentangled rule-based annotations.arXiv preprint arXiv:2410.13790, 2024

    work page arXiv 2024

  34. [34]

    PoseScript: 3D Human Poses from Natural Language

    Delmas, Ginger and Weinzaepfel, Philippe and Lucas, Thomas and Moreno-Noguer, Francesc and Rogez, Grégory. PoseScript: 3D Human Poses from Natural Language. InECCV, 2022

    2022

  35. [35]

    Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J. Black. Chatpose: Chatting about 3d human pose. InCVPR, 2024

    2024

  36. [36]

    HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips. InICCV, 2019

    2019

  37. [37]

    Mllm-4d: To- wards visual-based spatial-temporal intelligence

    Xingyilang Yin, Chengzhengxu Li, Jiahao Chang, Chi-Man Pun, and Xiaodong Cun. Mllm-4d: Towards visual-based spatial-temporal intelligence.arXiv preprint arXiv:2603.00515, 2026

    work page arXiv 2026

  38. [38]

    Motion-x: A large-scale 3d expressive whole-body human motion dataset.Advances in Neural Information Processing Systems, 36:25268–25280, 2023

    Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large-scale 3d expressive whole-body human motion dataset.Advances in Neural Information Processing Systems, 36:25268–25280, 2023

    2023

  39. [39]

    Prompthmr: Promptable human mesh recovery.2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1148–1159, 2025

    Yufu Wang, Yu Sun, Priyanka Patel, Kostas Daniilidis, Michael John Black, and Muhammed Kocabas. Prompthmr: Promptable human mesh recovery.2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1148–1159, 2025

    2025

  40. [40]

    Motionscript: Natural language descriptions for expressive 3d human motions

    Payam Jome Yazdian, Rachel Lagasse, Hamid Mohammadi, Eric Liu, Li Cheng, and Angelica Lim. Motionscript: Natural language descriptions for expressive 3d human motions. In2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2025

    2025

  41. [41]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  42. [42]

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. InProceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 10975–10985, 2019

    2019

  43. [43]

    Gemini 3 flash model card

    Google DeepMind. Gemini 3 flash model card. https://storage.googleapis.com/ deepmind-media/Model-Cards/Gemini-3-Flash-Model-Card.pdf , 2025. Accessed: 2026-05-01. 12

    2025

  44. [44]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguist...

    2024

  45. [45]

    How sensitive is the training to the number of frames?

  46. [46]

    How sensitive is the training to the resolution of the videos?

  47. [47]

    How does it perform if trained with a short reasoning trace?

  48. [48]

    inverted V

    How does our model generalize to joint-level reasoning benchmarks like ActionArt? 14 Table 5: Evaluation of video MLLM models on theRICHsplit. Results are reported across seven reasoning categories and an overall normalized score. Green, orange, and blue indicate the best, second-best, and third-best results per column, respectively for MLLMs. Model Frame...