Pith. sign in

REVIEW 4 major objections 5 minor 104 references

MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MMHU is a large-scale, unified benchmark that ties together human motion, trajectory, text descriptions, and 13 safety-critical behaviors for autonomous driving, and fine-tuning on it improves four behavior-understanding tasks.

desk verdict A genuinely useful unified benchmark, but the unvalidated VLM labels and a frame-count mismatch keep it from being trustworthy as-is. read the letter →

arxiv 2507.12463 v1 pith:2KDICDXY submitted 2025-07-16 cs.CV

Renjie Li , Ruijie Ye , Mingyang Wu , Hao Frank Yang , Zhiwen Fan , Hezhen Hu , Zhengzhong Tu This is my paper · ORCID
classification cs.CV
keywords MMHUhumanbehaviorunderstandingautonomousdrivingbenchmarkSMPLmotiontrajectorypredictiontext-to-motiongenerationvisualquestionansweringintention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a single benchmark can unify the different ways autonomous driving researchers study human behavior: body motion, trajectory, textual descriptions, intentions, and safety-critical actions. The benchmark, MMHU, contains 57,000 human instances and 1.73 million frames from driving datasets, web videos, and self-collected footage, each annotated with SMPL 3D motion, trajectories, hierarchical text captions, and 13 binary driving-relevant behavior labels such as crossing the street, riding a bike, or using a phone. The authors argue that existing driving datasets cover only one aspect of human behavior, leaving no common testbed, and that MMHU fills that gap by providing baselines for motion prediction, text-to-motion generation, behavior question-answering, and intention prediction. If the benchmark is accepted as ground truth, it would give driving-model and vision-language researchers a shared resource for training and evaluating human-behavior understanding in traffic.

What carries the argument

The load-bearing object is the annotation pipeline. It starts from monocular video, detects and tracks people, and reconstructs SMPL body parameters for each person, using bounding-box priors where available and unifying to 10 FPS elsewhere. Missing frames are filled by a spherical-interpolation formula $\hat{S}_{k+j} = \frac{\sin((1-j/m)\theta)}{\sin\theta}\hat{S}_{k-1} + \frac{\sin(j\theta/m)}{\sin\theta}\hat{S}_{k+m+1}$ with $\cos\theta = S_{k-1} \cdot S_{k+m+1}$. Text comes in two levels: low-level per-joint descriptions are generated by rule from the SMPL angles and distances, then a vision-language model summarizes them into a high-level caption. Critical behaviors are obtained by asking a vision-language model yes-or-no questions for 13 predefined behaviors; because pretrained vision-language models are noisy, 10% of instances are human-labeled and that set is used to fine-tune the annotation model, which labels the remaining 90%. The same human-labeled set supplies the testing subset used for behavior question-answering evaluation.

What would settle it

Take a random sample of MMHU clips that the vision-language model labeled, have independent human annotators relabel the 13 behaviors with the same interface, and compare: if agreement is near chance for any behavior, the reported VQA gains reflect learning the labeler's bias rather than visual understanding. Likewise, on a subset of driving clips with synchronized multi-view or motion-capture reference, compare the reconstructed SMPL motion against that reference; a large mean per-joint error would invalidate the motion-prediction and motion-generation evaluations.

Watch

Extended reading notes

Core claim

MMHU is a unified, human-centric benchmark for autonomous driving, built from 57k human motion clips and 1.73M frames. Each instance carries a 3D SMPL motion sequence and trajectory, a two-level text description (joint-level low-level descriptions aggregated into high-level captions), and labels for 13 critical behaviors that matter for driving safety. A human-in-the-loop pipeline keeps labeling scalable: a small subset is annotated by people, and a fine-tuned vision-language model labels the remainder. The paper demonstrates that fine-tuning on MMHU improves motion prediction on an existing 3D human motion benchmark, text-to-motion generation FID, behavior-VQA accuracy and F1, and intention prediction on a standard crossing dataset, supporting the claim that the dataset is a useful unified resource.

Load-bearing premise

The load-bearing premise is that the machine-generated labels, namely 3D body motion reconstructed from ordinary monocular video and the 13 behavior labels produced by a vision-language model, are accurate enough to count as ground truth, but the paper reports no independent check of either.

Editorial extensions

If this is right

  • A shared testbed now exists where motion, trajectory, captions, and behavior labels can be trained and evaluated together for driving scenes.
  • A physics-informed motion predictor achieves the lowest mean per-joint position error on the benchmark's test subset, establishing a first baseline for future motion-prediction work.
  • Fine-tuning on MMHU improves cross-dataset performance: motion prediction error on an existing 3D human motion benchmark drops, text-to-motion FID falls from roughly 39 to 1.86, behavior-VQA accuracy rises by about 16 points, and intention-prediction accuracy on a standard crossing dataset rises by about 7 points.
  • The benchmark's question-answering format extends intention prediction from a binary crossing-or-not decision to 13 simultaneous behavioral attributes.
  • Because the dataset is released, other researchers can reproduce the fine-tuning gains and use MMHU as a common evaluation set for human-centric driving models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the auto-labeled behaviors are noisy, the fine-tuning gains in behavior VQA may partly reflect learning the labeler's bias rather than genuine visual understanding; a human-validated subset would separate the two effects.
  • The 90/10 split between vision-language-labeled and human-labeled instances means the larger training portion is a soft label source, so researchers should test whether training on the human-labeled subset alone reproduces the reported gains.
  • The same annotation pipeline could be extended to rarer behaviors and to group interactions, where the current 13 labels are too coarse for questions like whether a person is about to wave at the driver.
  • Because motion and trajectory are extracted from the same reconstructed SMPL stream, the benchmark's motion tasks are only as independent as the reconstruction; treating reconstructed motion as ground truth limits its usefulness for evaluating reconstruction itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MMHU, a large-scale multimodal benchmark for human behavior understanding in driving scenarios, comprising 57k human instances and 1.73M frames sourced from Waymo, YouTube, and self-collected driving videos. Each instance is annotated with SMPL-based 3D motion and trajectory, hierarchical text descriptions (low-level joint-wise and high-level semantic captions), and 13 binary critical behavior labels (e.g., crossing street, using phone, riding bicycle). The annotation pipeline combines automated reconstruction (Wham for motion, VLMs for text and behavior) with a human-in-the-loop component: 10% of instances are human-labeled for behaviors, and a VLM fine-tuned on this subset labels the remaining 90%. The paper provides dataset statistics, evaluates baselines on motion prediction, motion generation, and behavior VQA, and reports fine-tuning experiments showing performance gains on 3DPW, JAAD, and MMHU test sets.

Significance. If the annotations are trustworthy, MMHU fills a genuine gap: no existing driving dataset unifies 3D motion, trajectory, hierarchical text, and a broad set of driving-relevant behavior labels at this scale. The multi-task evaluation suite and the demonstration that fine-tuning on MMHU improves downstream performance on external datasets (3DPW, JAAD) are valuable contributions. The paper also ships a project page and details of the annotation pipeline. However, the central premise — that the automated labels are accurate enough to serve as ground truth — is not yet validated in the manuscript, and several inconsistencies in the experimental setup need to be resolved before the resource can be fully trusted.

major comments (4)
  1. [Sec. 3.4, 'Critical Behavior Labling'; Sec. 5.1; Table 3] The fine-tuned annotation VLM is applied to 90% of the dataset (MMHU-V, 47k instances), and these labels are used as training targets in the fine-tuning experiments of Sec. 5.4. Yet the manuscript never reports the annotation VLM's accuracy, per-behavior precision/recall, or agreement with the human labels on the held-out MMHU-T subset (840 instances). Given that even the strongest generalist VLM in Table 3 reaches only 64.8 micro-F1 on this test set, the accuracy of the automated labeler is not self-evident. The paper's own Limitation section states that 'the capability of VLMs to understand human behavior limits the development of a fully automatic annotation pipeline,' which reinforces the need for direct validation. Please report the annotation VLM's performance on MMHU-T, including per-behavior F1, and compare it to human inter-annotator agreement. Without this, the benchmark's behavior labels and the reported fine-tuning improvements (Tables 5, 7) may reflect learning the annotation VLM's biases rather than true human behavior understanding.
  2. [Sec. 3.2, 'Motion Extraction' and 'Trajectory from Motion'; Table 4] The SMPL motion sequences reconstructed by Wham from monocular video are treated as ground truth for the motion prediction and motion generation tasks, with no validation against motion capture, manual inspection, or existing 3D ground truth in driving scenes. Since the motion prediction baselines in Table 4 are evaluated on these pseudo-ground-truth sequences, systematic reconstruction errors (e.g., in occlusion, small pedestrians, or fast motion) could dominate the reported MPJPE numbers and make the benchmark measure reconstruction noise rather than true motion. Please provide a validation study: for example, evaluate Wham on a subset with known 3D poses (from driving datasets with LiDAR or multi-view setups, or a small motion-capture session), report per-instance error statistics, and discuss failure modes affecting the benchmark.
  3. [Sec. 3.1, 'Videos Acquisition'; Sec. 3.5, 'Statistics'; Table 1] There is a numerical inconsistency in the frame counts. Section 3.1 states that 1.73M raw frames were collected in total, but the per-source counts given there are 73K (Waymo) + 318.25K (YouTube) + 2393.96K (self-collected) = 2785.21K, which is about 2.79M frames, not 1.73M. Section 3.5 and Table 1 derive 1.73M from 48 hours at 10 Hz, which likely refers to the final processed motion clips rather than raw collected video. Please clarify the distinction between raw collected frames and final dataset frames, and correct the per-source statistics so that the numbers are self-consistent.
  4. [Sec. 5.4, 'Behavior VQA'; App. B.3, 'Human Behavior VQA Details'] The fine-tuning experiment for Behavior VQA is presented in Table 5 as improving 'QWen2.5-VL', and the baseline in Table 3 is Qwen2.5-VL-7B (micro-F1 44.7, matching the baseline in Table 5). However, Appendix B.3 says that LoRA fine-tuning is applied to 'Qwen2.5-3B-Instruct' with the visual branch frozen. Qwen2.5-3B-Instruct is a text-only language model, not a vision-language model, and it is a different size and architecture from the stated baseline. If a different model was actually fine-tuned, the comparison in Table 5 is not a valid measure of MMHU's benefit. Please specify the exact model used, its parameter count, and whether the baseline in Table 5 is the same model; if the fine-tuned model is smaller or different, the claim of a 15+ point gain needs to be re-evaluated with a matched baseline.
minor comments (5)
  1. [Fig. 4 caption] The caption reads 'Statics of MMHU'; this should be 'Statistics of MMHU'.
  2. [Table 4 and Sec. 5.2] The motion prediction baseline 'AuxFormer' is cited as reference [1] in Table 4, but the reference list contains both [1] (Xu et al., Auxiliary Tasks Benefit 3D Skeleton-Based Human Motion Prediction) and [55] (Goncalves & Busso, Auxformer for audiovisual emotion recognition). Please clarify which paper is the actual baseline and correct the citation.
  3. [Sec. 5.2, 'Motion Prediction'] The text says the baselines take the first 25 frames of 50 randomly selected frames as input and predict the remaining 25, but Table 4 reports errors at frame_id 1, 3, 7, 9, 13, 17, 21, 24. Please specify how the frame_id corresponds to the prediction horizon and whether the same frame indices are used for all baselines.
  4. [Sec. 4, 'Behavior VQA' and App. B.1] The evaluation requires a model to answer both a direct question and a counter question correctly, but the main text does not specify how the two answers are combined in the reported accuracy and F1. Please state the aggregation rule clearly in the main text.
  5. [Sec. 3.3, 'Low-level Text Annotation'] The sentence 'As shown in 2' should refer to 'Fig. 2' explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MMHU is an empirical benchmark built with an explicit annotation pipeline, and its fine-tuning gains are measured against separately human-labeled or external evaluation sets.

full rationale

MMHU is a dataset/benchmark construction paper rather than a derivation, so the circularity patterns must be assessed against its empirical claims. The paper is transparent about its annotation pipeline: SMPL motion is reconstructed with Wham from monocular video (Sec 3.2), and the 13 critical-behavior labels on the large MMHU-V portion are produced by a VLM fine-tuned on a 10% human-labeled subset (Sec 3.4). These are data-generation steps, and the paper does not hide that the VLM labels are model outputs. The key evaluation claims are not circular: behavior VQA baselines and the fine-tuned Qwen2.5-VL are evaluated on MMHU-T, which is human-labeled and separate from the VLM-labeled training portion (Sec 5.1); motion prediction fine-tuning is evaluated on 3DPW (Table 6); intention prediction is evaluated on JAAD (Table 7). None of these evaluation sets reduces to the label generator by construction. The motion-generation FID improvement is measured on an MMHU test split whose ground truth is Wham-recovered SMPL, which raises a validity concern about the accuracy of the pseudo-ground-truth motion, but it is not a definitional circularity: the text-to-motion model must still map free-form language to SMPL sequences, and the FID gap is an empirical quantity. The paper itself acknowledges the label-source risk in its Limitation section: 'The capability of VLMs to understand human behavior limits the development of a fully automatic annotation pipeline.' No load-bearing self-citations appear; the cited Wham, PoseScript, MotionDiffuse, TrEP, and 3DPW/JAAD are external works. The skeptical concern about unvalidated VLM labels on the 90% auto-labeled split is a legitimate annotation-accuracy and benchmark-validity issue, not a case of a prediction being forced by construction or a fitted parameter being renamed as a result. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the accuracy of automated annotations (Wham for motion, VLM for behaviors and text) and on the completeness of the 13-behavior taxonomy. These are modeled as domain assumptions; no independent evidence is provided beyond the human-labeled 10% subset.

free parameters (3)
  • IoU threshold for detection matching and occlusion filtering = 0.2
    Used in Sec 3.2 to select reconstruction candidates and discard occluded frames; chosen without sensitivity analysis.
  • Minimum video fragment length = 10 seconds
    Fragments shorter than 10s are filtered in Sec 3.1; this shapes the motion sequence duration distribution and the 1.73M-frame total.
  • Joint mention frequency threshold for low-level text = 50% of frames
    In Appendix A.3, only joints mentioned in at least half the frames are retained; affects the content of both low- and high-level text descriptions.
assumptions (4)
  • domain assumption Wham's monocular SMPL reconstructions are accurate enough to serve as ground truth for human motion in driving scenes.
    Motion extraction in Sec 3.2 relies entirely on Wham; no validation against motion capture or manual inspection in driving scenes is reported.
  • domain assumption The 13 behaviors identified by prompting a VLM to summarize critical behaviors form a complete and useful taxonomy for autonomous driving safety.
    Sec 3.4 obtains the behavior list from a VLM; no expert panel or standard is cited, and no coverage analysis is provided.
  • ad hoc to paper The fine-tuned VLM labeler's outputs on the unlabeled 90% of instances are correct.
    Sec 3.4 fine-tunes on 10% human labels but reports no accuracy or agreement of the labeler against a held-out human-labeled set.
  • domain assumption LLM aggregation of low-level pose descriptions into high-level text preserves the semantics of the motion.
    The hierarchical text pipeline in Sec 3.3 assumes the LLM does not introduce errors; no quality check on the captions is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding." pith.science (2026). https://pith.science/paper/2KDICDXY

@misc{pith2026250712463,
  author       = {Pith},
  title        = {Pith review of: MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2KDICDXY}},
  note         = {Machine review of arXiv:2507.12463}
}
abstract

Humans are integral components of the transportation ecosystem, and understanding their behaviors is crucial to facilitating the development of safe driving systems. Although recent progress has explored various aspects of human behavior$\unicode{x2014}$such as motion, trajectories, and intention$\unicode{x2014}$a comprehensive benchmark for evaluating human behavior understanding in autonomous driving remains unavailable. In this work, we propose $\textbf{MMHU}$, a large-scale benchmark for human behavior analysis featuring rich annotations, such as human motion and trajectories, text description for human motions, human intention, and critical behavior labels relevant to driving safety. Our dataset encompasses 57k human motion clips and 1.73M frames gathered from diverse sources, including established driving datasets such as Waymo, in-the-wild videos from YouTube, and self-collected data. A human-in-the-loop annotation pipeline is developed to generate rich behavior captions. We provide a thorough dataset analysis and benchmark multiple tasks$\unicode{x2014}$ranging from motion prediction to motion generation and human behavior question answering$\unicode{x2014}$thereby offering a broad evaluation suite. Project page : https://MMHU-Benchmark.github.io.

Figures

Figures reproduced from arXiv: 2507.12463 by the authors.

Figure 1
Figure 1. We propose MMHU, a large-scale dataset for human behavior understanding. We collected 57k human instances with diverse behaviors such as playing mobile phone, holding object, or using mobility devices, from diverse scenes such as in the city, school, park, and alley. We provide rich annotations including motion and trajectory, text descriptions for human motions, and recognize the behaviors that are critical to driv… view at source ↗
Figure 2
Figure 2. Data Collection and Annotation. (Left) We collect data from three sources: the Waymo dataset, the YouTube videos, and the self-collected or paid driving videos. (Right) We demonstrate the annotation pipeline; we first filter and cut the raw videos based on the rough human detection results. Then we reconstruct the SMPL motion for each detected frame. The missing frames are further recovered by an interpolation proce… view at source ↗
Figure 3
Figure 3. Visualization of the MMHU dataset. For each human instance, the first line shows the video frames that is sampled from the video clips. The human instance is highlighted using a red bounding box. We crop and zoom-in the human instance for a clearer view. Under each of the frames shows the corresponding human motion rendered as mesh, followed by the text description for the human motion and the behavior labels. have … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Statics of MMHU. The average duration of motion sequences is 3s. The most common behavior is crossing the street, while the rarest behavior is using a wheelchair. Behavior definition, please refer to Sec. 3.4. Trajectory from Motion. The trajectory of the human is repr…
Figure 5
Figure 5. Figure 5: Qualitative comparison of Motion Generation. The baseline model (MotionDiffuse, MD) is not capable to generate proper motions in driving scenes. After fine-tuning on MMHU(second row), the model demonstrates the ability to generate human motions in autonomous driving sc…
Figure 6
Figure 6. Figure 6: Annotation Interface. Box 1. Example of Low-Level Descriptions Selected Key Frames for Reference: Low-Level (Joint-wise) Descriptions: • Elbow: The elbows are generally bent throughout the sequence, with some variation in the degree of bending. • Forearm: The forearm r…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

104 extracted references · 47 canonical work pages

  1. [1]

    Auxiliary tasks benefit 3d skeleton-based human motion prediction

    Chenxin Xu, Robby T Tan, Yuhong Tan, Siheng Chen, Xinchao Wang, and Yanfeng Wang. Auxiliary tasks benefit 3d skeleton-based human motion prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9509–9520, 2023

    2023

  2. [2]

    Tamformer: Multi-modal transformer with learned attention mask for early intent prediction

    Nada Osman, Guglielmo Camporese, and Lamberto Ballan. Tamformer: Multi-modal transformer with learned attention mask for early intent prediction. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  3. [3]

    GTransPDM: A Graph-embedded Transformer with Positional Decoupling for Pedestrian Crossing Intention Prediction

    Chen Xie, Ciyun Lin, Xiaoyu Zheng, Bowen Gong, Dayong Wu, and Antonio M López. Gtranspdm: A graph-embedded transformer with positional decoupling for pedestrian crossing intention prediction. arXiv preprint arXiv:2409.20223, 2024

    work page Pith review arXiv 2024

  4. [4]

    Predicting pedestrian crossing intention with feature fusion and spatio-temporal attention

    Dongfang Yang, Haolin Zhang, Ekim Yurtsever, Keith A Redmill, and Ümit Özgüner. Predicting pedestrian crossing intention with feature fusion and spatio-temporal attention. IEEE Transactions on Intelligent Vehicles, 7(2):221–230, 2022

    2022

  5. [5]

    Incorporating physics principles for precise human motion prediction

    Yufei Zhang, Jeffrey O Kephart, and Qiang Ji. Incorporating physics principles for precise human motion prediction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6164–6174, 2024

    2024

  6. [6]

    Behavioral intention prediction in driving scenes: A survey

    Jianwu Fang, Fan Wang, Jianru Xue, and Tat-Seng Chua. Behavioral intention prediction in driving scenes: A survey. IEEE Transactions on Intelligent Transportation Systems, 2024

    2024

  7. [7]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446–2454, 2020

    2020

  8. [8]

    nuscenes: A multimodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621– 11631, 2020

    2020

Show all 104 references
  1. [9]

    Vision meets robotics: The kitti dataset

    Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231–1237, 2013

    2013

  2. [10]

    Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction

    Amir Rasouli, Iuliia Kotseruba, Toni Kunic, and John K Tsotsos. Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6262–6271, 2019

    2019

  3. [11]

    Euro-pvi: Pedestrian vehicle interactions in dense urban centers

    Apratim Bhattacharyya, Daniel Olmeda Reino, Mario Fritz, and Bernt Schiele. Euro-pvi: Pedestrian vehicle interactions in dense urban centers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6408–6417, 2021

    2021

  4. [12]

    Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior

    Amir Rasouli, Iuliia Kotseruba, and John K Tsotsos. Are they going to cross? a benchmark dataset and baseline for pedestrian crosswalk behavior. In Proceedings of the IEEE international conference on computer vision workshops, pages 206–213, 2017

    2017

  5. [13]

    Recovering accurate 3d human pose in the wild using imus and a moving camera

    Timo V on Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European conference on computer vision (ECCV), pages 601–617, 2018

    2018

  6. [14]

    Pedestrian motion reconstruction: A large-scale benchmark via mixed reality rendering with multiple perspectives and modalities

    Yichen Wang, Yiyi Zhang, Xinhao Hu, Li Niu, Jianfu Zhang, Yasushi Makihara, Yasushi Yagi, Pai Peng, Wenlong Liao, Tao He, et al. Pedestrian motion reconstruction: A large-scale benchmark via mixed reality rendering with multiple perspectives and modalities. In The Thirteenth I...

  7. [15]

    Text to blind motion

    Hee Jae Kim, Kathakoli Sengupta, Masaki Kuribayashi, Hernisa Kacorri, and Eshed Ohn-Bar. Text to blind motion. Advances in Neural Information Processing Systems, 37:16272–16285, 2024

    2024

  8. [16]

    Learning to generate diverse pedestrian move- ments from web videos with noisy labels

    Zhizheng Liu, Joe Lin, Wayne Wu, and Bolei Zhou. Learning to generate diverse pedestrian move- ments from web videos with noisy labels. In The Thirteenth International Conference on Learning Representations, 2024

    2024

  9. [17]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 11

    2023 arXiv

  10. [18]

    Vila: On pre-training for visual language models, 2023

    Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023

    2023

  11. [19]

    Longvila: Scaling long-context visual language models for long videos, 2024

    Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, Ethan He, Hongxu Yin, Pavlo Molchanov, Jan Kautz, Linxi Fan, Yuke Zhu, Yao Lu, and Song Han. Longvila: Scaling long-context visual language models for lon...

    2024

  12. [20]

    Nvila: Efficient frontier visual language models, 2024

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang, Yukang Chen, Cheng-Yu Hsieh, De-An Huang, An-Chieh Cheng, Vishwesh Nath, Jinyi Hu, Sifei Liu, Ranjay Krishna, Daguang Xu, Xiaolon...

    2024

  13. [21]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  14. [22]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  15. [23]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  16. [24]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024

    2024

  17. [25]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  18. [26]

    2d human pose estimation calibration and keypoint visibility classification

    Zhongyu Jiang, Haorui Ji, Cheng-Yen Yang, and Jenq-Neng Hwang. 2d human pose estimation calibration and keypoint visibility classification. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6095–6099. IEEE, 2024

    2024

  19. [27]

    Recurrent human pose estimation

    Vasileios Belagiannis and Andrew Zisserman. Recurrent human pose estimation. In 2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017) , pages 468–475. IEEE, 2017

    2017

  20. [28]

    Rethinking the heatmap regression for bottom-up human pose estimation

    Zhengxiong Luo, Zhicheng Wang, Yan Huang, Liang Wang, Tieniu Tan, and Erjin Zhou. Rethinking the heatmap regression for bottom-up human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13264–13273, 2021

    2021

  21. [29]

    Tokenpose: Learning keypoint tokens for human pose estimation

    Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, and Erjin Zhou. Tokenpose: Learning keypoint tokens for human pose estimation. In Proceedings of the IEEE/CVF International conference on computer vision, pages 11313–11322, 2021

    2021

  22. [30]

    Whole- body human pose estimation in the wild

    Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo. Whole- body human pose estimation in the wild. In European Conference on Computer Vision, pages 196–214. Springer, 2020

    2020

  23. [31]

    Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1–248:16, October 2015

    2015

  24. [32]

    Javier Romero, Dimitrios Tzionas, and Michael J. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), November 2017

    2017

  25. [33]

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

  26. [34]

    Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325–1339, 2013

    2013

  27. [35]

    Motionbank: A large-scale video motion benchmark with disentangled rule-based annotations

    Liang Xu, Shaoyang Hua, Zili Lin, Yifan Liu, Feipeng Ma, Yichao Yan, Xin Jin, Xiaokang Yang, and Wenjun Zeng. Motionbank: A large-scale video motion benchmark with disentangled rule-based annotations. arXiv preprint arXiv:2410.13790, 2024. 12

    arXiv

  28. [36]

    Amass: Archive of motion capture as surface shapes

    Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5442–5451, 2019

    2019

  29. [37]

    Motion- x: A large-scale 3d expressive whole-body human motion dataset

    Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion- x: A large-scale 3d expressive whole-body human motion dataset. Advances in Neural Information Processing Systems, 36:25268–25280, 2023

    2023

  30. [38]

    Dynamic multi-person mesh recovery from uncalibrated multi-view cameras

    Buzhen Huang, Yuan Shu, Tianshu Zhang, and Yangang Wang. Dynamic multi-person mesh recovery from uncalibrated multi-view cameras. In 2021 International Conference on 3D Vision (3DV), pages 710–720. IEEE, 2021

    2021

  31. [39]

    Motion capture from internet videos

    Junting Dong, Qing Shuai, Yuanqing Zhang, Xian Liu, Xiaowei Zhou, and Hujun Bao. Motion capture from internet videos. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 210–227. Springer, 2020

    2020

  32. [40]

    Scene-aware 3d multi-human motion capture from a single camera

    Diogo C Luvizon, Marc Habermann, Vladislav Golyanik, Adam Kortylewski, and Christian Theobalt. Scene-aware 3d multi-human motion capture from a single camera. In Computer Graphics Forum, volume 42, pages 371–383. Wiley Online Library, 2023

    2023

  33. [41]

    D &d: Learning human dynamics from dynamic camera

    Jiefeng Li, Siyuan Bian, Chao Xu, Gang Liu, Gang Yu, and Cewu Lu. D &d: Learning human dynamics from dynamic camera. In European Conference on Computer Vision, pages 479–496. Springer, 2022

    2022

  34. [42]

    Decoupling human and camera motion from videos in the wild

    Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 21222–21232, 2023

    2023

  35. [43]

    Glamr: Global occlusion-aware human mesh recovery with dynamic cameras

    Ye Yuan, Umar Iqbal, Pavlo Molchanov, Kris Kitani, and Jan Kautz. Glamr: Global occlusion-aware human mesh recovery with dynamic cameras. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11038–11049, 2022

    2022

  36. [44]

    Implicit neural representations for variable length human motion generation

    Pablo Cervantes, Yusuke Sekikawa, Ikuro Sato, and Koichi Shinoda. Implicit neural representations for variable length human motion generation. In European Conference on Computer Vision, pages 356–372. Springer, 2022

    2022

  37. [45]

    Human motion diffusion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.14916, 2022

    2022 arXiv

  38. [46]

    Motiondiffuse: Text-driven human motion generation with diffusion model

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE transactions on pattern analysis and machine intelligence, 46(6):4115–4128, 2024

    2024

  39. [47]

    Home action genome: Cooperative compositional action understanding

    Nishant Rai, Haofeng Chen, Jingwei Ji, Rishi Desai, Kazuki Kozuka, Shun Ishizaka, Ehsan Adeli, and Juan Carlos Niebles. Home action genome: Cooperative compositional action understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag...

    2021

  40. [48]

    Babel: Bodies, action and behavior with english labels

    Abhinanda R Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J Black. Babel: Bodies, action and behavior with english labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 722–731, 2021

    2021

  41. [49]

    Mining actionlet ensemble for action recognition with depth cameras

    Jiang Wang, Zicheng Liu, Ying Wu, and Junsong Yuan. Mining actionlet ensemble for action recognition with depth cameras. In 2012 IEEE conference on computer vision and pattern recognition , pages 1290–1297. IEEE, 2012

    2012

  42. [50]

    Action recognition based on a bag of 3d points

    Wanqing Li, Zhengyou Zhang, and Zicheng Liu. Action recognition based on a bag of 3d points. In 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, pages 9–14. IEEE, 2010

    2010

  43. [51]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012

    2012 arXiv

  44. [52]

    Modeling temporal structure of decomposable motion segments for activity classification

    Juan Carlos Niebles, Chih-Wei Chen, and Li Fei-Fei. Modeling temporal structure of decomposable motion segments for activity classification. In European conference on computer vision, pages 392–405. Springer, 2010

    2010

  45. [53]

    The kinetics human action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017. 13

    2017 arXiv

  46. [54]

    Actions in context

    Marcin Marszalek, Ivan Laptev, and Cordelia Schmid. Actions in context. In 2009 IEEE conference on computer vision and pattern recognition, pages 2929–2936. IEEE, 2009

    2009

  47. [55]

    Auxformer: Robust approach to audiovisual emotion recognition

    Lucas Goncalves and Carlos Busso. Auxformer: Robust approach to audiovisual emotion recognition. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7357–7361. IEEE, 2022

    2022

  48. [56]

    Context-based inter- pretable spatio-temporal graph convolutional network for human motion forecasting

    Edgar Medina, Leyong Loh, Namrata Gurung, Kyung Hun Oh, and Niels Heller. Context-based inter- pretable spatio-temporal graph convolutional network for human motion forecasting. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3232–3241, 2024

    2024

  49. [57]

    Gcnext: Towards the unity of graph convo- lutions for human motion prediction

    Xinshun Wang, Qiongjie Cui, Chen Chen, and Mengyuan Liu. Gcnext: Towards the unity of graph convo- lutions for human motion prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5642–5650, 2024

    2024

  50. [58]

    Back to mlp: A simple baseline for human motion prediction

    Wen Guo, Yuming Du, Xi Shen, Vincent Lepetit, Xavier Alameda-Pineda, and Francesc Moreno-Noguer. Back to mlp: A simple baseline for human motion prediction. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 4809–4819, 2023

    2023

  51. [59]

    History repeats itself: Human motion prediction via motion attention

    Wei Mao, Miaomiao Liu, and Mathieu Salzmann. History repeats itself: Human motion prediction via motion attention. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pages 474–489. Springer, 2020

    2020

  52. [60]

    Trep: Transformer-based evidential prediction for pedestrian intention with uncertainty

    Zhengming Zhang, Renran Tian, and Zhengming Ding. Trep: Transformer-based evidential prediction for pedestrian intention with uncertainty. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 3534–3542, 2023

    2023

  53. [61]

    Visual–motion–interaction-guided pedestrian intention prediction framework

    Neha Sharma, Chhavi Dhiman, and S Indu. Visual–motion–interaction-guided pedestrian intention prediction framework. IEEE Sensors Journal, 23(22):27540–27548, 2023

    2023

  54. [62]

    Benchmark for evaluating pedestrian action prediction

    Iuliia Kotseruba, Amir Rasouli, and John K Tsotsos. Benchmark for evaluating pedestrian action prediction. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1258–1268, 2021

    2021

  55. [63]

    Agreeing to cross: How drivers and pedestrians communicate

    Amir Rasouli, Iuliia Kotseruba, and John K Tsotsos. Agreeing to cross: How drivers and pedestrians communicate. In IEEE Intelligent Vehicles Symposium (IV), pages 264–269, 2017

    2017

  56. [64]

    Pedestrian intention prediction based on dynamic fuzzy automata for vehicle driving at nighttime

    Joon-Young Kwak, Byoung Chul Ko, and Jae-Yeal Nam. Pedestrian intention prediction based on dynamic fuzzy automata for vehicle driving at nighttime. Infrared Physics & Technology, 81:41–51, 2017

    2017

  57. [65]

    Pedestrian intention and pose prediction through dynamical models and behaviour classification

    Raúl Quintero, Ignacio Parra, David Fernández Llorca, and MA Sotelo. Pedestrian intention and pose prediction through dynamical models and behaviour classification. In 2015 IEEE 18th International Conference on Intelligent Transportation Systems, pages 83–88. IEEE, 2015

    2015

  58. [66]

    Pedestrian path prediction with recursive bayesian filters: A comparative study

    Nicolas Schneider and Dariu M Gavrila. Pedestrian path prediction with recursive bayesian filters: A comparative study. In german conference on pattern recognition, pages 174–183. Springer, 2013

    2013

  59. [67]

    Dolphins: Multimodal language model for driving

    Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. In European Conference on Computer Vision, pages 403–420. Springer, 2024

    2024

  60. [68]

    Drivelm: Driving with graph visual question answering

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In European Conference on Computer Vision, pages 256–274. Springer, 2024

    2024

  61. [69]

    Tem-adapter: Adapting image-text pretraining for video question answer

    Guangyi Chen, Xiao Liu, Guangrun Wang, Kun Zhang, Philip HS Torr, Xiao-Ping Zhang, and Yansong Tang. Tem-adapter: Adapting image-text pretraining for video question answer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13945–13955, 2023

    2023

  62. [70]

    Lmdrive: Closed-loop end-to-end driving with large language models

    Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15120–15130, 2024

    2024

  63. [71]

    Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning

    Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01533, 2024

    2024 arXiv

  64. [72]

    Rag-driver: Generalisable driving explanations with retrieval-augmented in-context learning in multi- modal large language model

    Jianhao Yuan, Shuyang Sun, Daniel Omeiza, Bo Zhao, Paul Newman, Lars Kunze, and Matthew Gadd. Rag-driver: Generalisable driving explanations with retrieval-augmented in-context learning in multi- modal large language model. arXiv preprint arXiv:2402.10828, 2024. 14

    2024

  65. [73]

    Driving with llms: Fusing object-level vector modality for explainable autonomous driving

    Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vector modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Automa...

    2024

  66. [74]

    Lingoqa: Visual question answering for autonomous driving

    Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Visual question answering for autonomous driving. In European Conference on Computer Vision , pages 2...

    2024

  67. [75]

    Covla: Comprehensive vision-language-action dataset for autonomous driving

    Hidehisa Arai, Keita Miwa, Kento Sasaki, Yu Yamaguchi, Kohei Watanabe, Shunsuke Aoki, and Issei Yamamoto. Covla: Comprehensive vision-language-action dataset for autonomous driving. arXiv preprint arXiv:2408.10845, 2024

    2024

  68. [76]

    Rea- son2drive: Towards interpretable and chain-based reasoning for autonomous driving

    Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Rea- son2drive: Towards interpretable and chain-based reasoning for autonomous driving. In European Conference on Computer Vision, pages 292–308. Springer, 2024

    2024

  69. [77]

    Nuscenes-mqa: Integrated evaluation of captions and qa for autonomous driving datasets using markup annotations

    Yuichi Inoue, Yuki Yada, Kotaro Tanahashi, and Yu Yamaguchi. Nuscenes-mqa: Integrated evaluation of captions and qa for autonomous driving datasets using markup annotations. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 930–938, 2024

    2024

  70. [78]

    Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario

    Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. arXiv preprint arXiv:2305.14836, 2023

    2023 arXiv

  71. [79]

    1 year, 1000 km: The oxford robotcar dataset

    Will Maddern, Geoffrey Pascoe, Chris Linegar, and Paul Newman. 1 year, 1000 km: The oxford robotcar dataset. The International Journal of Robotics Research, 36(1):3–15, 2017

    2017

  72. [80]

    Dada: Driver attention prediction in driving accident scenarios

    Jianwu Fang, Dingxin Yan, Jiahuan Qiao, Jianru Xue, and Hongkai Yu. Dada: Driver attention prediction in driving accident scenarios. IEEE transactions on intelligent transportation systems, 23(6):4959–4971, 2021

    2021

  73. [81]

    Snow removal in video: A new dataset and a novel method

    Haoyu Chen, Jingjing Ren, Jinjin Gu, Hongtao Wu, Xuequan Lu, Haoming Cai, and Lei Zhu. Snow removal in video: A new dataset and a novel method. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023

    2023

  74. [82]

    Semantic foggy scene understanding with synthetic data

    Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Semantic foggy scene understanding with synthetic data. International Journal of Computer Vision, 126:973–992, 2018

    2018

  75. [83]

    Wham: Reconstructing world-grounded humans with accurate 3d motion

    Soyong Shin, Juyong Kim, Eni Halilaj, and Michael J Black. Wham: Reconstructing world-grounded humans with accurate 3d motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2070–2080, 2024

    2024

  76. [84]

    Recov- ering accurate 3d human pose in the wild using imus and a moving camera

    Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recov- ering accurate 3d human pose in the wild using imus and a moving camera. In European Conference on Computer Vision (ECCV), sep 2018

    2018

  77. [85]

    Drama: Joint risk localization and captioning in driving

    Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. Drama: Joint risk localization and captioning in driving. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1043–1052, 2023

    2023

  78. [86]

    Posescript: 3d human poses from natural language

    Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez. Posescript: 3d human poses from natural language. In European Conference on Computer Vision, pages 346–362. Springer, 2022

    2022

  79. [87]

    Motiongpt: Human motion as a foreign language

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36:20067–20079, 2023

    2023

  80. [88]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017

    2017

  81. [89]

    Phi-4-mini technical report: Com- pact yet powerful multimodal language models via mixture-of-loras

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Com- pact yet powerful multimodal language models via mixture-of-loras. arXiv preprint...

    2025 arXiv

  82. [90]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  83. [91]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  84. [92]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  85. [93]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  86. [94]

    Mantis: Interleaved multi-image instruction tuning

    Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483, 2024

    2024 arXiv

  87. [95]

    Building and better understanding vision-language models: insights and future directions., 2024

    Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions., 2024

    2024

  88. [96]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073, 2024

    arXiv

  89. [97]

    Gemma 3 technical report

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  90. [98]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302, 2024

    2024 arXiv

  91. [99]

    Kimi-vl technical report

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025

    2025 arXiv

  92. [100]

    Joint attention in autonomous driving (jaad)

    Iuliia Kotseruba, Amir Rasouli, and John K Tsotsos. Joint attention in autonomous driving (jaad). arXiv preprint arXiv:1609.04741, 2016

    2016 arXiv

  93. [101]

    Open source software available from https://github.com/yt-dlp/yt-dlp

    YT-DLP. Open source software available from https://github.com/yt-dlp/yt-dlp

  94. [102]

    Label Studio: Data labeling software, 2020-2025

    Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. Label Studio: Data labeling software, 2020-2025. Open source software available from https://github.com/HumanSignal/label- studio

    2020

  95. [103]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  96. [104]

    drive” or “driving

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System ...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.