Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

Being-M0.5: A Real-Time Controllable Vision-Language-Motion Model

T0 review · 5 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Being-M0.5 is a 7-billion-parameter vision-language-motion model that generates controllable 3D human motion in real time from text, video, or pose inputs, trained on a new 100-million-instruction motion dataset.

desk verdict Genuinely new part-aware tokenizer with solid ablations and honest reporting; the controllability claims rest on in-distribution benchmarks that need outside validation before they land. read the letter →

arxiv 2508.07863 v1 pith:6CAEW4H5 submitted 2025-08-11 cs.CV cs.LG

Bin Cao , Sipeng Zheng , Ye Wang , Lujie Xia , Qianshan Wei , Qin Jin , Jing Liu , Zongqing Lu This is my paper
classification cs.CVcs.LG
keywords humanmotiongenerationvision-language-motionmodelreal-timeinferencepart-levelcontrolresidualvectorquantizationlarge-scaledatasettext-to-motioninstructionfollowing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Being-M0.5 sets out to show that the bottleneck in human motion generation is controllability, not just quality: a practical model must follow arbitrary instructions, start from any pose, generate long sequences, handle unseen actions, and move individual limbs on command. The paper claims that all five can be solved at once with a 7-billion-parameter model trained on HuMo100M, a dataset of over five million web-extracted motion sequences and 100 million instruction instances with part-level annotations, long concatenated motions, and text-aligned video clips. On standard benchmarks it reports a HumanML3D FID of 0.056 and top-1 R-precision of 0.535, while running at 20–28 frames per second with 4-bit quantization. The enabling mechanism is part-aware residual quantization, which turns motion into per-limb discrete codes that can be decoded frame by frame. If the claims hold, motion generation moves from offline batch synthesis to interactive use in games, film, and robotics.

What carries the argument

Part-aware residual quantization (PRQ) is the load-bearing mechanism. Standard residual quantization iteratively refines a whole-body latent vector through stacked codebooks; PRQ instead splits every pose into five anatomically meaningful parts before quantization, applies residual refinement independently to each part, and shares a single codebook and shared joints across parts. This does two jobs: it makes limb identity legible to the language model (a part token can be conditioned on a phrase like 'raise your left arm'), and it enables frame-by-frame decoding, because each frame's part codes are produced progressively rather than waiting for all quantization layers. A second mechanism is

What would settle it

Take a random sample of the auto-generated part-level captions from the HuMo100M pipeline and have independent raters judge whether each caption matches the limb motion in the paired video; a high mismatch rate would undercut the I2PM gains and the 76% left-right exchange score, since those numbers depend on the same annotation machinery.

Watch

Extended reading notes

Core claim

The paper's central claim is that fine-grained part-level control and real-time speed are compatible in one vision-language-motion model, and that both come from the tokenizer rather than from a bigger backbone. Being-M0.5 decomposes each body pose into five parts — left arm, right arm, left leg, right leg, torso — with shared joints across parts, and quantizes each part with four layers of residual vector quantization against a shared codebook of 1,024 entries. This part-aware residual quantization (PRQ) expands the number of representable configurations multiplicatively without extra storage, and its frame-by-frame decoding lets the language model stream motion codes instead of waiting for

Load-bearing premise

The part-control results rest on the accuracy of automatically generated limb-level text descriptions; if those captions often disagree with which limb actually moves, the reported gains would come from mapping templates to the training distribution rather than from genuine kinematic control.

Editorial extensions

If this is right

  • If the reported results hold, a single 7B model can replace separate text-to-motion, pose-conditioned prediction, long-motion, and part-control systems with one real-time system.
  • A 100-million-instruction motion dataset of this size would give the community a common training ground for scaling experiments in motion generation, analogous to the role of large image-text datasets.
  • Part-level discrete codes with shared joints could make interactive motion editing possible, since individual limbs can be re-decoded without regenerating the whole body.
  • The frame-by-frame decoding result suggests that residual-quantization tokenizers, not just diffusion or autoregressive whole-body models, can meet real-time latency budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The auto-annotation pipeline is the part most likely to be stress-tested: until a human validation rate is reported for the part-level captions, the magnitude of the I2PM gains should be treated with caution.
  • The five-way part decomposition is a design choice, not a natural kind; a finer or activity-dependent decomposition (e.g., splitting hands, or grouping by functional synergies) is a testable extension that could change the control granularity.
  • If HuMo100M is released as claimed, the dataset itself may end up mattering more than the model: it would enable third parties to test whether the controllability gains come from scale, from task diversity, or from the PRQ architecture, by controlling for each factor.
  • The left-right exchange benchmark (76%) is a promising cheap probe; extending it to a larger, more varied prompt set with intermediate part annotations would give a more reliable measure of part-level control.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper presents Being-M0.5, a 7B vision-language-motion model trained on a newly collected HuMo100M dataset (5M motion sequences, 100M instructional instances). The central contributions are a part-aware residual quantization (PRQ) tokenizer that decomposes motion into five body-part code streams with shared joints, a three-stage curriculum including multi-task instruction tuning, and a HuMo263 motion feature that avoids inverse kinematics. The authors report state-of-the-art or competitive results on HumanML3D/KIT-ML T2M, new in-house benchmarks for instruct-to-motion, part-motion, long-motion, and unseen-motion, reconstruction, prediction, and captioning, and real-time 20–28 FPS inference on the 7B model. Extensive ablations of quantization layers, shared joints, part labels, visual stage, and multi-task composition are also provided.

Significance. If the empirical claims are validated, this is a substantial advance: a million-scale dataset, a compact part-level motion tokenizer, and a real-time 7B model are all useful community assets. The paper's strengths include transparent limitations (e.g., §5.4.4 concedes the FID deficit vs MoMask), extensive ablations (Tables 6–8), and efficiency analysis across GPUs. The main risk is that the evaluation of controllability relies on benchmarks produced by the same automatic annotation pipeline used for training, so the headline 'controllable' and 'unseen generalization' claims require additional outside-distribution human-validated evidence.

major comments (5)
  1. [§5.1.1, §4.2.1, Table 6] The I2PM benchmark is created by sampling 200K instances from HuMo100M, and the part-level labels in HuMo100M are generated with the same Gemini-1.5-Pro prompts and PoseScript rule templates used for training. Thus the central I2PM comparisons in Table 6 (PRQ4 with part labels FID 1.831 vs 4.281 without) do not yet establish kinematic control; they may reflect the model learning template statistics of the annotation pipeline. The paper reports no human validation rate for the generated part descriptions. To support the controllability claim, the authors should provide an outside-distribution part-level test set with human-validated labels and report per-part success metrics. The 50-sentence left-right exchange test in §5.3.3 is too small and too close to the training template style to rule out interpolation.
  2. [§5.1.1, Table 3] The I2U 'unseen' benchmark is curated by sampling 200K 'novel' sequences from HuMo100M that are excluded from training. Because HuMo100M is also the training distribution, this is a random split rather than an out-of-distribution generalization test. The large FID gap in Table 3 (HuMo100M 8.65 vs MotionX 43.28) may simply reflect distribution overlap between training and test. The authors should define how 'unseen' was measured (e.g., semantic/text-embedding clustering) and evaluate on a genuinely held-out external dataset.
  3. [Abstract, §5.4.4, Table 1] The paper claims 'state-of-the-art performance across multiple motion generation benchmarks,' but Table 1 shows Being-M0.5-PRQ4 FID 0.056 vs MoMask-RQ6 0.045 on HumanML3D, and §5.4.4 openly concedes that PRQ exhibits higher FID scores on HumanML3D. The SOTA claim is metric-dependent; please qualify it to the metrics where it holds (R@1, R@3, MMDist) or provide an aggregate justification.
  4. [§A.1, §5.1.1, Tables 1 and 4] HuMo100M integrates existing motion datasets, including Motion-X, 3DPW, PROX, and BEHAVE, and the evaluation uses HumanML3D, KIT-ML, AMASS, and 3DPW. The paper does not disclose whether the test sets of these external benchmarks were excluded from HuMo100M during training. If any of them are included, the benchmark numbers in Tables 1, 4, 5 and Figure 4 could be contaminated. Please state explicitly how each evaluation set was kept out of HuMo100M.
  5. [§5.2, Tables 2 and 6] The evaluation protocol for the new benchmarks is under-specified. The text says 'all baseline methods utilize identical motion data within each benchmark' but does not state whether MotionGPT/MoMask baselines were fine-tuned on HuMo-I2M/HuMo-I2PM or only evaluated zero-shot. If baselines were not trained on the same instruction templates, the comparison is not apples-to-apples. Specify the training/evaluation protocol for baselines on each new benchmark.
minor comments (6)
  1. [§5.2, Table 2 title] The title says 'Image-to-Motion (I2M)' but the task is Instruct-to-Motion; please correct the terminology.
  2. [§5.2, Figure 4 caption] The caption and text use 'HuMo-MP' while §5.1.1 defines 'HuMo-MPI'; standardize the naming.
  3. [Appendix B] The body-part bullet list uses 'Left Hand'/'Right Hand' while the main text uses 'left arm'/'right arm'; align the terminology.
  4. [§4.1, §1] Typos: 'sequential stepts' and 'ecent efforts' should read 'sequential steps' and 'recent efforts'.
  5. [Eq. (4), Appendix B] The reconstruction loss uses \tilde{m}_j and the aggregation of shared joints is only described in Appendix B. Define these quantities at first use in Section 3.2.2.
  6. [§5.3.2] The FPS figure reports LLM token-generation throughput via llama.cpp but not end-to-end latency including the motion encoder/decoder and tokenizer post-processing. A full pipeline latency breakdown would strengthen the real-time claim.

Circularity Check

2 steps flagged · score 4.0 of 10

Part-level and 'unseen' controllability benchmarks are sampled from the same HuMo100M pool and the same Gemini-1.5-Pro/PoseScript annotation pipeline used to generate training labels, so the headline controllability gains partly reduce to in-distribution template matching; the PRQ architecture itself is otherwise self-contained.

  1. fitted input called prediction [Section 4.2.1 and Section 5.1.1/5.2 (HuMo-I2PM and HuMo-I2U construction)]
    "These annotations are generated using Gemini-1.5-Pro [44] with carefully engineered prompts and further enhanced through PoseScript integration [45]. ... To address this limitation, we leverage Gemini-1.5 Pro to collect motion sequences with specific part-level instructions (e.g., 'raise your left arm') and construct the HuMo-I2PM benchmark."

    The part-level training labels and the I2PM/I2U test instructions come from the same fitted annotator (Gemini-1.5-Pro prompts plus PoseScript rule templates) applied to the same HuMo100M motion pool. The I2U 'unseen' set is only excluded from the training split, not from the annotation distribution. Therefore, the model's I2PM/I2U gains (Figure 4, Tables 2, 3, 6) are essentially a within-distribution inversion of the annotation templates it was trained on; the benchmark is not an independent test of kinematic controllability. The conclusion 'part-level control' is thus statistically forced by the shared construction pipeline rather than by an external human-validated set.

  2. other [Section 5.1.1 (Instruct-to-Unseen) and Table 3]
    "To assess generalization capabilities, we curate 200,000 novel motion sequences to build HuMo-Unseen from HuMo100M that are completely absent from training data, providing rigorous evaluation of the model's ability to generate previously unseen motion patterns."

    Calling a held-out split of HuMo100M 'unseen' is a renaming of standard cross-validation. Because the testbed is sampled from the same dataset and labeled by the same automatic pipeline as the training set, the 'unseen' generalization in Table 3 (FID 65.04 to 8.65 when training on HuMo100M) is not evidence of handling out-of-distribution scenarios; it is the expected result of training and testing on the same video-motion-annotation distribution. The claim of 'unseen motion control' therefore reduces to in-distribution interpolation.

full rationale

Most of Being-M0.5's technical derivation is self-contained: PRQ's loss (Eq. 4) is a standard reconstruction/embedding objective with part decomposition, the real-time analysis is based on measured FPS, and HumanML3D/KIT/AMASS comparisons plus the ParCo left-right exchange test provide external checks. The paper's central vulnerability is that the new controllability benchmarks (I2M, I2PM, I2U, I2LM) are constructed from the same HuMo100M data and the same Gemini/PoseScript annotation templates used to produce training instances. Thus the 'state-of-the-art controllable VLMM' claim rests substantially on evaluating the model against its own data-generation pipeline. The two steps above are the concrete instances of this circular evaluation. Self-citation [9] for the WHAM+PHC pipeline is not load-bearing because WHAM and PHC are external, and no uniqueness theorem is invoked. Therefore the circularity score is moderate (4), not higher.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on two kinds of upstream contributions the reader did not pay for: (1) off-the-shelf estimators (WHAM, PHC, YOLO, Gemini, PoseScript) assumed accurate enough to create 5M reliable motion-text pairs; and (2) modeling choices (five body parts, shared joints, 4 quantization layers, 5:5:3:2:1 task ratio, 224x224/64-frame visual input) selected on the paper's own benchmarks. No new physical entities are postulated. The headline count of 100M instructional instances follows from combinatorial template expansion over motion sequences, so it carries a diversity caveat.

free parameters (5)
  • PRQ quantization depth (PRQ4) = 4 layers
    Chosen as the quality-efficiency tradeoff in Sections 5.3.1 and 5.4.4; deeper PRQ6 improves reconstruction FID but degrades generation on the model's own HuMo-I2PM benchmark.
  • Multi-task data ratio T2M:I2M:MPI:I2PM:I2LM = 5:5:3:2:1
    Hand-set configuration in Section 5.4.3 (Table 8); ablations show sensitivity to task composition, but the ratio itself is not derived from any principle.
  • PRQ codebook size and embedding dimension = 1024 entries, 512 dims
    Given in Section 5.1.3 without a sweep; the beta weight for the embedding loss in Eq. 4 is never given a numeric value.
  • Visual input resolution and frame count = 224x224, 64 frames
    Selected in Section 3.3 for the real-time tradeoff; the paper reports only marginal quality gains at higher resolution.
  • Data filtering thresholds (keypoint confidence, minimum sequence length) = not specified
    Section 4.1 describes occlusion and length filtering but gives no numeric thresholds, so the composition of HuMo100M is not reproducible from the text.
assumptions (5)
  • domain assumption WHAM and PHC produce ground-truth-quality SMPL motion from web video
    The whole dataset rests on these off-the-shelf estimators being accurate enough that 5M web-video sequences can supervise a generation model; the paper relies on same-author prior work [9] for this quality claim (Section 4.1).
  • domain assumption Gemini-1.5-Pro and PoseScript part-level descriptions are semantically correct and aligned with the estimated motion
    Part-level supervision (I2PM) and the I2PM benchmark both rely on these auto-generated labels (Section 4.2.1); no human validation rate is reported.
  • ad hoc to paper Five-part body decomposition with shared spine/pelvis joints is an adequate grouping for control
    The choice of five parts and the shared-joint structure in Section 3.2.2 and Appendix B is specific to this paper; its validity is only shown on the paper's own I2PM benchmark.
  • domain assumption HuMo263 SMPL-based features are a sufficient motion representation for generation
    The new feature format (Section 3.3) drops IK-based computation in favor of SMPL outputs; the paper argues this is better, but the Table 4 baselines may use a different feature format, leaving the comparison unisolated.
  • standard math Next-token prediction on discrete motion codes is a suitable training objective
    Standard autoregressive objective in Eq. 1, consistent with prior motion LLMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Being-M0.5: A Real-Time Controllable Vision-Language-Motion Model." pith.science (2026). https://pith.science/paper/6CAEW4H5

@misc{pith2026250807863,
  author       = {Pith},
  title        = {Pith review of: Being-M0.5: A Real-Time Controllable Vision-Language-Motion Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6CAEW4H5}},
  note         = {Machine review of arXiv:2508.07863}
}
read the original abstract

Human motion generation has emerged as a critical technology with transformative potential for real-world applications. However, existing vision-language-motion models (VLMMs) face significant limitations that hinder their practical deployment. We identify controllability as a main bottleneck, manifesting in five key aspects: inadequate response to diverse human commands, limited pose initialization capabilities, poor performance on long-term sequences, insufficient handling of unseen scenarios, and lack of fine-grained control over individual body parts. To overcome these limitations, we present Being-M0.5, the first real-time, controllable VLMM that achieves state-of-the-art performance across multiple motion generation tasks. Our approach is built upon HuMo100M, the largest and most comprehensive human motion dataset to date, comprising over 5 million self-collected motion sequences, 100 million multi-task instructional instances, and detailed part-level annotations that address a critical gap in existing datasets. We introduce a novel part-aware residual quantization technique for motion tokenization that enables precise, granular control over individual body parts during generation. Extensive experimental validation demonstrates Being-M0.5's superior performance across diverse motion benchmarks, while comprehensive efficiency analysis confirms its real-time capabilities. Our contributions include design insights and detailed computational analysis to guide future development of practical motion generators. We believe that HuMo100M and Being-M0.5 represent significant advances that will accelerate the adoption of motion generation technologies in real-world applications. The project page is available at https://beingbeyond.github.io/Being-M0.5.

Figures

Figures reproduced from arXiv: 2508.07863 by the authors.

Figure 1
Figure 1. Leveraging our million-scale dataset HuMo100M, we present Being-M0.5, the first real-time, control￾lable vision-language-motion model (VLMM) that achieves both high performance and practical efficiency. Being-M0.5 enables comprehensive controllability through five key capabilities: diverse natural language instruction following, flexible pose initialization, long-term motion sequence generation, handling of unseen m… view at source ↗
Figure 2
Figure 2. Model Illustration. Being-M0.5 supports multi-modal inputs/outputs, built on a 7B LLM backbone. It employs SigLIP+2MLP for visual encoding and projection with a slow-fast strategy, alongside part-aware residual quantization for motion tokenization. detailed analyses of controllable motion generation strategies (Section 3.2) and real-time design considerations (Section 3.3). 3.1 Overview of VLMM Our VLMM is built on … view at source ↗
Figure 3
Figure 3. Illustration of data pipeline. We introduce HuMo100M, the largest multimodal dataset to date, featuring over 5 million motion sequences, paired visual clips, threefold more hierarchical and part-level textual descriptions, and 100 million multi-task instruction instances. 4 The HuMo100M Dataset While scaling training data has proven fundamental to the success of large multimodal models, this strategy faces significa… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Comparison with previous SoTAs across nine [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Being-M0.5 inference speed across different GPUs. We accelerate motion generation via modern LLM inference framework llama.cpp [57]. Our model achieves real-time inference speed using the 7B-parameter LLaMA backbone. 5.3.3 Part-level Controllability Being-M0.5 addresse…
Figure 6
Figure 6. Figure 6: Qualitative examples generated by Being-M0.5 for Instruct-to-PartMotion (I2PM) and Instruct-to￾LongMotion (I2LM). The results demonstrate the ability of our model to generate motion sequences that accurately align with both part-level and long-term instructions. Due to…
Figure 7
Figure 7. Figure 7: Distribution of motion sequence lengths across different subsets in [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]
Figure 8
Figure 8. Figure 8: Distribution of motion instance counts across different subsets in [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]
Figure 9
Figure 9. Figure 9: Word cloud visualization of rule-based motion descriptions in [PITH_FULL_IMAGE:figures/full_fig_p023_9.png]
Figure 10
Figure 10. Figure 10: Word cloud visualization of body-level motion descriptions in [PITH_FULL_IMAGE:figures/full_fig_p024_10.png]
Figure 11
Figure 11. Figure 11: Word cloud visualization of part-level motion descriptions in [PITH_FULL_IMAGE:figures/full_fig_p024_11.png]
Figure 12
Figure 12. Figure 12: Visualization results generated by Being-M0.5 from diverse natural language instructions, demon￾strating the model’s versatility in interpreting and executing varied motion commands. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Visualization results generated by Being-M0.5 given the long-term instruction showcasing the model’s capability for coherent long-duration motion synthesis. A person punches with their left hand before he does a clockwise spin. A person waves with right hand. A person…
Figure 14
Figure 14. Figure 14: Visualization results generated by Being-M0.5 givne the part-level instruction, illustrating fine￾grained controllability over individual anatomical regions [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Prompt template utilized for generating part-level motion descriptions from video content using [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CoMoVi: Co-Generation of 3D Human Motions and Realistic Videos

    cs.CV 2026-01 unverdicted novelty 7.0 of 10

    CoMoVi co-generates 3D human motions and 2D videos synchronously in a single diffusion denoising loop using 3D-to-2D projection and dual-branch diffusion with 3D-2D cross attentions.

  2. IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generation

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Interleaving motion generation with text-motion assessment and refinement improves alignment between generated human motion and goal text.

Reference graph

Works this paper leans on

65 extracted references · 33 canonical work pages · cited by 2 Pith papers

  1. [1]

    Momask: Generative masked modeling of 3d human motions

    Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900–1910, 2024

    1900

  2. [2]

    Motiongpt: Human motion as a foreign language

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36:20067–20079, 2023

    2023

  3. [3]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152–5161, 2022

    2022

  4. [4]

    Motion-x: A large-scale 3d expressive whole-body human motion dataset.Advancesin Neural Information Processing Systems, 36, 2024

    Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large-scale 3d expressive whole-body human motion dataset.Advancesin Neural Information Processing Systems, 36, 2024

    work page 2024

  5. [5]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  6. [6]

    Improved baselines with visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023

    2023

  7. [7]

    A large-scale rgb-d database for arbitrary-view human action recognition

    Yanli Ji, Feixiang Xu, Yang Yang, Fumin Shen, Heng Tao Shen, and Wei-Shi Zheng. A large-scale rgb-d database for arbitrary-view human action recognition. In Proceedings of the 26th ACM international Conference on Multimedia, pages 1510–1518, 2018

    work page 2018

  8. [8]

    M$^3$GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generation

    Mingshuang Luo, Ruibing Hou, Hong Chang, Zimo Liu, Yaowei Wang, and Shiguang Shan. M3gpt: An advanced multimodal, multitask framework for motion comprehension and generation.arXiv preprint arXiv:2405.16273, 2024

    work page Pith review arXiv 2024

Show all 65 references
  1. [9]

    Scaling large motion models with million-level human motions

    Ye Wang, Sipeng Zheng, Bin Cao, Qianshan Wei, Weishuai Zeng, Qin Jin, and Zongqing Lu. Scaling large motion models with million-level human motions. InInternational Conference on Machine Learning, 2025

    2025

  2. [10]

    Autoregressive image generation using residual quantization

    Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523–11532, 2022

    2022

  3. [11]

    Temos: Generating diverse human motions from textual descriptions

    Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision, pages 480–497. Springer, 2022

    2022

  4. [12]

    Language2pose: Natural language grounded pose forecasting

    Chaitanya Ahuja and Louis-Philippe Morency. Language2pose: Natural language grounded pose forecasting. In 2019 International Conference on 3D Vision (3DV), pages 719–728. IEEE, 2019

    2019

  5. [13]

    Motiongpt: Finetuned llms are general-purpose motion generators

    Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned llms are general-purpose motion generators. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7368–7376, 2024

    2024

  6. [14]

    Motiongpt-2: A general-purpose motion-language model for motion generation and understanding

    Yuan Wang, Di Huang, Yaqi Zhang, Wanli Ouyang, Jile Jiao, Xuetao Feng, Yan Zhou, Pengfei Wan, Shixiang Tang, and Dan Xu. Motiongpt-2: A general-purpose motion-language model for motion generation and understanding. arXiv preprint arXiv:2410.21747, 2024

    2024 arXiv

  7. [15]

    Recurrent network models for human dynamics

    Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE international conference on computer vision, pages 4346–4354, 2015

    2015

  8. [16]

    A neural temporal model for human motion prediction

    Anand Gopalakrishnan, Ankur Mali, Dan Kifer, Lee Giles, and Alexander G Ororbia. A neural temporal model for human motion prediction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12116–12125, 2019

    2019

  9. [17]

    A stochastic conditioning scheme for diverse human motion prediction

    Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Lars Petersson, and Stephen Gould. A stochastic conditioning scheme for diverse human motion prediction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5223–5232, 2020

    2020

  10. [18]

    Learning diverse stochastic human-action generators by learning smooth latent transitions

    Zhenyi Wang, Ping Yu, Yang Zhao, Ruiyi Zhang, Yufan Zhou, Junsong Yuan, and Changyou Chen. Learning diverse stochastic human-action generators by learning smooth latent transitions. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 12281–12288, 2020. 17

    2020

  11. [19]

    Motionchain: Conversa- tional motion controllers via multimodal prompts.arXiv preprint arXiv:2404.01700, 2024

    Biao Jiang, Xin Chen, Chi Zhang, Fukun Yin, Zhuoyuan Li, Gang Yu, and Jiayuan Fan. Motionchain: Conversa- tional motion controllers via multimodal prompts.arXiv preprint arXiv:2404.01700, 2024

    2024 arXiv

  12. [20]

    Motionllm: Understanding human behaviors from human motions and videos.arXiv preprint arXiv:2405.20340, 2024

    Ling-Hao Chen, Shunlin Lu, Ailing Zeng, Hao Zhang, Benyou Wang, Ruimao Zhang, and Lei Zhang. Motionllm: Understanding human behaviors from human motions and videos.arXiv preprint arXiv:2405.20340, 2024

    2024 arXiv

  13. [21]

    Large motion model for unified multi-modal motion generation

    Mingyuan Zhang, Daisheng Jin, Chenyang Gu, Fangzhou Hong, Zhongang Cai, Jingfang Huang, Chongzhi Zhang, Xinying Guo, Lei Yang, Ying He, et al. Large motion model for unified multi-modal motion generation. In European Conference on Computer Vision, pages 397–421. Springer, 2024

    2024

  14. [22]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716...

    2022

  15. [23]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  16. [24]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  17. [25]

    Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    2023 arXiv

  18. [26]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024

    2024

  19. [27]

    Neural discrete representation learning.Advancesin neural information processing systems, 30, 2017

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning.Advancesin neural information processing systems, 30, 2017

    2017

  20. [28]

    Locally hierarchical auto-regressive modeling for image generation.Advances in Neural Information Processing Systems, 35:16360–16372, 2022

    Tackgeun You, Saehoon Kim, Chiheon Kim, Doyup Lee, and Bohyung Han. Locally hierarchical auto-regressive modeling for image generation.Advances in Neural Information Processing Systems, 35:16360–16372, 2022

    2022

  21. [29]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023

    2023 arXiv

  22. [30]

    Finite scalar quantization: Vq-vae made simple

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309.15505, 2023

    2023 arXiv

  23. [31]

    The language of motion: Unifying verbal and non-verbal language of 3d human motion.arXiv preprint arXiv:2412.10523, 2024

    Changan Chen, Juze Zhang, Shrinidhi K Lakshmikanth, Yusu Fang, Ruizhi Shao, Gordon Wetzstein, Li Fei-Fei, and Ehsan Adeli. The language of motion: Unifying verbal and non-verbal language of 3d human motion.arXiv preprint arXiv:2412.10523, 2024

    2024 arXiv

  24. [32]

    Human- tomato: Text-aligned whole-body motion generation.arXiv preprint arXiv:2310.12978, 2023

    Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung-Yeung Shum. Human- tomato: Text-aligned whole-body motion generation.arXiv preprint arXiv:2310.12978, 2023

    2023 arXiv

  25. [33]

    Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data.arXiv preprint arXiv:2410.02713, 2024

    2024 arXiv

  26. [34]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vis...

    2024

  27. [35]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  28. [36]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023

    2023

  29. [37]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  30. [38]

    Smpl: A skinned multi-person linear model

    Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. InSeminal Graphics Papers: Pushing the Boundaries, Volume2, pages 851–866, 2023. 18

    2023

  31. [39]

    Scamo: Exploring the scaling law in autoregressive motion generation model.arXiv preprint arXiv:2412.14559, 2024

    Shunlin Lu, Jingbo Wang, Zeyu Lu, Ling-Hao Chen, Wenxun Dai, Junting Dong, Zhiyang Dou, Bo Dai, and Ruimao Zhang. Scamo: Exploring the scaling law in autoregressive motion generation model.arXiv preprint arXiv:2412.14559, 2024

    2024 arXiv

  32. [40]

    You only look once: Unified, real-time object detection

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779–788, 2016

    2016

  33. [41]

    Wham: Reconstructing world-grounded humans with accurate 3d motion

    Soyong Shin, Juyong Kim, Eni Halilaj, and Michael J Black. Wham: Reconstructing world-grounded humans with accurate 3d motion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2070–2080, 2024

  34. [42]

    Perpetual humanoid control for real-time simulated avatars

    Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10895–10904, 2023

    2023

  35. [43]

    Motionbank: A large-scale video motion benchmark with disentangled rule-based annotations.arXiv preprint arXiv:2410.13790, 2024

    Liang Xu, Shaoyang Hua, Zili Lin, Yifan Liu, Feipeng Ma, Yichao Yan, Xin Jin, Xiaokang Yang, and Wenjun Zeng. Motionbank: A large-scale video motion benchmark with disentangled rule-based annotations.arXiv preprint arXiv:2410.13790, 2024

    2024 arXiv

  36. [44]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  37. [45]

    Posescript: Linking 3d human poses and natural language.IEEE transactions on pattern analysis and machine intelligence, 2024

    Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez. Posescript: Linking 3d human poses and natural language.IEEE transactions on pattern analysis and machine intelligence, 2024

    2024

  38. [46]

    Unipose: A unified multimodal framework for human pose comprehension, generation and editing.arXiv preprint arXiv:2411.16781, 2024

    Yiheng Li, Ruibing Hou, Hong Chang, Shiguang Shan, and Xilin Chen. Unipose: A unified multimodal framework for human pose comprehension, generation and editing.arXiv preprint arXiv:2411.16781, 2024

    2024 arXiv

  39. [47]

    The kit motion-language dataset.Big data, 4(4):236– 252, 2016

    Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset.Big data, 4(4):236– 252, 2016

    2016

  40. [48]

    Amass: Archive of motion capture as surface shapes

    Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. InProceedings of the IEEE/CVF international conference on computer vision, pages 5442–5451, 2019

    2019

  41. [49]

    Recovering accurate 3d human pose in the wild using imus and a moving camera

    Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. InProceedings of the European conference on computer vision (ECCV), pages 601–617, 2018

    2018

  42. [50]

    Motion-x: A large-scale 3d expressive whole-body human motion dataset.Advancesin Neural Information Processing Systems, 36:25268–25280, 2023

    Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large-scale 3d expressive whole-body human motion dataset.Advancesin Neural Information Processing Systems, 36:25268–25280, 2023

    2023

  43. [51]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  44. [52]

    Executing your commands via motion diffusion in latent space

    Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18000–18010, 2023

    2023

  45. [53]

    Motiondiffuse: Text-driven human motion generation with diffusion model.arXiv preprint arXiv:2208.15001, 2022

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model.arXiv preprint arXiv:2208.15001, 2022

    2022 arXiv

  46. [54]

    Generating human motion from textual descriptions with discrete representations

    Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pag...

    2023

  47. [55]

    Motionllm: Multimodal motion-language learning with large language models.arXiv preprint arXiv:2405.17013, 2024

    Qi Wu, Yubo Zhao, Yifan Wang, Yu-Wing Tai, and Chi-Keung Tang. Motionllm: Multimodal motion-language learning with large language models.arXiv preprint arXiv:2405.17013, 2024. 19

    2024 arXiv

  48. [56]

    Avatargpt: All-in-one framework for motion understanding planning generation and beyond

    Zixiang Zhou, Yu Wan, and Baoyuan Wang. Avatargpt: All-in-one framework for motion understanding planning generation and beyond. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1357–1366, 2024

    2024

  49. [57]

    llamacpp.https://github.com/ggml-org/llama.cpp, 2024

    llamacpp project members. llamacpp.https://github.com/ggml-org/llama.cpp, 2024

    2024

  50. [58]

    Parco: Part- coordinating text-to-motion synthesis

    Qiran Zou, Shangyuan Yuan, Shian Du, Yu Wang, Chang Liu, Yi Xu, Jie Chen, and Xiangyang Ji. Parco: Part- coordinating text-to-motion synthesis. InEuropean Conference on Computer Vision, pages 126–143. Springer, 2024

    2024

  51. [59]

    Fg-t2m++: Llms-augmented fine-grained text driven human motion generation.International Journal of Computer Vision, pages 1–17, 2025

    Yin Wang, Mu Li, Jiapeng Liu, Zhiying Leng, Frederick WB Li, Ziyao Zhang, and Xiaohui Liang. Fg-t2m++: Llms-augmented fine-grained text driven human motion generation.International Journal of Computer Vision, pages 1–17, 2025

    2025

  52. [60]

    Motiondiffuse: Text-driven human motion generation with diffusion model.IEEE transactions on pattern analysis and machine intelligence, 46(6):4115–4128, 2024

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model.IEEE transactions on pattern analysis and machine intelligence, 46(6):4115–4128, 2024

    2024

  53. [61]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings,...

    2014

  54. [62]

    Posetrack: A benchmark for human pose estimation and tracking

    Mykhaylo Andriluka, Umar Iqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. Posetrack: A benchmark for human pose estimation and tracking. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 5167–5176, 2018

    2018

  55. [63]

    Resolving 3d human pose ambiguities with 3d scene constraints

    Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J Black. Resolving 3d human pose ambiguities with 3d scene constraints. InProceedings of the IEEE/CVF international conference on computer vision, pages 2282–2292, 2019

    2019

  56. [64]

    Behave: Dataset and method for tracking human object interactions

    Bharat Lal Bhatnagar, Xianghui Xie, Ilya A Petrov, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll. Behave: Dataset and method for tracking human object interactions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 159...

    2022

  57. [65]

    the left hand is positioned below the right hand

    Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. InIEEE International Conference on Computer Vision, 2021. 20 Appendix In this appendix, we provide comprehensive details and analysis of theHuMo1...

    2021

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.