Pith. sign in

REVIEW 4 major objections 5 minor 117 references

From Motion to Behavior: Hierarchical Modeling of Humanoid Generative Behavior Control

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A hierarchical LLM-plus-physics pipeline generates semantically coherent human behaviors roughly ten times longer than prior text-to-motion methods, and the new GBC-100K dataset supplies the hierarchical annotations to train and evaluate…

desk verdict The GBC-100K dataset is the real contribution; the paper's headline claim of 10× longer generation is unsupported by the experiments as presented. read the letter →

arxiv 2506.00043 v1 pith:USMGFBBN submitted 2025-05-28 cs.RO cs.CV

Jusheng Zhang , Jinzhou Tang , Sidi Liu , Mingyan Li , Sheng Zhang , Jian Wang , Keze Wang This is my paper
classification cs.ROcs.CV
keywords generativebehaviorcontrolhumanmotiongenerationlong-horizonsynthesislargelanguagemodelplanningtaskandphysics-basedcharactermotion-languagedatasethierarchicalscripts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to move human motion generation from isolated, short actions to goal-oriented behavior: given a text instruction such as "a person changes a tire in a garage," the system should produce minutes of continuous, physically plausible movement. Its central proposal, Generative Behavior Control (GBC), treats a behavior as a hierarchy of an abstract description, static pose scripts, and transition motion scripts, with an LLM doing the planning and a physics-enabled controller guaranteeing feasibility. To train and test this, the authors built GBC-100K, roughly 124,000 video-SMPL-text clips with multilevel annotations, and PHYLOMAN, an inference-time pipeline that composes LLM plans, text-to-pose generation, diffusion-based in-betweening, and simulation-based control. If the claims hold, text-to-behavior models would gain a benchmark and a recipe for producing long, semantically meaningful, physically consistent humanoid animation without motion capture.

What carries the argument

The load-bearing object is the BehaviorScript, defined as B={D,P,A}: a high-level description D with five semantic slots ([Subject], [Emotion/State/Style], [Action], [Direction/Goal], [Environment/Background]), a sequence of PoseScripts P={p_0,...,p_n} that are static keyframe postures expressed in a standardized pose language, and MotionScripts A={a_0,...,a_{n-1}} that describe transitions between consecutive poses. This representation lets the framework treat behavior generation as task and motion planning: the LLM outputs the script, a pre-trained PoseScript VAE maps each pose text to an SMPL configuration x_i, a diffusion model (CondMDI) synthesizes motion between configurations, and a physics controller (PHC or HOVER) refines the trajectory under MuJoCo dynamics. The script is what keeps semantics and physics aligned: task-level constraints C_T enforce joint and semantic consistency, while motion-level constraints C_M enforce joint limits, collision avoidance, and dynamic feasibility along each trajectory.

What would settle it

Take a random sample of GBC-100K clips, run the same actions under marker-based motion capture, and compare the TRAM-estimated SMPL to the mocap SMPL; if the average per-joint error is large (or if the Phys-Err of generated PHYLOMAN sequences is measured against mocap rather than estimated ground truth and rises sharply), the central claim that the system is physically consistent is falsified.

Watch

Extended reading notes

Core claim

Stated on the paper's terms: human behavior is a hierarchical script B = {D, P, A} in which a high-level description D decomposes into PoseScripts (static keyframe postures) and MotionScripts (transitions between them), and this script can be executed by a TAMP-style stack. PHYLOMAN uses an LLM to turn a natural-language goal into such a script, a PoseScript-pretrained VAE to map each keyframe text to an SMPL pose, a diffusion motion in-betweening model to fill transitions, and a physics-based controller in simulation to refine the whole trajectory. The paper reports that this pipeline generates sequences of 1024 frames that are roughly 10x longer than what standard text-to-motion methods produce, with physical error reduced by about 91% and success rate increased by about 133% relative to the same pipeline without task-and-motion planning. The authors also show that fine-tuning an LLM on the hierarchical annotations of GBC-100K improves planning quality beyond zero-shot closed-source LLMs.

Load-bearing premise

The framework's training and evaluation ground truth is SMPL pose sequences estimated from internet videos by an automatic model (TRAM) after heuristic filtering, and if those estimates are systematically noisy, the physical-error, fidelity, and retrieval metrics, plus the trained models themselves, inherit that noise.

Editorial extensions

If this is right

  • Text-to-human-behavior systems could move from clips of a few seconds to multi-minute sequences while retaining a single coherent goal, opening applications in digital avatars, animation pre-visualization, and embodied-agent training.
  • GBC-100K provides a large, hierarchical, goal-oriented motion-language benchmark whose average text length is several times longer than HumanML3D's, making it a harder and more realistic testbed for long-horizon generation.
  • Because the framework is stage-wise and inference-time (each component trained separately), its LLM planner, pose generator, in-betweener, and controller can be upgraded independently, so improvements in any module should translate directly to better behavior generation.
  • Fine-tuned open LLMs on hierarchical scripts beat zero-shot closed LLMs in the paper's planning metrics, implying that dataset annotations themselves can teach planning skills that general-purpose LLMs lack by default.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the TRAM-estimated SMPL poses in GBC-100K are accepted as ground truth, the same internet-video pipeline could be reused to generate training data for embodied agents at scale, replacing expensive mocap studios; that extension is not in the paper.
  • The hierarchical script representation suggests a natural swap test: replace the diffusion in-betweener with any newer long-context motion model and keep the rest fixed; if the 10x horizon claim is robust, gains should persist, and if not, the bottleneck is in the script or controller rather than the generator.
  • Because the evaluation metrics are computed with a CLIP-based encoder fine-tuned on the same automatically annotated dataset, an independent reader should treat reported success rates as relative improvements within that distribution; validation on held-out mocap or human ratings from outside the distribution would test the framework's generality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a new task, Generative Behavior Control (GBC), and a corresponding framework PHYLOMAN that couples LLM-based hierarchical behavior planning with physics-based motion control. It also introduces GBC-100K, a large-scale dataset of roughly 123.7K video-SMPL clips annotated with hierarchical PoseScript, MotionScript, and BehaviorScript descriptions. The central claims are that PHYLOMAN can generate behavior sequences about 10 times longer than existing text-to-motion methods while maintaining semantic coherence and physical plausibility, and that GBC-100K provides a more comprehensive and challenging benchmark than existing motion-language datasets. Experiments are reported in three tables: Table 2 evaluates baseline text-to-motion models on three dataset configurations, Table 3 ablates PHYLOMAN components at 1024 frames, and Table 4 compares fine-tuned versus zero-shot LLM planners.

Significance. If the central claims were fully supported, the paper would make a useful contribution: GBC-100K would be a large-scale, hierarchically annotated resource for long-horizon behavior generation, and the PHYLOMAN architecture would provide a concrete recipe for combining LLM planning with physics-based control. The authors promise to release the dataset and source code, and they provide detailed dataset statistics and supplementary implementation details that would help others reproduce the pipeline. The strengths are the scale and structure of the proposed dataset and the conceptually appealing separation of task-level and motion-level planning. However, the current evidence does not substantiate the headline 10x length claim, the semantic metrics are computed with a dataset-specific encoder that risks circularity, and the SMPL ground truth obtained from TRAM is not validated against any independent mocap data. These are load-bearing gaps that affect the credibility of both the framework and the dataset evaluation.

major comments (4)
  1. [§1, §5.2, Table 3] The central claim that PHYLOMAN generates sequences '10× longer' than state-of-the-art methods is not supported by the reported experiments. Table 2, the only direct comparison with existing text-to-motion models, standardizes all outputs to 196 frames and does not include PHYLOMAN; Table 3 evaluates PHYLOMAN at 1024 frames but compares only ablated variants and two baselines (MoMask and T2M-GPT) used as internal motion-generator modules, not as end-to-end methods run at their own native maximum lengths. The ratio between 1024 and 196 is about 5.2, not 10, and no same-length baseline run is reported. To support the claim, the authors should run each external baseline at its longest feasible length, report the actual generated lengths, and compare PHYLOMAN with them at identical lengths; if the baselines cannot produce long sequences, that failure should be quantified and discussed rather than assumed.
  2. [§5 (Implementation Details), Supp. §10.2] The semantic metrics (R-Precision, FID, MM Distance, Diversity, MultiModality) are all computed with an ActionCLIP encoder fine-tuned on GBC-100K, the same dataset on which the evaluated methods are trained. Although the CLIP model and the diffusion model are trained on different splits, an encoder tuned to the dataset's domain can systematically favor methods trained on that same dataset, so the reported semantic-fidelity improvements (e.g., the MultiModality comparisons in Table 2) are at risk of circularity. The authors should report at least one independent evaluation using a fixed, publicly available text-motion encoder (for example, a HumanML3D-pretrained TMR or MotionCLIP) and should discuss any discrepancies; they should also provide results when ActionCLIP is used without fine-tuning.
  3. [§4 (Data Collection), Supp. §10.6] The dataset's SMPL pose sequences are estimated by TRAM from internet videos and filtered with heuristic quality metrics, but no validation against ground-truth motion capture is provided. Because these sequences serve as both training targets and as reference ground truth for FID, R-Precision, and Phys-Err, systematic estimation error (e.g., in foot contact, body proportions, or global trajectory) would propagate into the models and into all physical plausibility metrics. The authors should validate TRAM reconstructions on a mocap subset (for example, AMASS or HumanML3D sequences) by reporting per-joint error and per-component Phys-Err on known motions, and they should quantify how the filtering changes the downstream metrics.
  4. [§5.2, Table 2, Table 3] The evaluation protocol is internally inconsistent. Table 2 is introduced as evaluating six SOTA baselines (MotionCLR, MDM, MotionLCM, T2M-GPT, MoMask, CondMDI) but reports results only for MotionCLR, MDM, and MotionLCM. In Section 5.2 the text states that PHYLOMAN improves success rate 'from 0.3 to 0.7' and reduces physical error 'from 1.224 to 0.105', but Table 3 contains none of these values (the closest are 0.328, 0.179, 0.821 and 1.031, 0.093). The 'without TAMP' comparison is not tied to any identifiable row. These discrepancies make the headline ablation results impossible to verify and should be corrected with a table that matches the text and includes all baselines actually used.
minor comments (5)
  1. [Abstract and §1] The symbols '10*' and '10×' are used interchangeably to denote the claimed length improvement; please use a single consistent notation.
  2. [Supp. §10.4] In the user-study description, the sentence 'we conducted a human evaluation study using, where participants were presented with generated motion sequences' contains a grammatical error; remove 'using'.
  3. [§5.2] The sentence listing the optimal components ('CondMDI serves as the Motion Generator; Chain-of-Thought functions as the LLM Planner; ...') is not consistent with Table 3, whose 'LLM Planner' rows are only 'Heuristic' and 'Discard', with no explicit CoT row; the table should be updated to identify which variant corresponds to the optimal configuration.
  4. [Supp. §9.2, Eq. (3)] The semi-implicit Euler update writes q_{t+Δ} = q_t + Δ·v_{t+Δ}; please clarify whether v_{t+Δ} denotes the velocity after the force evaluation or whether the subscript should be t.
  5. [Table 2] The 'Real' rows show R-Precision around 0.5 rather than 1.0 for ground-truth motions on the same dataset; this suggests limited sensitivity of the retrieval metric on GBC-100K and deserves a brief discussion.

Circularity Check

1 steps flagged · score 5.0 of 10

The framework's derivation is not circular, but the semantic benchmark is partly self-referential: ActionCLIP is fine-tuned on GBC-100K and then used to compute FID/R-Precision on the same dataset, so the semantic quality claims are partially an artifact of a fitted evaluator.

  1. fitted input called prediction [Section 5 (Implementation Details) and Supplementary 10.2 (Evaluation Metrics)]
    "The CLIP-based similarity metric is trained on our dataset to ensure domain-specific evaluation. ... Leveraging ActionCLIP fine-tuned on GBC-100K, we extract feature embeddings for both the generated motions and their textual counterparts. ... FID is calculated by extracting embeddings using ActionCLIP and computing the Fréchet distance between the distributions of real and generated motions. ... R-Precision is computed as the percentage of ground-truth motions among the retrieved sequences."

    ActionCLIP is fine-tuned on roughly 25k clips from GBC-100K, and the same ActionCLIP embeddings are then used to compute MM Dist, FID, and R-Precision for methods listed as 'Trained & Evaluated on GBC-100K'. The real-data distribution in FID and the ground-truth retrieval targets in R-Precision are therefore drawn from the same dataset used to fit the evaluator. The evaluator has been trained to place each GBC-100K text description near its paired motion, so the measured semantic fidelity is in part a memory of the metric model's training pairs rather than an independent test of the generated motion's meaning. This makes the semantic evaluation partially self-referential; Phys-Err and the user-study Success Rate are not affected because they do not use this fitted encoder.

full rationale

No mathematical derivation in the paper is circular by construction: PHYLOMAN composes externally pretrained components (CondMDI, PoseScript VAE, HOVER/PHC controllers), and the physics-based Phys-Err and the human-study Success Rate provide evaluation signals that are independent of the GBC-100K-trained text-motion encoder. The '10x longer' claim is unsupported by a same-length baseline comparison, but that is a correctness/evidence problem rather than a circularity problem. The main circularity concern is the semantic evaluation loop: the ActionCLIP metric is fine-tuned on GBC-100K and then used to compute FID, MM Dist, and R-Precision on the same dataset whose text labels were themselves generated by a VLM. This fitted evaluator is reused as the scoring function, so the semantic quality claims of the dataset and methods trained on it are partially self-referential. The central behavioral results (Phys-Err, SR) retain independent content, so the overall circularity is partial rather than total.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on pre-trained components, the accuracy of TRAM-extracted SMPL, and hand-chosen constraint weights; no new physical entities are introduced.

free parameters (4)
  • Constraint weights w1, w2, wa, wb in C_T = not reported
    Defined in Supp 9.1 for the high-level transition constraint; values are hand-chosen and not listed, so the behavior of C_T cannot be reproduced.
  • Constraint weights w3, w4, w5 in C_M = not reported
    Defined in Supp 9.2 for low-level motion constraints; values not given.
  • Semantic tolerance eps_s, steepness kappa, collision distance dmin, collision steepness kappa_c = not reported
    Parameters of the soft penalty functions in Supp 9; no values or sensitivity analysis are provided.
  • Interpolated frame count C = not reported
    Section 4.2 defines K = N + C(N-1) but does not state C, which sets the temporal density of the synthesized trajectory.
assumptions (4)
  • domain assumption Pre-trained PoseScript VAE, CondMDI diffusion, and PHC/HOVER controllers remain effective when composed in a long-horizon pipeline without joint training.
    Section 4.2 states individual components are trained separately; no experiment verifies compatibility of the composed system beyond the final pipeline results.
  • domain assumption LLM-generated PoseScript and MotionScript sequences are semantically faithful and physically plausible enough to condition the motion generator.
    Section 4.1 relies on the LLM to decompose behavior; no quantitative measure of decomposition quality is reported except the final SR.
  • domain assumption TRAM-extracted SMPL from internet videos is accurate enough to serve as ground truth for training and evaluation.
    Section 4 Data Collection; the filtering uses heuristics but no validation against ground truth is described.
  • domain assumption MuJoCo simulation is a faithful proxy for physical plausibility in the Phys-Err metric.
    Supp 9.2 and 10.2 use MuJoCo to compute physical error; this is standard but assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Motion to Behavior: Hierarchical Modeling of Humanoid Generative Behavior Control." pith.science (2026). https://pith.science/paper/USMGFBBN

@misc{pith2026250600043,
  author       = {Pith},
  title        = {Pith review of: From Motion to Behavior: Hierarchical Modeling of Humanoid Generative Behavior Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/USMGFBBN}},
  note         = {Machine review of arXiv:2506.00043}
}
read the original abstract

Human motion generative modeling or synthesis aims to characterize complicated human motions of daily activities in diverse real-world environments. However, current research predominantly focuses on either low-level, short-period motions or high-level action planning, without taking into account the hierarchical goal-oriented nature of human activities. In this work, we take a step forward from human motion generation to human behavior modeling, which is inspired by cognitive science. We present a unified framework, dubbed Generative Behavior Control (GBC), to model diverse human motions driven by various high-level intentions by aligning motions with hierarchical behavior plans generated by large language models (LLMs). Our insight is that human motions can be jointly controlled by task and motion planning in robotics, but guided by LLMs to achieve improved motion diversity and physical fidelity. Meanwhile, to overcome the limitations of existing benchmarks, i.e., lack of behavioral plans, we propose GBC-100K dataset annotated with a hierarchical granularity of semantic and motion plans driven by target goals. Our experiments demonstrate that GBC can generate more diverse and purposeful high-quality human motions with 10* longer horizons compared with existing methods when trained on GBC-100K, laying a foundation for future research on behavioral modeling of human motions. Our dataset and source code will be made publicly available.

Figures

Figures reproduced from arXiv: 2506.00043 by the authors.

Figure 1
Figure 1. From motion to behavior. (a) Simple periodic motion patterns without complex, behavioral semantic meaning, (b) Complex, semantically meaningful human behaviors, demonstrating our framework’s ability to generate goal-oriented, coherent behavior sequences. (c) Our proposed Generative Behavior Control (GBC) framework bridges this gap between low-level motions and high-level behavioral understanding. Abstract Human moti… view at source ↗
Figure 2
Figure 2. Introduction of our PHYLOMAN Framework. To address Generative Behavior Control, our PHYLOMAN decom￾poses behavior generation into two primary stages: Task and Mo￾tion Planning (TAMP). In Task Planning, a text prompt (e.g., ‘Conductor directs the orchestra on stage’) is processed by an LLM to generate a hierarchical Composed Script, consisting of keyframe-based PoseScripts interleaved with transition-focused MotionSc… view at source ↗
Figure 3
Figure 3. Overview of collection process of our proposed GBC-100k dataset. We begin by collecting large-scale raw videos from the internet and filtering them to retain only clips featuring a single, full-body person. Next, we apply advanced motion estimation and captioning techniques to extract human motions and generate textual annotations from the filtered clips, as illustrated in (a). Finally, we structure our data in the … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative Comparison of Motion Generation Meth￾ods. Visual demonstration of motion sequences generated for the textual prompt: “A mechanic changes a tire on a bike in a garage.” Our proposed approach produces temporally extended sequences that better capture the comp…
Figure 6
Figure 6. Figure 6: Failed sample. 0 5 10 15 20 25 30 35 Duration (seconds) 0 50 100 150 200 250 300 350 Sampled Observation Count Distribution of Video Duration 0 25 50 75 100 125 150 175 200 Length (frames) 0 500 1000 1500 2000 2500 Sampled Observation Count Distribution of Segment Leng…
Figure 7
Figure 7. Figure 7: The duration statistics of sampled 3780 video clips and [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Data composition in GBC-100k. The dataset comprises contributions from multiple sources: HM3.6M (25.34%), Activ￾ityNet (23.10%), Kinetics (16.17%), HMDB (2.94%), UCF101 (0.11%), and others (32.34%). The “Others” category includes cu￾rated subsets from the Motion-X [47]…
Figure 9
Figure 9. Figure 9: The Length statistics of sampled 1000 BehaviorScripts [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

117 extracted references · 55 canonical work pages

  1. [1]

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Cheb- otar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario M Jauregui Ruano, Kyle Jeffrey, Sally Jes- month, Nikhil Jayant Joshi, Ryan C. Julian, Dmitry Kalash- nikov, Yuheng...

    2022

  2. [2]

    Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet

    Sonnet Anthropic. Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet. 8

  3. [3]

    Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Es- sam Sleiman, Deyao Zhu, Jian Ding, and Mohamed El- hoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.03413, 2024. 3

    arXiv 2024

  4. [4]

    En- hancing action recognition by leveraging the hierarchical structure of actions and textual context

    Manuel Benavent-Lledo, David Mulero-P ´erez, David Ortiz- Perez, Jose Garcia-Rodriguez, and Antonis Argyros. En- hancing action recognition by leveraging the hierarchical structure of actions and textual context. arXiv preprint arXiv:2410.21275, 2024. 3

    work page arXiv

  5. [5]

    Instance-based generalization in rein- forcement learning

    Martin Bertran, Natalia Martinez, Mariano Phielipp, and Guillermo Sapiro. Instance-based generalization in rein- forcement learning. Advances in Neural Information Pro- cessing Systems, 33:11333–11344, 2020. 3

    2020

  6. [6]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023. 2

    arXiv 2023

  7. [7]

    Activitynet: A large-scale video benchmark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceed- ings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015. 3

    2015

  8. [8]

    Motionclr: Motion generation and training-free edit- ing via understanding attention mechanisms

    Ling-Hao Chen, Wenxun Dai, Xuan Ju, Shunlin Lu, and Lei Zhang. Motionclr: Motion generation and training-free edit- ing via understanding attention mechanisms. arXiv preprint arXiv:2410.18977, 2024. 7, 2

    arXiv 2024

Show all 117 references
  1. [9]

    Autotamp: Autoregressive task and motion planning with llms as translators and check- ers

    Yongchao Chen, Jacob Arkin, Charles Dawson, Yang Zhang, Nicholas Roy, and Chuchu Fan. Autotamp: Autoregressive task and motion planning with llms as translators and check- ers. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6695–6702. IEEE, 2024. 2

    2024

  2. [10]

    Nod-tamp: Multi-step manipulation planning with neural object descriptors

    Shuo Cheng, Caelan Reed Garrett, Ajay Mandlekar, and Danfei Xu. Nod-tamp: Multi-step manipulation planning with neural object descriptors. In CoRL 2023 Workshop on Learning Effective Abstractions for Planning (LEAP), 2023. 2

    2023

  3. [11]

    Guided search for task and motion plans using learned heuristics

    Rohan Chitnis, Dylan Hadfield-Menell, Abhishek Gupta, Siddharth Srivastava, Edward Groshev, Christopher Lin, and Pieter Abbeel. Guided search for task and motion plans using learned heuristics. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 447–...

    2016

  4. [12]

    Flexible motion in-betweening with diffusion models

    Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, and Michiel van de Panne. Flexible motion in-betweening with diffusion models. In ACM SIGGRAPH 2024 Conference Pa- pers, pages 1–9, 2024. 2, 7, 8

    2024

  5. [13]

    Motionlcm: Real-time controllable motion generation via latent consistency model

    Wenxun Dai, Ling-Hao Chen, Jingbo Wang, Jinpeng Liu, Bo Dai, and Yansong Tang. Motionlcm: Real-time controllable motion generation via latent consistency model. In ECCV, pages 390–408, 2025. 2, 7, 3

    2025

  6. [14]

    Deepseek-v3 technical report, 2024

    DeepSeek-AI. Deepseek-v3 technical report, 2024. 8

    2024

  7. [15]

    Posescript: 3d human poses from natural language

    Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Gr ´egory Rogez. Posescript: 3d human poses from natural language. In European Con- ference on Computer Vision, pages 346–362. Springer, 2022. 2, 6, 8

    2022

  8. [16]

    Task and motion planning with large language models for ob- ject rearrangement

    Yan Ding, Xiaohan Zhang, Chris Paxton, and Shiqi Zhang. Task and motion planning with large language models for ob- ject rearrangement. In 2023 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), pages 2086–

    2023

  9. [17]

    Learning multi-modal whole-body control for real-world humanoid robots

    Pranay Dugar, Aayam Shrestha, Fangzhou Yu, Bart van Marum, and Alan Fern. Learning multi-modal whole-body control for real-world humanoid robots. arXiv preprint arXiv:2408.07295, 2024. 2

    2024 arXiv

  10. [18]

    Challenges of real-world reinforcement learning: definitions, benchmarks and analysis

    Gabriel Dulac-Arnold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 110(9):2419– 2468, 2021. 4

    2021

  11. [19]

    Optimal task and motion planning and execution for multiagent sys- tems in dynamic environments

    Marco Faroni, Alessandro Umbrico, Manuel Beschi, Andrea Orlandini, Amedeo Cesta, and Nicola Pedrocchi. Optimal task and motion planning and execution for multiagent sys- tems in dynamic environments. IEEE Transactions on Cy- bernetics, 2023. 3

    2023

  12. [20]

    Stratified avatar generation from sparse observations

    Han Feng, Wenchao Ma, Quankai Gao, Xianwei Zheng, Nan Xue, and Huijuan Xu. Stratified avatar generation from sparse observations. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 153–163, 2024. 2

    2024

  13. [21]

    Chatpose: Chatting about 3d human pose

    Yao Feng, Jing Lin, Sai Kumar Dwivedi, Yu Sun, Priyanka Patel, and Michael J Black. Chatpose: Chatting about 3d human pose. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2093–2103,

  14. [22]

    Integrated task and motion planning

    Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tom´as Lozano-P´erez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4(1):265–293, 2021. 2, 3

    2021

  15. [23]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 8

    2024 arXiv

  16. [24]

    Ac- tion2motion: Conditioned generation of 3d human motions

    Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Ac- tion2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2021–2029, 2020. 3

    2021

  17. [25]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152–5161, 2022. 2, 3

    2022

  18. [26]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5152–5161, 2022. 2

    2022

  19. [27]

    Momask: Generative masked model- ing of 3d human motions

    Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked model- ing of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1900–1910, 2024. 7, 8, 2

    1900

  20. [28]

    Robust motion in-betweening

    F ´elix G Harvey, Mike Yurick, Derek Nowrouzezahrai, and Christopher Pal. Robust motion in-betweening. ACM Trans- actions on Graphics (TOG), 39(4):60–1, 2020. 2

    2020

  21. [29]

    Omnih2o: Universal and dexterous human- to-humanoid whole-body teleoperation and learning

    Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous human- to-humanoid whole-body teleoperation and learning. arXiv preprint arXiv:2406.08858, 2024. 2

    2024 arXiv

  22. [30]

    Hover: Versatile neural whole-body controller for humanoid robots

    Tairan He, Wenli Xiao, Toru Lin, Zhengyi Luo, Zhen- jia Xu, Zhenyu Jiang, Jan Kautz, Changliu Liu, Guanya Shi, Xiaolong Wang, et al. Hover: Versatile neural whole-body controller for humanoid robots. arXiv preprint arXiv:2410.21229, 2024. 3, 6

    arXiv

  23. [31]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3, 6

    2020

  24. [32]

    An embodied generalist agent in 3d world

    Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. In Proceedings of the International Con- ference on Machine Learning (ICML), 2024. 2

    2024

  25. [33]

    Language models as zero-shot planners: Extract- ing actionable knowledge for embodied agents

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extract- ing actionable knowledge for embodied agents. In Interna- tional conference on machine learning , pages 9118–9147. PMLR, 2022. 3

    2022

  26. [34]

    Inner monologue: Em- bodied reasoning through planning with language models

    Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Em- bodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022. 2

    2022 arXiv

  27. [35]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 8

    2024 arXiv

  28. [36]

    Modelling human behaviour in cogni- tive tasks with latent dynamical systems

    Paul I Jaffe, Russell A Poldrack, Robert J Schafer, and Patrick G Bissett. Modelling human behaviour in cogni- tive tasks with latent dynamical systems. Nature Human Be- haviour, 7(6):986–1000, 2023. 2

    2023

  29. [37]

    A large-scale varying-view rgb-d action dataset for arbitrary-view human action recognition

    Yanli Ji, Feixiang Xu, Yang Yang, Fumin Shen, Heng Tao Shen, and Wei-Shi Zheng. A large-scale varying-view rgb-d action dataset for arbitrary-view human action recognition. arXiv preprint arXiv:1904.10681, 2019. 3

    1904 arXiv

  30. [38]

    Badler, and Tsukasa Noma

    Moon-Ryul Jung, Norman I. Badler, and Tsukasa Noma. An- imated human agents with motion planning capability for 3d-space postural goals. Comput. Animat. Virtual Worlds , 5:225–246, 1994. 8

    1994

  31. [39]

    Conditional motion in- betweening

    Jihoon Kim, Taehyun Byun, Seungyoun Shin, Jung- dam Won, and Sungjoon Choi. Conditional motion in- betweening. Pattern Recognition, 132:108894, 2022. 2

    2022

  32. [40]

    A survey of zero-shot generalisation in deep reinforcement learning

    Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rockt¨aschel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Re- search, 76:201–264, 2023. 3

    2023

  33. [41]

    Hmdb: a large video database for human motion recognition

    Hildegard Kuehne, Hueihan Jhuang, Est ´ıbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 Inter- national conference on computer vision , pages 2556–2563. IEEE, 2011. 3

    2011

  34. [42]

    Action recognition by hierarchical mid-level ac- tion elements

    Tian Lan, Yuke Zhu, Amir Roshan Zamir, and Silvio Savarese. Action recognition by hierarchical mid-level ac- tion elements. In Proceedings of the IEEE international con- ference on computer vision, pages 4552–4560, 2015. 3

    2015

  35. [43]

    The problem of serial order in behavior

    Karl Spencer Lashley et al. The problem of serial order in behavior. Bobbs-Merrill Oxford, 1951. 6

    1951

  36. [44]

    Planning non-repetitive robotic assembly processes with task and motion planning (tamp)

    Pok Yin Victor Leung, Yijiang Huang, Caelan Garret, Fabio Gramazio, and Matthias Kohler. Planning non-repetitive robotic assembly processes with task and motion planning (tamp). In Proceedings of Robotic Fabrication in Architec- ture, Art and Design 2024. Springer, 2024. 2

    2024

  37. [45]

    A diversity-promoting objective func- tion for neural conversation models

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective func- tion for neural conversation models. arXiv preprint arXiv:1510.03055, 2015. 3

    2015 arXiv

  38. [46]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In Conference on Empirical Methods in Natural Language Processing, 2023. 3

    2023

  39. [47]

    Motion-x: A large- scale 3d expressive whole-body human motion dataset

    Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large- scale 3d expressive whole-body human motion dataset. Ad- vances in Neural Information Processing Systems, 2023. 5

    2023

  40. [48]

    Modular multi-level replanning tamp framework for dynamic environ- ment

    Tao Lin, Chengfei Yue, Ziran Liu, and Xibin Cao. Modular multi-level replanning tamp framework for dynamic environ- ment. IEEE Robotics and Automation Letters, 2024. 2

    2024

  41. [49]

    Plan, posture and go: Towards open-vocabulary text-to-motion generation

    Jinpeng Liu, Wen-Dao Dai, Chunyu Wang, Yiji Cheng, Yan- song Tang, and Xin Tong. Plan, posture and go: Towards open-vocabulary text-to-motion generation. In European Conference on Computer Vision, 2024. 2

    2024

  42. [50]

    Humantomato: Text-aligned whole-body motion generation

    Shunlin Lu, Ling-Hao Chen, Ailing Zeng, Jing Lin, Ruimao Zhang, Lei Zhang, and Heung-Yeung Shum. Humantomato: Text-aligned whole-body motion generation. arXiv preprint arXiv:2310.12978, 2023. 2, 3, 4

    2023 arXiv

  43. [51]

    Posegpt: Quantization-based 3d human mo- tion generation and forecasting

    Thomas Lucas, Fabien Baradel, Philippe Weinzaepfel, and Gr´egory Rogez. Posegpt: Quantization-based 3d human mo- tion generation and forecasting. In European Conference on Computer Vision, pages 417–435. Springer, 2022. 2

    2022

  44. [52]

    Perpetual humanoid control for real-time simulated avatars

    Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10895–10904, 2023. 2, 6, 8

    2023

  45. [53]

    Universal hu- manoid motion representations for physics-based control

    Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, and Weipeng Xu. Universal hu- manoid motion representations for physics-based control. arXiv preprint arXiv:2310.04582, 2023. 2

    2023 arXiv

  46. [54]

    Isaac gym: High performance gpu-based physics simulation for robot learning

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470, 2021. 2

    2021 arXiv

  47. [55]

    Challenges and opportunities in deep reinforcement learning with graph neural networks: A comprehensive review of algorithms and applications

    Sai Munikoti, Deepesh Agarwal, Laya Das, Mahantesh Ha- lappanavar, and Balasubramaniam Natarajan. Challenges and opportunities in deep reinforcement learning with graph neural networks: A comprehensive review of algorithms and applications. IEEE transactions on neural networks...

    2023

  48. [56]

    Orb-slam: a versatile and accurate monocular slam system

    Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE transactions on robotics , 31(5):1147–1163,

  49. [57]

    Neural foundations of mental simula- tion: Future prediction of latent representations on dynamic scenes

    Aran Nayebi, Rishi Rajalingham, Mehrdad Jazayeri, and Guangyu Robert Yang. Neural foundations of mental simula- tion: Future prediction of latent representations on dynamic scenes. Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  50. [58]

    The neural and computational systems of social learning

    Andreas Olsson, Ewelina Knapska, and Bj ¨orn Lindstr ¨om. The neural and computational systems of social learning. Nature Reviews Neuroscience, 21(4):197–212, 2020. 2

    2020

  51. [59]

    Factored task and motion planning with combined optimization, sampling and learning

    Joaquim Ortiz-Haro. Factored task and motion planning with combined optimization, sampling and learning. arXiv preprint arXiv:2404.03567, 2024. 2

    2024 arXiv

  52. [60]

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pa...

    2019

  53. [61]

    The kit motion-language dataset

    Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, 4(4):236–252,

  54. [62]

    Babel: Bodies, action and behavior with english la- bels

    Abhinanda R Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, and Michael J Black. Babel: Bodies, action and behavior with english la- bels. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 722–731, 2021. 3

    2021

  55. [63]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learn- ing Research, 21(140):1–67, 2020. 7

    2020

  56. [64]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 3

    2022

  57. [65]

    Robot motion diffusion model: Motion generation for robotic characters

    Agon Serifi, Eth Z ¨urich, Switzerland Disney Research, Disney Research Switzerland Espen Knoop RUBEN GRANDIA, Eth Z¨urich Switzerland MARKUS GROSS, and Switzerland Moritz B ¨acher. Robot motion diffusion model: Motion generation for robotic characters. In ACM SIG- GRAPH Confe...

    2024

  58. [66]

    Human motion diffusion as a generative prior

    Yoni Shafir, Guy Tevet, Roy Kapon, and Amit Haim Bermano. Human motion diffusion as a generative prior. In The Twelfth International Conference on Learning Rep- resentations, 2024. 2

    2024

  59. [67]

    Ntu rgb+ d: A large scale dataset for 3d human activity anal- ysis

    Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+ d: A large scale dataset for 3d human activity anal- ysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1010–1019, 2016. 3

    2016

  60. [68]

    Aios: All-in-one-stage expressive human pose and shape estimation

    Qingping Sun, Yanjun Wang, Ailing Zeng, Wanqi Yin, Chen Wei, Wenjia Wang, Haiyi Mei, Chi-Sing Leung, Ziwei Liu, Lei Yang, et al. Aios: All-in-one-stage expressive human pose and shape estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2024

  61. [69]

    Deep reinforce- ment learning for robotics: A survey of real-world successes

    Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Mart´ın-Mart´ın, and Peter Stone. Deep reinforce- ment learning for robotics: A survey of real-world successes. arXiv preprint arXiv:2408.03539, 2024. 3

    2024 arXiv

  62. [70]

    Coin: A large-scale dataset for comprehensive instructional video analysis

    Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1207– 12...

    2019

  63. [71]

    Flag3d: A 3d fitness activity dataset with language instruction

    Yansong Tang, Jinpeng Liu, Aoyang Liu, Bin Yang, Wenxun Dai, Yongming Rao, Jiwen Lu, Jie Zhou, and Xiu Li. Flag3d: A 3d fitness activity dataset with language instruction. In CVPR, 2023. 5

    2023

  64. [72]

    Maskedmimic: Unified physics-based char- acter control through masked motion inpainting.ACM Trans

    Chen Tessler, Yunrong Guo, Ofir Nabati, Gal Chechik, and Xue Bin Peng. Maskedmimic: Unified physics-based char- acter control through masked motion inpainting.ACM Trans. Graph., 43:209:1–209:21, 2024. 2

    2024

  65. [73]

    Human motion dif- fusion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion dif- fusion model. arXiv preprint arXiv:2209.14916, 2022. 7

    2022 arXiv

  66. [74]

    Closd: Closing the loop between simulation and diffusion for multi-task character control

    Guy Tevet, Sigal Raab, Setareh Cohan, Daniele Reda, Zhengyi Luo, Xue Bin Peng, Amit H Bermano, and Michiel van de Panne. Closd: Closing the loop between simulation and diffusion for multi-task character control. arXiv preprint arXiv:2410.03441, 2024. 2

    arXiv

  67. [75]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, 2012. 2

    2012

  68. [76]

    Truong, Michael Piseno, Zhaoming Xie, and Karen Liu

    Takara E. Truong, Michael Piseno, Zhaoming Xie, and Karen Liu. Pdp: Physics-based character animation via dif- fusion policy. In ACM SIGGRAPH Conference and Exhi- bition on Computer Graphics and Interactive Techniques in Asia, 2024. 2

    2024

  69. [77]

    Large language models for robotics: Op- portunities, challenges, and perspectives

    Jiaqi Wang, Zihao Wu, Yiwei Li, Hanqi Jiang, Peng Shu, Enze Shi, Huawen Hu, Chong Ma, Yiheng Liu, Xuhui Wang, et al. Large language models for robotics: Op- portunities, challenges, and perspectives. arXiv preprint arXiv:2401.04334, 2024. 2

    2024 arXiv

  70. [78]

    Actionclip: A new paradigm for video action recognition

    Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472, 2021. 2

    2021 arXiv

  71. [79]

    Llm3: Large lan- guage model-based task and motion planning with motion failure reasoning

    Shu Wang, Muzhi Han, Ziyuan Jiao, Zeyu Zhang, Yingnian Wu, Song-Chun Zhu, and Hangxin Liu. Llm3: Large lan- guage model-based task and motion planning with motion failure reasoning. 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 12086– ...

    2024

  72. [80]

    Tram: Global trajectory and motion of 3d humans from in- the-wild videos

    Yufu Wang, Ziyun Wang, Lingjie Liu, and Kostas Daniilidis. Tram: Global trajectory and motion of 3d humans from in- the-wild videos. In European Conference on Computer Vi- sion, 2024. 2

    2024

  73. [81]

    Tram: Global trajectory and motion of 3d humans from in- the-wild videos

    Yufu Wang, Ziyun Wang, Lingjie Liu, and Kostas Daniilidis. Tram: Global trajectory and motion of 3d humans from in- the-wild videos. In European Conference on Computer Vi- sion, pages 467–487. Springer, 2025. 3

    2025

  74. [82]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models. Advances in neural information processing systems, 35:24824–24837, 2022. 8

    2022

  75. [83]

    Motion-agent: A conversational framework for human motion generation with llms

    Qi Wu, Yubo Zhao, Yifan Wang, Xinhang Liu, Yu-Wing Tai, and Chi-Keung Tang. Motion-agent: A conversational framework for human motion generation with llms. arXiv preprint arXiv:2405.17013, 2024. 3

    2024 arXiv

  76. [84]

    Physics-based human motion es- timation and synthesis from videos

    Kevin Xie, Tingwu Wang, Umar Iqbal, Yunrong Guo, Sanja Fidler, and Florian Shkurti. Physics-based human motion es- timation and synthesis from videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11532–11541, 2021. 3

    2021

  77. [85]

    I-ctrl: Imitation to control humanoid robots through constrained reinforcement learning

    Yashuai Yan, Esteve Valls Mascaro, Tobias Egle, and Dongheui Lee. I-ctrl: Imitation to control humanoid robots through constrained reinforcement learning. arXiv preprint arXiv:2405.08726, 2024. 2

    2024 arXiv

  78. [86]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  79. [87]

    Moconvq: Unified physics- based motion control via scalable discrete representations

    Heyuan Yao, Zhenhua Song, Yuyang Zhou, Tenglong Ao, Baoquan Chen, and Libin Liu. Moconvq: Unified physics- based motion control via scalable discrete representations. ACM Transactions on Graphics (TOG), 43(4):1–21, 2024. 2

    2024

  80. [88]

    Motionscript: Natural language descriptions for expressive 3d human motions

    Payam Jome Yazdian, Eric Liu, Li Cheng, and Angelica Lim. Motionscript: Natural language descriptions for expressive 3d human motions. arXiv preprint arXiv:2312.12634, 2023. 2, 1

    2023

  81. [89]

    Physdiff: Physics-guided human motion diffusion model

    Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVF international con- ference on computer vision , pages 16010–16021, 2023. 2, 3

    2023

  82. [90]

    Towards vision-based deep reinforce- ment learning for robotic motion control

    Fangyi Zhang, J ¨urgen Leitner, Michael Milford, Ben Up- croft, and Peter Corke. Towards vision-based deep reinforce- ment learning for robotic motion control. In Australasian Conference on Robotics and Automation, ACRA 2015, Aus- tralia, 2015. Australian Robotics and Automati...

    2015

  83. [91]

    Generating human motion from textual descriptions with discrete representations

    Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In CVPR, pages 14730–14740,

  84. [92]

    Kolmogorov-arnold fourier networks, 2025

    Jusheng Zhang, Yijia Fan, Kaitong Cai, and Keze Wang. Kolmogorov-arnold fourier networks, 2025. 3

    2025

  85. [93]

    Kabb: Knowledge-aware bayesian bandits for dy- namic expert coordination in multi-agent systems, 2025

    Jusheng Zhang, Zimeng Huang, Yijia Fan, Ningyuan Liu, Mingyan Li, Zhuojie Yang, Jiawei Yao, Jian Wang, and Keze Wang. Kabb: Knowledge-aware bayesian bandits for dy- namic expert coordination in multi-agent systems, 2025. 3

    2025

  86. [94]

    Motiondif- fuse: Text-driven human motion generation with diffusion model

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondif- fuse: Text-driven human motion generation with diffusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46:4115–4128, 2022. 2, 3, 4, 6

    2022

  87. [95]

    Incorporating physics principles for precise human motion prediction

    Yufei Zhang, Jeffrey O Kephart, and Qiang Ji. Incorporating physics principles for precise human motion prediction. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 6164–6174, 2024. 3

    2024

  88. [96]

    Motion mamba: Efficient and long sequence motion generation

    Zeyu Zhang, Akide Liu, Ian Reid, Richard Hartley, Bohan Zhuang, and Hao Tang. Motion mamba: Efficient and long sequence motion generation. In European Conference on Computer Vision, pages 265–282. Springer, 2025. 2

    2025

  89. [97]

    A survey of optimization- based task and motion planning: from classical to learn- ing approaches

    Zhigen Zhao, Shuo Cheng, Yan Ding, Ziyi Zhou, Shiqi Zhang, Danfei Xu, and Ye Zhao. A survey of optimization- based task and motion planning: from classical to learn- ing approaches. IEEE/ASME Transactions on Mechatronics,

  90. [99]

    PoseScript

    Terminology Here, we explain three key terms in our PHYLOMAN for readers unfamiliar with related topics. PoseScript. The term “PoseScript” is used to describe the specific configuration of the human body at a given mo- ment in time. This configuration is expressed through a de...

  91. [100]

    Notations We summarize notations in this paper as follows: X Configuration space of the SMPL model A Action space of MotionScripts T Trajectory space J Set of all joints K Set of all collision pairs xi,j Angle of joint j at configuration xi ¯vj, ¯aj Maximum allowable velocity ...

  92. [101]

    Behavior Constraints In this section, we expound on the technical and theoretical details of our proposed approach in Section 4.1. To synthe- size human motion that aligns with semantic expectations and physical feasibility, constraints are defined over three spaces: the confi...

  93. [102]

    Baselines We evaluate our PHYLOMAN on a variety of baselines that achieve state-of-the-art performance in generative qual- ity, diversity, and semantic alignment

    Additional Experimental Details 10.1. Baselines We evaluate our PHYLOMAN on a variety of baselines that achieve state-of-the-art performance in generative qual- ity, diversity, and semantic alignment. We briefly introduce each baseline as follows: • T2M-GPT [91]: Combines Vect...

  94. [103]

    Leveraging ActionCLIP fine- tuned on GBC-100K, we extract feature embeddings for both the generated motions and their textual counter- parts

    Multimodal Distance (MM Dist): Measures the align- ment between generated motions and their correspond- ing textual descriptions. Leveraging ActionCLIP fine- tuned on GBC-100K, we extract feature embeddings for both the generated motions and their textual counter- parts. The a...

  95. [104]

    MultiModality, in contrast, measures the diversity of generated motions conditioned on the same textual description

    Diversity and MultiModality: Diversity captures the variance of generated motions across the entire dataset by calculating the pairwise feature distances between randomly sampled motion sequences, following the def- initions outlined in MotionLCM [13]. MultiModality, in contra...

  96. [105]

    To compute SR, we conducted a human evaluation study using, where participants were presented with gen- erated motion sequences and their corresponding high- level directives

    Success Rate (SR): Evaluates the practical utility of the generated motion sequences in completing intended tasks. To compute SR, we conducted a human evaluation study using, where participants were presented with gen- erated motion sequences and their corresponding high- leve...

  97. [106]

    Penetrate mea- sures the distance between the ground and the lowest mesh vertex below it, while Float measures the distance of the lowest mesh vertex above the ground

    Physical Error (Phys-Err): Computed following the methodology from PhysDiff [89], it includes three com- ponents: ground penetration (Penetrate), floating viola- tions (Float), and foot sliding (Skate). Penetrate mea- sures the distance between the ground and the lowest mesh v...

  98. [107]

    FID is calculated by extracting embeddings using ActionCLIP and computing the Fr ´echet distance between the distri- butions of real and generated motions

    Fr´echet Inception Distance (FID): Used to evaluate the quality of generated motions by comparing their feature distributions to those of ground-truth motions. FID is calculated by extracting embeddings using ActionCLIP and computing the Fr ´echet distance between the distri- ...

  99. [108]

    summary":

    R-Precision: Assesses text-motion alignment by mea- suring the proportion of correct matches between gen- erated motions and ground-truth motions, given a tex- tual description. For each description, the top-k closest motions in the embedding space are retrieved, and R- Precis...

  100. [109]

    Identify Key Body Parts: Focus on arms, legs, head, torso

  101. [110]

    Use Defined Posecodes: • Angle Posecodes: – straight – slightly bent – partially bent – bent at a right angle – almost completely bent – completely bent • Distance Posecodes: – close – shoulder width apart – spread – wide apart • Relative Position Posecodes: – X-axis: ’at the ...

  102. [111]

    Subject Selection: Identify the most active joint as the subject

  103. [112]

    Ensure descriptions indicate static posture, not dynamic motion Rules for Transition:

  104. [113]

    Provide overview of human action

  105. [114]

    Use specified posecodes for describing changes

  106. [115]

    Include movement directions: Forward, Back- ward, Left, Right

  107. [116]

    Describe speed and magnitude of movements

  108. [117]

    A mechanic changes a tire on a bike in a garage

    Maintain temporal relationships between concur- rent movements Example Outputs: Keyframe: The person is standing upright with a slight forward lean. The left arm is slightly bent and extended out- ward. The right arm is bent at a right angle, with the hand positioned near the ...

  109. [2024]

    2, 3 From Motion to Behavior: Hierarchical Modeling of Humanoid Generative Behavior Control Supplementary Material

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.