Pith. sign in

REVIEW 4 major objections 5 minor 8 cited by

Learning from Massive Human Videos for Universal Humanoid Pose Control

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that massive human-video data, automatically converted into text-action pairs, gives a single model text-conditioned humanoid pose control that generalizes to new commands and deploys on a real robot at near-100% success.

desk verdict A serious dataset-and-pipeline paper with a strong real-robot demo, but the main benchmark is built by the same retargeting pipeline and the RL action labels lack per-sample validation, so the 'universal' claim needs support. read the letter →

arxiv 2412.14172 v1 pith:D6A6RHHS submitted 2024-12-18 cs.RO cs.AIcs.CLcs.CV

Jiageng Mao , Siheng Zhao , Siqi Song , Tianheng Shi , Junjie Ye , Mingtong Zhang , Haoran Geng , Jitendra Malik
show 2 more authors
Vitor Guizilini Yue Wang
This is my paper · ORCID
classification cs.ROcs.AIcs.CLcs.CV
keywords humanoidrobotcontroltext-conditionedmotiongenerationlarge-scalevideodatasetretargetingreinforcementlearningactiontokenizationTransformerreal-robotdeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that the bottleneck in humanoid robotics is data, not architecture. It builds Humanoid-X, a dataset of 163,800 video clips (over 20 million frames, about 240 hours) that are automatically converted into text-action pairs: each clip is captioned, reconstructed as a 3D human pose, retargeted onto a humanoid skeleton, and labeled with physically deployable joint actions by a goal-conditioned reinforcement-learning policy. On top of this dataset it trains UH-1, a Transformer that takes a text instruction and auto-regressively generates action tokens. The paper's central assertion is that training on this massive video-derived corpus achieves a level of generalization in humanoid control that was previously unattainable, supporting this with improved FID on the HumanoidML3D benchmark and near-100 percent success on 12 real-robot commands.

What carries the argument

The load-bearing machinery has two parts. First, the Humanoid-X annotation pipeline converts raw Internet video into five aligned modalities (video, text, SMPL human pose, humanoid keypoints, and humanoid actions), with the key transfer step being motion retargeting that fits human shape parameters to the robot and solves inverse kinematics for joint angles, followed by a goal-conditioned RL policy that turns keypoints into physically deployable target joint positions. Second, UH-1 uses a VQ-VAE action tokenizer that maps short action clips to discrete motion-primitive tokens and an autoregressive Transformer conditioned on a CLIP text embedding to generate those tokens, with a first-order reconstruction loss to keep generated motions temporally smooth. This design is what lets the model be trained on 20 million action frames and then decoded into PD-controller torques for a real robot.

What would settle it

Run the goal-conditioned policy of Appendix B.6 on a random sample of Humanoid-X keypoint sequences and record the fraction of episodes that keep the robot stable and within a defined tracking error (for example, root position error below a threshold with no fall); if a substantial fraction of sequences cannot be tracked, then a large share of the dataset's physically deployable action labels are wrong, which would invalidate the claim that UH-1 learns deployable actions from massive video data.

Watch

Extended reading notes

Core claim

The central claim is that massive, automatically labeled human-video data is a viable substitute for expensive teleoperation and narrow simulation when learning whole-body humanoid control. Concretely, the paper claims that UH-1 maps natural-language instructions directly to either humanoid keypoints (closed-loop, via a goal-conditioned policy) or joint target positions (open-loop), and that this mapping generalizes across a vocabulary of thousands of action words. The evidence reported is that UH-1 beats two-stage text-to-motion-plus-retargeting baselines on the retargeted HumanoidML3D benchmark, that pre-training on Humanoid-X improves FID from 0.445 to 0.379, that scaling the training set from 1% to 100% monotonically improves FID and diversity, and that on a real humanoid robot the model completes 12 distinct text commands at roughly 90-100% success in both control modes.

Load-bearing premise

The dataset's action labels come from a goal-conditioned RL policy trained on a curated subset of CMU MoCap from AMASS, not on the Humanoid-X videos themselves, and the paper reports no per-sample check that this policy successfully tracks the noisy, video-derived keypoints for all 163,800 samples; if the policy silently fails on motions outside its MoCap training distribution, much of the 20-million-sample training signal would be mislabeled.

Editorial extensions

If this is right

  • Text-conditioned humanoid control can scale with data: pre-training on Humanoid-X improves FID from 0.445 to 0.379 on HumanoidML3D, and scaling the training set from 1% to 100% improves FID from 0.689 to 0.463.
  • One model can cover a broad action vocabulary with a single text-to-action mapping, without an explicit intermediate human-motion generation step.
  • Open-loop text-to-action control is sufficient for many upper-body tasks when paired with a separate locomotion policy, while closed-loop text-to-keypoint control is more robust in simulation (mean success 94.33% versus 89.83%).
  • On the tested commands, the model transfers to a real humanoid robot with roughly 90-100% success in both control modes.
  • The first-order reconstruction loss in the action tokenizer improves the quality and temporal smoothness of generated humanoid actions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: if the RL-labeling assumption holds, the same pipeline should scale further by simply adding more video, since captioning, pose estimation, and retargeting are all automated; only scaling within a fixed 163,800-clip pool is demonstrated.
  • A testable extension of the central claim is full-body lower-limb control from video: the real-robot experiments mostly evaluate upper-body gestures with a separate locomotion policy, so whether massive video data improves legs, balance, and locomotion in the wild remains open.
  • The comparison against text-to-motion-plus-retargeting baselines suggests the intermediate human-motion generation step is a bottleneck; if UH-1's one-stage mapping is the cause of its gain, future work should focus on direct action-space modeling rather than improving generated 3D human motion quality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents two artifacts: Humanoid-X, a 163,800-sample dataset with over 20 million frames of text-labeled humanoid keypoints and target joint-position actions mined from videos, and UH-1, a VQ-VAE-tokenized autoregressive Transformer that maps text instructions to humanoid keypoints or actions. The data pipeline consists of clip mining, video captioning, SMPL pose estimation, retargeting to a humanoid skeleton, and goal-conditioned RL to produce deployable actions. The experiments report FID, diversity, and retrieval metrics on a retargeted HumanoidML3D benchmark, scaling and ablation curves, and success rates on 12 real-robot commands.

Significance. The scale of Humanoid-X and the attempt to turn unlabeled Internet video into physically deployable humanoid action labels are timely and potentially valuable; the controlled codebook-size and architecture ablations and the real-robot demonstrations are useful contributions. However, the evidence for the central generalization claim is currently incomplete: the RL-generated action labels are not validated per sample, and the main benchmark is generated with the paper's own retargeting pipeline. If the label-quality and benchmark concerns are addressed, this could be an important dataset and model; at present the load-bearing evidence is not yet convincing.

major comments (4)
  1. [§3.2 / Appendix B.6] The action labels Arobot are produced by a goal-conditioned RL policy trained only on a curated subset of CMU MoCap from AMASS, with interactions, heavy objects, and rough terrain excluded, and then applied to keypoints retargeted from noisy in-the-wild videos. The paper reports no per-sample tracking success, episode return, fall rate, keypoint-error threshold, or any other quality filter for the resulting 163,800 Arobot sequences. Because Arobot is the training target of the action tokenizer (Eq. 10) and of the autoregressive objective (Eq. 13), a substantial fraction of degenerate labels would contaminate the tokenizer and the transformer, and the scaling gains in Fig. 7 and Table 2 could partly reflect learning from invalid actions. I ask for a quantitative label-quality study, e.g., the fraction of clips whose policy rollout satisfies a motion-tracking reward or fall-avoidance threshold, and a corresponding filtering rule in the dataset release.
  2. [§5.1 / Table 1] The HumanoidML3D benchmark is created by the authors by retargeting HumanML3D with the same motion-retargeting pipeline used to build Humanoid-X. The comparisons against MDM and T2M-GPT therefore measure, at least in part, how well each method matches the authors' retargeting transformation rather than how well it controls a humanoid robot. The paper should add an independent evaluation, for example deploying the generated actions in a physics simulator with fixed balance and tracking rewards, or comparing with a whole-body humanoid control baseline under identical retargeting. Without such validation, the 'level of generalizability ... previously unattainable' claim in Section 1 is not fully supported.
  3. [§5.2 / Table 2] Table 2 claims that pre-training on Humanoid-X 'greatly improves' FID from 0.445 to 0.379. The reported 95% confidence intervals (0.445±0.078 and 0.379±0.046) overlap, so the improvement is not established without a paired test or a confidence interval on the difference. Please report paired bootstrap significance tests, or a repeated-evaluation protocol with fixed seeds, before using this result to support the scaling claim.
  4. [§5.3 / Table 3] The real-robot evaluation covers 12 commands, most of which are upper-body poses, and for text-to-action open-loop control the lower body is handled by a pre-trained locomotion policy (Section 5.3). Success is determined by a majority vote of five human evaluators (Appendix E.1), and each command is tested only 10 times. These results are encouraging evidence that the selected commands are deployable, but they do not certify the full 20M-sample action space or support the 'universal' scope implied by the paper's framing. I recommend either broadening the command set (including whole-body and locomotion tasks) or explicitly limiting the generalizability claim to the tested upper-body distribution.
minor comments (5)
  1. [§4 (after Eq. 13)] In the sentence 'we first tokenize each Arobot into Ztoken using Eq. 3', the reference should be to Eq. (10), not Eq. (3), which defines forward kinematics.
  2. [References [72] and [73]] References [72] and [73] are the same CVPR 2023 paper by Zhang et al.; the duplicate should be removed and the citations disambiguated.
  3. [Figure 7] The caption and surrounding text do not state whether the FID and diversity values are computed on a held-out subset of Humanoid-X or on HumanoidML3D after fine-tuning; please clarify the evaluation protocol so the numbers can be related to Tables 1 and 2.
  4. [Table 3 / Appendix E.1] The success rates are point estimates from 10 trials; please report binomial confidence intervals or the raw counts, and specify how the five evaluators were recruited or whether they were blinded to the commanded instruction.
  5. [Appendix C.3] The release statement says data and code will be released 'in the future' but gives no timeline or plan; for a dataset-centric paper, a concrete release protocol (including split definitions and source-video handling) should be described.

Circularity Check

1 steps flagged · score 5.0 of 10

HumanoidML3D is described as a selected set of Humanoid-X, yet Section 5.2 pre-trains on Humanoid-X and evaluates on HumanoidML3D, making the headline FID gain partially in-distribution.

  1. fitted input called prediction [Appendix D.3; Section 5.2, Table 2]
    "Training the UH-1 Action Tokenizer and the UH-1 Transformer on HumanoidML3D (a selected set of Humanoid-X) requires approximately 8 hours and 30 hours, respectively, on a single NVIDIA RTXTM 6000 Ada GPU ... we first pre-trained our UH-1 model on the Humanoid-X dataset and then finetuned and evaluated the performance on the HumanoidML3D benchmark."

    Appendix D.3 identifies HumanoidML3D as a selected set of Humanoid-X. Section 5.2 then pre-trains on the full Humanoid-X and evaluates on HumanoidML3D. Consequently the Table 2 FID/MM Dist/Diversity improvements are measured on data contained in the pretraining set, so the reported gain partially reflects in-distribution memorization rather than generalization to an external benchmark. Section 5.1 describes HumanoidML3D as a transform of the external HumanML3D, creating a direct internal contradiction, and the paper never states that the HumanoidML3D subset was held out from Humanoid-X pretraining. The 'prediction' of improved generalization is therefore not fully independent of the training input.

full rationale

The paper's main derivation chain is a data-processing pipeline (video -> SMPL -> retargeting -> RL-policy labels -> UH-1 tokenizer/transformer), and the text-to-action model is tested on real hardware, so the central method is not a tautology. However, the principal quantitative benchmark is compromised: Appendix D.3 calls HumanoidML3D 'a selected set of Humanoid-X,' while the pretraining experiment in Section 5.2 uses all of Humanoid-X and then evaluates on HumanoidML3D. Unless the benchmark subset was explicitly excluded from pretraining—which the paper does not state—Table 2's headline FID improvement is an in-distribution comparison, not an external generalization result. A second, non-circular but load-bearing concern is that the goal-conditioned RL policy used to generate Arobot is trained only on a curated CMU MoCap subset (Appendix B.6) and is applied to all 163,800 video-derived samples without per-sample tracking or quality filtering; this makes the physical-deployability of many labels an unsupported assumption. The real-robot trials, the simulated control-mode success rates, and the baseline comparisons under identical retargeting provide independent evidence that keeps the paper from being fully circular, so a moderate score is appropriate.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a long pipeline of hand-chosen thresholds, loss weights, and a policy trained on MoCap data outside the video distribution. No new physical entities are introduced. The main unvalidated ingredient is the RL policy's coverage over the 20M samples.

free parameters (8)
  • Action tokenizer codebook size = 2048 x 512
    Chosen by hand; ablation shows FID and diversity improve from 512 to 2048, so the result depends on this design choice.
  • Temporal downsampling rate k = 4
    Selected for action tokenizer; each token encodes 4 frames.
  • Token loss weights (alpha, beta, gamma) = not reported
    Weights for embedding, forward-difference, and root-regularization terms in Eq. (13); values are not given.
  • RL reward weights = see Appendix Table 2/3
    Hand-selected weights such as 3.0, 2.0, 6.0, 1.0 for imitation and regularization rewards; these determine whether retargeted motions are tracked.
  • IK smoothing term lambda = 0.05
    Regularizes inverse kinematics across frames (Appendix B.5).
  • Shape parameter bound = |beta_i| < 5
    Limits SMPL shape optimization during retargeting (Appendix B.5).
  • Assumed focal length f for depth = predefined, value not reported
    Used to convert weak-perspective camera to root translation t_z (Appendix B.4).
  • Motion detection thresholds = not reported
    Grayscale frame-difference thresholds and batch thresholds decide which clips enter Humanoid-X.
assumptions (5)
  • domain assumption VIBE estimates 3D SMPL pose and root translation accurately enough from in-the-wild single-person videos.
    Invoked in Section 3.2 '3D Human Pose Estimation'; noisy pose estimates propagate through retargeting and RL labels.
  • domain assumption The 12 matched joints between humans and the Unitree H1-2 robot make joint-position retargeting a valid transfer.
    Section 3.2 Motion Retargeting; ignores dynamics, contact, and mass distribution differences.
  • ad hoc to paper A goal-conditioned RL policy trained on a curated CMU MoCap subset generalizes to all retargeted video motions without quality filtering.
    Appendix B.6; no per-sample success statistics are reported, weakening the dataset-label premise.
  • domain assumption Action captions generated by Video-LLaMA with a fixed prompt are semantically aligned with the retargeted motions.
    Section 3.2 Video Captioning; caption errors would mislabel text-conditioned training.
  • domain assumption PD controllers with official Kp/Kd gains convert target joint positions into stable torques on the physical robot.
    Appendix E.1; standard assumption in robot control.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning from Massive Human Videos for Universal Humanoid Pose Control." pith.science (2026). https://pith.science/paper/D6A6RHHS

@misc{pith2026241214172,
  author       = {Pith},
  title        = {Pith review of: Learning from Massive Human Videos for Universal Humanoid Pose Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D6A6RHHS}},
  note         = {Machine review of arXiv:2412.14172}
}
read the original abstract

Scalable learning of humanoid robots is crucial for their deployment in real-world applications. While traditional approaches primarily rely on reinforcement learning or teleoperation to achieve whole-body control, they are often limited by the diversity of simulated environments and the high costs of demonstration collection. In contrast, human videos are ubiquitous and present an untapped source of semantic and motion information that could significantly enhance the generalization capabilities of humanoid robots. This paper introduces Humanoid-X, a large-scale dataset of over 20 million humanoid robot poses with corresponding text-based motion descriptions, designed to leverage this abundant data. Humanoid-X is curated through a comprehensive pipeline: data mining from the Internet, video caption generation, motion retargeting of humans to humanoid robots, and policy learning for real-world deployment. With Humanoid-X, we further train a large humanoid model, UH-1, which takes text instructions as input and outputs corresponding actions to control a humanoid robot. Extensive simulated and real-world experiments validate that our scalable training approach leads to superior generalization in text-based humanoid control, marking a significant step toward adaptable, real-world-ready humanoid robots.

Figures

Figures reproduced from arXiv: 2412.14172 by the authors.

Figure 1
Figure 1. Overview. We introduce Humanoid-X, a large-scale dataset to facilitate humanoid robot learning from massive human videos. On top of Humanoid-X, we introduce UH-1, a large humanoid model for universal language-conditioned pose control of humanoid robots. Abstract Scalable learning of humanoid robots is crucial for their deployment in real-world applications. While traditional approaches primarily rely on reinforcemen… view at source ↗
Figure 2
Figure 2. Learning Humanoid Pose Control from Massive Videos. We mine massive human-centric video clips V from the Inter￾net. We then extract text-based action descriptions T and 3D human poses Phuman from the video clips. Next, we retarget the mo￾tions from humans to humanoid robots, resulting in humanoid keypoints Probot for high-level control. Finally, we employ reinforce￾ment learning to generate physically deployable hum… view at source ↗
Figure 3
Figure 3. Dataset Statistics. Humanoid-X features extensive scale, diverse sources, a rich action vocabulary, and multiple data modalities. videos, we designed over 400 unique search terms covering a range of human activities from daily tasks to professional sports, and then utilized the Google Cloud API* to retrieve the top 20 videos for each specified search term. Original videos are often noisy, including segments with no … view at source ↗
Figures from the paper (47 more)
Figure 4
Figure 4. Figure 4: UH-1 Model Architecture. UH-1 leverages the Trans￾former for scalable learning. Humanoid actions are first tokenized into discrete action tokens. Then, we train the UH-1 Transformer that takes text commands as inputs and auto-regressively generates the corresponding hu…
Figure 5
Figure 5. Figure 5: Text-to-keypoint and text-to-action control modes. UH-1 can either generate high-level humanoid keypoints (text￾to-keypoint) for the goal-conditioned policy π to control the hu￾manoid robot in closed-loop, or generate robotic actions Arobot for direct open-loop control…
Figure 6
Figure 6. Figure 6: Real robot experiment. UH-1 model can be reliably deployed on the real humanoid robot with a nearly 100% success rate. Dataset FID ↓ MM Dist ↓ Diversity ↑ R Precision ↑ Oracle 0.005±.001 3.140±.010 9.846±.062 0.780±.003 HumanoidML3D 0.445±.078 3.249±.016 10.157±.106 0.…
Figure 7
Figure 7. Figure 7: Effectiveness of scaling up training data. Points indi￾cate the mean values, and error bars indicate the 95% confidence interval. Increasing the dataset size from 1% to 100% leads to sig￾nificant improvements in both FID and Diversity metric. improvement in all metrics…
Figure 8
Figure 8. Figure 8: Simulated experiments on the UH-1 control modes. Bars indicate success rates for specific commands and dash lines show the mean success rate on 12 different text instructions. While Text-to-Action mode with a separate locomotion policy is sufficient for most tasks, Tex…
Figure 9
Figure 9. Figure 9: Ablation on the vocabulary sizes of the UH-1 action tokenizer. Increasing the vocabulary size of the action tokenizer provides more motion primitives for humanoid robots and thus leads to an improvement in both FID and Diversity metric. the diffusion-based model. 6. Co…
Figure 1
Figure 1. Figure 1: Video Processing Pipeline. generated terms are “karate front kick training,” “taek￾wondo spinning hook kick demonstration,” and “wushu staff spin practice.” Fitness and exercise drills focused on isolated movements like “yoga handstand practice,” and “calisthenics plan…
Figure 2
Figure 2. Figure 2: Video captioning example by using Video-LLaMA. down-sampled for efficiency, interpolated for smooth tran￾sitions, and standardized to 20 FPS. The resulting clips fo￾cus exclusively on single-person actions, discarding distrac￾tions such as multiple individuals or irrel…
Figure 3
Figure 3. Figure 3: SMPL 3D human model estimation example. where Wimg represents the width of the input image. The root translation vector Troot is then formed as: Troot =   tx ty tz   , (2) where t = (tx, ty) corresponds to the 2D translations from the camera parameters, and tz is t…
Figure 4
Figure 4. Figure 4: Motion Retargeting, including optimization of human shape parameters and solving humanoid motor DoF positions. where sroot denotes robot root states including root transla￾tion and root orientation, Frk denotes robot forward kine￾matics which maps from qrobot, sroot to…
Figure 5
Figure 5. Figure 5: Distribution of video length (in seconds) and captioning [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Verbs Word Cloud and Top-40 Frequent Verbs. (a) Nouns WordCloud (b) Top-40 Nouns [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Nouns Word Cloud and Top-40 Frequent Nouns. sample the SMPL visualization of the corresponding frames in the video clip. For the humanoid pose, we set the hu￾manoid keypoints in MuJoCo and collected the MuJoCo￾rendered images of the corresponding frames in the video cl…
Figure 8
Figure 8. Figure 8: Adjectives Word Cloud and Top-40 Frequent Adjectives. (a) Adverb WordCloud (b) Top-40 Adverb [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Adverbs Word Cloud and Top-40 Frequent Adverbs. In this formulation, α is a hyperparameter that regulates the relative influence of each loss term, and sg[·] denotes the stop gradient operator. The embedding loss Lembed pro￾motes the quantized codebook embeddings to mo…
Figure 10
Figure 10. Figure 10: Ablation on different RL policies, measured by task cumulative reward value. The solid line represents the mean re￾turn value, while the shaded regions correspond to the standard deviation, both calculated across five different random seeds. Our retargeted training da…
Figure 11
Figure 11. Figure 11: Data samples in Humanoid-X. Human Pose Humanoid Pose Human Video Humanoid Action Text A man is riding a scooter on the sidewalk near the water [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Data examples in Humanoid-X. Human Pose Humanoid Pose Human Video Humanoid Action Text The human is a man doing something in the video [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14 [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Data examples in Humanoid-X. Human Pose Humanoid Pose Human Video Humanoid Action Text The human is jumping up and down in the air [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16 [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Data examples in Humanoid-X. Human Pose Humanoid Pose Human Video Humanoid Action Text The human is doing a workout in a gym, specifically jumping rope [PITH_FULL_IMAGE:figures/full_fig_p024_17.png]
Figure 18
Figure 18. Figure 18 [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]
Figure 19
Figure 19. Figure 19: Data examples in Humanoid-X. Human Pose Humanoid Pose Human Video Humanoid Action Text The human is doing yoga in the living room, standing on a wooden floor and stretching his arms and legs [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20 [PITH_FULL_IMAGE:figures/full_fig_p025_20.png]
Figure 21
Figure 21. Figure 21: Data examples in Humanoid-X. Human Pose Humanoid Pose Human Video Humanoid Action Text A woman is doing yoga in a room with a large window, a green plant, and a brown wooden chair [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 22
Figure 22. Figure 22 [PITH_FULL_IMAGE:figures/full_fig_p026_22.png]
Figure 23
Figure 23. Figure 23: Data examples in Humanoid-X. Human Pose Humanoid Pose Human Video Humanoid Action Text A man is standing in a room and using a treadmill to exercise [PITH_FULL_IMAGE:figures/full_fig_p027_23.png]
Figure 24
Figure 24. Figure 24 [PITH_FULL_IMAGE:figures/full_fig_p027_24.png]
Figure 25
Figure 25. Figure 25: Data examples in Humanoid-X. Human Pose Humanoid Pose Human Video Humanoid Action Text The human is sitting on a chair in a room and doing some exercises with a resistance band [PITH_FULL_IMAGE:figures/full_fig_p028_25.png]
Figure 26
Figure 26. Figure 26 [PITH_FULL_IMAGE:figures/full_fig_p028_26.png]
Figure 27
Figure 27. Figure 27: Data examples in Humanoid-X. Human Pose Humanoid Pose Human Video Humanoid Action Text A man is standing on a balcony and pouring water on his head [PITH_FULL_IMAGE:figures/full_fig_p029_27.png]
Figure 28
Figure 28. Figure 28 [PITH_FULL_IMAGE:figures/full_fig_p029_28.png]
Figure 29
Figure 29. Figure 29: Data examples in Humanoid-X. Human Pose Humanoid Pose Human Video Humanoid Action Text A man is standing in an office building, holding a white plastic cup in his hand [PITH_FULL_IMAGE:figures/full_fig_p030_29.png]
Figure 30
Figure 30. Figure 30 [PITH_FULL_IMAGE:figures/full_fig_p030_30.png]
Figure 31
Figure 31. Figure 31: Data examples in Humanoid-X. Human Pose Humanoid Pose Human Video Humanoid Action Text A man is standing in an office and doing something with his hands [PITH_FULL_IMAGE:figures/full_fig_p031_31.png]
Figure 32
Figure 32. Figure 32 [PITH_FULL_IMAGE:figures/full_fig_p031_32.png]
Figure 33
Figure 33. Figure 33: Real robot demonstrations. Text instruction: Shooting a Ball to the Basket. Playing Guitar [PITH_FULL_IMAGE:figures/full_fig_p032_33.png]
Figure 34
Figure 34. Figure 34: Real robot demonstrations. Text instruction: Playing Guitar [PITH_FULL_IMAGE:figures/full_fig_p032_34.png]
Figure 35
Figure 35. Figure 35: Real robot demonstrations. Text instruction: Putting in a Golf Tournament. Waving to a Friend [PITH_FULL_IMAGE:figures/full_fig_p033_35.png]
Figure 36
Figure 36. Figure 36: Real robot demonstrations. Text instruction: Waving to a Friend [PITH_FULL_IMAGE:figures/full_fig_p033_36.png]
Figure 37
Figure 37. Figure 37: Real robot demonstrations. Text instruction: Playing Violin. Punching with the Right Hand [PITH_FULL_IMAGE:figures/full_fig_p034_37.png]
Figure 38
Figure 38. Figure 38: Real robot demonstrations. Text instruction: Punching with the Right Hand [PITH_FULL_IMAGE:figures/full_fig_p034_38.png]
Figure 39
Figure 39. Figure 39: Real robot demonstrations. Text instruction: Playing Drums [PITH_FULL_IMAGE:figures/full_fig_p035_39.png]
Figure 40
Figure 40. Figure 40: Demonstration of human-humanoid interactions. Text instruction: High-Five [PITH_FULL_IMAGE:figures/full_fig_p035_40.png]
Figure 41
Figure 41. Figure 41: Demonstration of human-humanoid interactions. Text instruction: Embrace [PITH_FULL_IMAGE:figures/full_fig_p035_41.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LangWBC: Language-directed Humanoid Whole-Body Control via End-to-end Learning

    cs.RO 2025-04 conditional novelty 7.0 of 10

    LangWBC maps text commands and proprioceptive history directly to humanoid actions via a conditional variational autoencoder distilled from an RL teacher, enabling diverse, smoothly transitioning whole-body motions on...

  2. MRBench: A Comprehensive Benchmark for Human Motion-Text Retrieval

    cs.CV 2026-08 conditional novelty 6.0 of 10

    MRBench is a multi-source, balanced, multi-granular human motion-text retrieval benchmark, and the proposed granularity-aware adapters improve mixed-granularity retrieval without degrading standard-caption retrieval.

  3. ThorArena: Benchmarking Humanoid Physical Interaction with Human Motion-Force Demonstrations

    cs.RO 2026-07 conditional novelty 6.0 of 10

    A force-aware humanoid benchmark pairs synchronized human motion-force data with simulation-based force replay to evaluate whole-body control policies under realistic physical disturbances.

  4. OmniMotion-X: Versatile Multimodal Whole-Body Motion Generation

    cs.CV 2025-10 conditional novelty 6.0 of 10

    A single autoregressive diffusion model, trained on a new 286-hour SMPL-X dataset, generates whole-body motion from text, audio, and spatial-temporal control signals, with reference-motion conditioning.

  5. Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation

    cs.RO 2025-10 conditional novelty 6.0 of 10

    A 10,300-demonstration, 260-task multimodal humanoid manipulation dataset with baseline policy evaluations and a cloud evaluation platform.

  6. GMT: General Motion Tracking for Humanoid Whole-Body Control

    cs.RO 2025-06 conditional novelty 6.0 of 10

    GMT trains a single unified humanoid policy using adaptive sampling and mixture-of-experts, achieving lower tracking errors than a re-implemented ExBody2 across diverse whole-body motions.

  7. KungfuBot: Physics-Based Humanoid Whole-Body Control for Learning Highly-Dynamic Skills

    cs.RO 2025-06 conditional novelty 6.0 of 10

    A robot control method that adaptively tightens motion-tracking reward tolerances achieves lower tracking errors on dynamic skills and transfers zero-shot to a real humanoid.

  8. RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learning

    cs.RO 2025-04 conditional novelty 5.0 of 10

    RoboVerse unifies seven simulators, 15 benchmarks, and 510.5k migrated trajectories into one platform with a four-level generalization benchmark, claiming better robot learning and sim-to-real transfer.

Reference graph

Works this paper leans on

79 extracted references · 53 canonical work pages · cited by 8 Pith papers

  1. [1]

    Human- to-robot imitation in the wild

    Shikhar Bahl, Abhinav Gupta, and Deepak Pathak. Human- to-robot imitation in the wild. In RSS, 2022. 2

    work page 2022

  2. [2]

    Affordances from human videos as a versa- tile representation for robotics

    Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as a versa- tile representation for robotics. In CVPR, 2023. 1, 2

    work page 2023

  3. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 1

    arXiv 2023

  4. [4]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In CoRL, 2023. 1

    work page 2023

  5. [5]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. In RSS, 2023. 1

    work page 2023

  6. [6]

    Humman: Multi-modal 4d human dataset for ver- satile sensing and modeling

    Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, et al. Humman: Multi-modal 4d human dataset for ver- satile sensing and modeling. In ECCV. Springer, 2022. 3

    work page 2022

  7. [7]

    A short note on the kinetics-700 human action dataset

    Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zis- serman. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987, 2019. 3

    arXiv 1907

  8. [8]

    Learning smooth humanoid locomotion through lipschitz-constrained policies

    Zixuan Chen, Xialin He, Yen-Jen Wang, Qiayuan Liao, Yan- jie Ze, Zhongyu Li, S Shankar Sastry, Jiajun Wu, Koushil Sreenath, Saurabh Gupta, et al. Learning smooth humanoid locomotion through lipschitz-constrained policies. arXiv preprint arXiv:2410.11825, 2024. 2

    arXiv

Show all 79 references
  1. [9]

    Expressive whole-body con- trol for humanoid robots

    Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive whole-body con- trol for humanoid robots. arXiv preprint arXiv:2402.16796,

    arXiv

  2. [10]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024. 4, 3

    2024 arXiv

  3. [11]

    Haa500: Human-centric atomic action dataset with curated videos

    Jihoon Chung, Cheng-hsin Wuu, Hsuan-ru Yang, Yu-Wing Tai, and Chi-Keung Tang. Haa500: Human-centric atomic action dataset with curated videos. In ICCV, 2021. 3

    2021

  4. [12]

    Video language plan- ning

    Yilun Du, Mengjiao Yang, Pete Florence, Fei Xia, Ayzaan Wahid, Brian Ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B Tenenbaum, et al. Video language plan- ning. In ICLR, 2024. 2

    2024

  5. [13]

    Humanplus: Humanoid shadowing and imita- tion from humans

    Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wetzstein, and Chelsea Finn. Humanplus: Humanoid shadowing and imita- tion from humans. arXiv preprint arXiv:2406.10454, 2024. 2

    2024 arXiv

  6. [14]

    Humans in 4d: Re- constructing and tracking humans with transformers

    Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. Humans in 4d: Re- constructing and tracking humans with transformers. In ICCV, 2023. 3

    2023

  7. [15]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In CVPR, 2024. 1

    2024

  8. [16]

    Humanoid- gym: Reinforcement learning for humanoid robot with zero- shot sim2real transfer

    Xinyang Gu, Yen-Jen Wang, and Jianyu Chen. Humanoid- gym: Reinforcement learning for humanoid robot with zero- shot sim2real transfer. arXiv preprint arXiv:2404.05695 ,

    arXiv

  9. [17]

    Advanc- ing humanoid locomotion: Mastering challenging terrains with denoising world model learning

    Xinyang Gu, Yen-Jen Wang, Xiang Zhu, Chengming Shi, Yanjiang Guo, Yichen Liu, and Jianyu Chen. Advanc- ing humanoid locomotion: Mastering challenging terrains with denoising world model learning. arXiv preprint arXiv:2408.14472, 2024. 2

    2024 arXiv

  10. [18]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In CVPR, 2022. 3, 6

    2022

  11. [19]

    Omnih2o: Universal and dexterous human- to-humanoid whole-body teleoperation and learning

    Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous human- to-humanoid whole-body teleoperation and learning. arXiv preprint arXiv:2406.08858, 2024. 2, 3, 9

    2024 arXiv

  12. [20]

    Learning human- to-humanoid real-time whole-body teleoperation

    Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Learning human- to-humanoid real-time whole-body teleoperation. arXiv preprint arXiv:2403.04436, 2024. 2, 4

    2024 arXiv

  13. [21]

    Hover: Versatile neural whole-body controller for humanoid robots

    Tairan He, Wenli Xiao, Toru Lin, Zhengyi Luo, Zhen- jia Xu, Zhenyu Jiang, Jan Kautz, Changliu Liu, Guanya Shi, Xiaolong Wang, et al. Hover: Versatile neural whole-body controller for humanoid robots. arXiv preprint arXiv:2410.21229, 2024. 2

    arXiv

  14. [22]

    Motiongpt: Human motion as a foreign language

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. In NeurIPS, 2023. 3

    2023

  15. [23]

    Harmon: Whole-body motion generation of humanoid robots from language descriptions

    Zhenyu Jiang, Yuqi Xie, Jinhan Li, Ye Yuan, Yifeng Zhu, and Yuke Zhu. Harmon: Whole-body motion generation of humanoid robots from language descriptions. arXiv preprint arXiv:2410.12773, 2024. 3

    2024 arXiv

  16. [24]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. 1

    2024 arXiv

  17. [25]

    Adam: A method for stochastic opti- mization

    Diederik P Kingma. Adam: A method for stochastic opti- mization. In ICLR, 2014. 5, 4

    2014

  18. [26]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In ICCV, 2023. 1

    2023

  19. [27]

    Vibe: Video inference for human body pose and shape estimation

    Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. Vibe: Video inference for human body pose and shape estimation. In CVPR, 2020. 4, 3

    2020

  20. [28]

    Ram: Retrieval-based affordance transfer for gen- eralizable zero-shot robotic manipulation

    Yuxuan Kuang, Junjie Ye, Haoran Geng, Jiageng Mao, Congyue Deng, Leonidas Guibas, He Wang, and Yue Wang. Ram: Retrieval-based affordance transfer for gen- eralizable zero-shot robotic manipulation. arXiv preprint arXiv:2407.04689, 2024. 1

    2024 arXiv

  21. [29]

    Okami: Teaching hu- manoid robots manipulation skills through single video imi- tation

    Jinhan Li, Yifeng Zhu, Yuqi Xie, Zhenyu Jiang, Mingyo Seo, Georgios Pavlakos, and Yuke Zhu. Okami: Teaching hu- manoid robots manipulation skills through single video imi- tation. arXiv preprint arXiv:2410.11792, 2024. 2

    2024 arXiv

  22. [30]

    Robust and versatile bipedal jumping control through reinforcement learning

    Zhongyu Li, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, and Koushil Sreenath. Robust and versatile bipedal jumping control through reinforcement learning. In RSS, 2023. 2

    2023

  23. [31]

    Intergen: Diffusion-based multi-human motion genera- tion under complex interactions

    Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi-human motion genera- tion under complex interactions. IJCV, 2024. 3

    2024

  24. [32]

    Motion-x: A large- scale 3d expressive whole-body human motion dataset

    Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: A large- scale 3d expressive whole-body human motion dataset. In NeurIPS, 2024. 3

    2024

  25. [33]

    Smpl: a skinned multi- person linear model

    Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: a skinned multi- person linear model. TOG, 34(6), 2015. 3, 4

    2015

  26. [34]

    Perpetual humanoid control for real-time simulated avatars

    Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. In ICCV, 2023. 3

    2023

  27. [35]

    Universal hu- manoid motion representations for physics-based control

    Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, and Weipeng Xu. Universal hu- manoid motion representations for physics-based control. In ICLR, 2024. 3

    2024

  28. [36]

    Vip: Towards universal visual reward and representation via value-implicit pre-training

    Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Os- bert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. In ICLR, 2023. 2

    2023

  29. [37]

    Troje, Ger- ard Pons-Moll, and Michael J

    Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Ger- ard Pons-Moll, and Michael J. Black. Amass: Archive of motion capture as surface shapes. In ICCV, 2019. 4

    2019

  30. [38]

    Struc- tured world models from human videos

    Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Struc- tured world models from human videos. In RSS, 2023. 2

    2023

  31. [39]

    R3m: A universal visual repre- sentation for robot manipulation

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual repre- sentation for robot manipulation. In CoRL, 2022. 2

    2022

  32. [40]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al. Open x-embodiment: Robotic learning datasets and rt-x models. In ICRA, 2023. 1

    2023

  33. [41]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 1

    2023 arXiv

  34. [42]

    Amp: Adversarial motion priors for styl- ized physics-based character control

    Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for styl- ized physics-based character control. TOG, 40(4), 2021. 3

    2021

  35. [43]

    Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters

    Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. TOG, 41 (4), 2022. 3

    2022

  36. [44]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML. PMLR, 2021. 1

    2021

  37. [45]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML. PMLR, 2021. 6, 8

    2021

  38. [46]

    Robot learning with sen- sorimotor pre-training

    Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot learning with sen- sorimotor pre-training. In CoRL. PMLR, 2023. 2

    2023

  39. [47]

    Learning humanoid locomotion over challenging terrain

    Ilija Radosavovic, Sarthak Kamat, Trevor Darrell, and Jiten- dra Malik. Learning humanoid locomotion over challenging terrain. arXiv preprint arXiv:2410.03654, 2024. 2

    2024 arXiv

  40. [48]

    Real-world humanoid locomotion with reinforcement learning

    Ilija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, and Koushil Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 9(89), 2024. 2

    2024

  41. [49]

    Humanoid locomotion as next token pre- diction

    Ilija Radosavovic, Bike Zhang, Baifeng Shi, Jathushan Ra- jasegaran, Sarthak Kamat, Trevor Darrell, Koushil Sreenath, and Jitendra Malik. Humanoid locomotion as next token pre- diction. arXiv preprint arXiv:2402.19469, 2024. 2

    2024 arXiv

  42. [50]

    Real-time flying object detection with yolov8.arXiv preprint arXiv:2305.09972, 2023

    Dillon Reis, Jordan Kupec, Jacqueline Hong, and Ahmad Daoudi. Real-time flying object detection with yolov8.arXiv preprint arXiv:2305.09972, 2023. 4, 2

    2023 arXiv

  43. [51]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022. 1

    2022

  44. [52]

    Proximal policy optimization algo- rithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms. arXiv preprint arXiv:1707.06347, 2017. 5

    2017 arXiv

  45. [53]

    Deep imita- tion learning for humanoid loco-manipulation through hu- man teleoperation

    Mingyo Seo, Steve Han, Kyutae Sim, Seung Hyeon Bang, Carlos Gonzalez, Luis Sentis, and Yuke Zhu. Deep imita- tion learning for humanoid loco-manipulation through hu- man teleoperation. In Humanoids. IEEE, 2023. 2

    2023

  46. [54]

    Human motion diffusion as a generative prior

    Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as a generative prior. In ICLR, 2024. 3

    2024

  47. [55]

    Sigurdsson, G ¨ul Varol, Xiaolong Wang, Ivan Laptev, Ali Farhadi, and Abhinav Gupta

    Gunnar A. Sigurdsson, G ¨ul Varol, Xiaolong Wang, Ivan Laptev, Ali Farhadi, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity under- standing. In ECCV, 2016. 3

    2016

  48. [56]

    Grab: A dataset of whole-body human grasp- ing of objects

    Omid Taheri, Nima Ghorbani, Michael J Black, and Dim- itrios Tzionas. Grab: A dataset of whole-body human grasp- ing of objects. In ECCV. Springer, 2020. 3

    2020

  49. [57]

    Humanmimic: Learning natural locomo- tion and transitions for humanoid robot via wasserstein ad- versarial imitation

    Annan Tang, Takuma Hiraoka, Naoki Hiraoka, Fan Shi, Kento Kawaharazuka, Kunio Kojima, Kei Okada, and Masayuki Inaba. Humanmimic: Learning natural locomo- tion and transitions for humanoid robot via wasserstein ad- versarial imitation. In ICRA. IEEE, 2024. 2

    2024

  50. [58]

    Calm: Conditional adversar- ial latent models for directable virtual characters

    Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, and Xue Bin Peng. Calm: Conditional adversar- ial latent models for directable virtual characters. In ACM SIGGRAPH 2023 Conference Proceedings, 2023. 3

    2023

  51. [59]

    Human motion diffu- sion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffu- sion model. In ICLR, 2023. 6

    2023

  52. [60]

    Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H. Bermano. Human motion diffusion model. In ICLR, 2023. 3

    2023

  53. [61]

    Aist dance video database: Multi-genre, multi-dancer, and multi-camera database for dance informa- tion processing

    Shuhei Tsuchida, Satoru Fukayama, Masahiro Hamasaki, and Masataka Goto. Aist dance video database: Multi-genre, multi-dancer, and multi-camera database for dance informa- tion processing. In ISMIR, 2019. 3

    2019

  54. [62]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS, 2017. 5, 6

    2017

  55. [63]

    Attention is all you need

    A Vaswani. Attention is all you need. In NeurIPS, 2017. 5

    2017

  56. [64]

    A scalable approach to control diverse behaviors for physically simulated characters

    Jungdam Won, Deepak Gopinath, and Jessica Hodgins. A scalable approach to control diverse behaviors for physically simulated characters. TOG, 39(4), 2020. 3

    2020

  57. [65]

    Masked visual pre-training for motor control

    Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173, 2022. 2

    2022 arXiv

  58. [66]

    Omnicontrol: Control any joint at any time for human motion generation

    Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. Omnicontrol: Control any joint at any time for human motion generation. In ICLR, 2023. 3

    2023

  59. [67]

    Flow as the cross-domain manipulation interface

    Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gor- don Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface. arXiv preprint arXiv:2407.15208, 2024. 1, 2

    2024 arXiv

  60. [68]

    Learning interactive real-world simulators

    Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In ICLR, 2024. 1, 2

    2024

  61. [69]

    General flow as foundation affordance for scalable robot learning

    Chengbo Yuan, Chuan Wen, Tong Zhang, and Yang Gao. General flow as foundation affordance for scalable robot learning. arXiv preprint arXiv:2401.11439, 2024. 1, 2

    2024 arXiv

  62. [70]

    Physdiff: Physics-guided human motion diffusion model

    Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In ICCV, 2023. 3

    2023

  63. [71]

    Generalizable humanoid manipulation with improved 3d diffusion policies

    Yanjie Ze, Zixuan Chen, Wenhao Wang, Tianyi Chen, Xialin He, Ying Yuan, Xue Bin Peng, and Jiajun Wu. Generalizable humanoid manipulation with improved 3d diffusion policies. arXiv preprint arXiv:2410.10803, 2024. 2

    2024 arXiv

  64. [72]

    Generating human motion from textual descriptions with discrete representations

    Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In CVPR, 2023. 3, 9

    2023

  65. [73]

    Generating human motion from textual descriptions with discrete representations

    Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In CVPR, 2023. 6

    2023

  66. [74]

    Motiondif- fuse: Text-driven human motion generation with diffusion model

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondif- fuse: Text-driven human motion generation with diffusion model. PAMI, 2024. 3

    2024

  67. [75]

    single person

    Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Marc Pollefeys, Federica Bogo, and Siyu Tang. Ego- body: Human body shape and motion of interacting people from head-mounted devices. In ECCV. Springer, 2022. 3 Appendix A . Ethics Statement 1 B . Details on Humanoid-...

    2022

  68. [76]

    Do not describe the appearance of the human

  69. [77]

    a man/woman doing something [adverb]

    You must at least answer “a man/woman doing something [adverb]”

  70. [78]

    If applicable, you should describe the [item] the human is interacting with, the [body part] the hu- man is using, or the [location] the human is in

  71. [79]

    in the video

    Your answer must be within one sentence, and do not begin with “in the video”. Please describe what the human is doing in the video in one sentence. For video captioning, we implemented a video caption- ing pipeline using Video LLaMA [10], with a video pro- cessing framework w...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.