Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

LangWBC: Language-directed Humanoid Whole-Body Control via End-to-end Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a single neural network can turn natural-language commands into agile whole-body actions on a real humanoid, using a Conditional Variational Autoencoder to align language and motion in one latent space.

desk verdict A real system result—closed-loop language-to-action on a humanoid via a distilled CVAE—backed by believable hardware videos, but the quantitative evaluation is thin and the latent-space smoothness claim is more asserted than shown. read the letter →

arxiv 2504.21738 v1 pith:4CRMX2JK submitted 2025-04-30 cs.RO

Yiyang Shao , Xiaoyu Huang , Bike Zhang , Qiayuan Liao , Yuman Gao , Yufeng Chi , Zhongyu Li , Sophia Shao
show 1 more author
Koushil Sreenath
This is my paper · ORCID
classification cs.RO
keywords language-directedcontrolwhole-bodyhumanoidrobotsconditionalvariationalautoencoderpolicydistillationreinforcementlearningnaturallanguagecommandssim-to-realtransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a humanoid robot can be controlled directly by natural language through a single end-to-end policy, without a separate motion-generation module and tracking controller. The authors train a motion-tracking teacher policy with reinforcement learning on retargeted motion-capture clips, then distill it into a student policy that maps text embeddings and two-second proprioceptive histories to joint actions through a Conditional Variational Autoencoder. In real-robot trials the policy executes agile and expressive motions, recovers from pushes and kicks, switches smoothly between motions, and answers paraphrased commands. The broader claim is that a structured latent space shared by language and action is what makes language-to-motion control generalizable enough for real hardware.

What carries the argument

The load-bearing object is the CVAE latent space. It is built by a student policy whose encoder maps the concatenation of a text embedding and a two-second history of proprioception to a mean and diagonal-covariance Gaussian; the decoder maps the sampled latent code plus the current observation to desired joint positions. This structured latent space is what is claimed to align language with motion dynamics, producing clusters for similar motions, left/right mirror symmetry, a common standing-pose region near the origin, smooth transitions, and meaningful interpolation. The training machinery around it is reinforcement-learning teacher distillation with a relative-displacement tracking objective and DAgger-style iterative data collection.

What would settle it

Present the policy with a fixed set of fifty held-out paraphrases spanning the paper's three semantic-distance categories and measure the fraction that maintain balance for a full rollout and match reference keypoints; then take twenty random latent interpolations between walking and side-stepping and count how many produce stable, non-jittery gaits. A high failure rate on either test would directly contradict the claimed latent-space smoothness and generalization.

Watch

Extended reading notes

Core claim

The central discovery, as the paper states it, is that a Conditional Variational Autoencoder can serve as the student policy in a teacher-student distillation pipeline and produce a unified latent space in which text embeddings and motor actions jointly live. The encoder takes the concatenation of a text embedding and a 20-step history of proprioceptive observations, outputs a Gaussian posterior, and the decoder maps a latent sample to 27 joint-position actions. Because semantically similar commands cluster in this latent space, the policy generalizes to unseen phrasings that a plain MLP baseline handles worse, transitions between agile motions in a single rollout, and interpolating two latent codes yields a novel, stable walking-sideways motion that was not in the training distribution. The paper also reports that the distilled student transfers zero-shot to real hardware.

Load-bearing premise

The load-bearing premise is that the CVAE latent space is smooth and semantically aligned: text embeddings of paraphrases must land near their training counterparts, and interpolated latent codes must decode to stable, executable actions, or the generalization and novel-motion claims collapse.

Editorial extensions

If this is right

  • A single distilled policy can act as a closed-loop text-to-action controller on real hardware, eliminating the need for a separate kinematic motion generator and tracking controller.
  • Because transitions happen within one policy, language-commanded sequences such as walking, running, stopping, and waving can run continuously without resets.
  • The structured latent space gives the policy a route to unseen commands: paraphrases and semantically nearby phrasings decode to higher-quality motions than a plain MLP student achieves.
  • Latent interpolation can synthesize novel, stable whole-body motions that were not present in the training data.
  • Wiring the policy to a large language model lets an abstract social instruction be decomposed into timed motion primitives and executed on the robot.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the latent-space smoothness claim scales with dataset size, then adding more mocap-and-text data should extend the same architecture toward a broad language-conditioned motion foundation model rather than a fixed skill set; this is an extrapolation the paper does not itself demonstrate.
  • The generalization mechanism is partly delegated to the text encoder: any improvement in the embedding model could shift where unseen commands land, so evaluating with a different text encoder would isolate how much robustness comes from the CVAE versus the encoder.
  • A more expressive generative model, such as a diffusion decoder, might reduce the sim-to-real gap the authors acknowledge, since the CVAE's limited expressiveness is their stated bottleneck.
  • The interpolation property suggests a practical motion-morphing interface, where users could blend two command embeddings to get a hybrid gait, but the paper only shows one interpolated example, so a systematic study of many pairs would be needed to confirm it.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents LangWBC, a framework for language-directed whole-body control of a Unitree G1 humanoid. The method first trains a motion-tracking teacher policy via reinforcement learning on retargeted MoCap data, then distills it into a CVAE-based student policy through DAgger behavior cloning, with CLIP text embeddings and a history of proprioceptive observations as inputs and joint position targets as outputs. The authors claim zero-shot sim-to-real transfer, execution of diverse motions, generalization to unseen paraphrases, smooth transitions between motions, latent-space interpolation producing novel motions, robustness to disturbances, and LLM-guided composition of motion sequences. Experiments include real-hardware demos, a t-SNE latent-space visualization, an unseen-command comparison against a CLIP+MLP baseline, an interpolation comparison, simulation ablations, and an LLM integration example.

Significance. If the central claims are substantiated, this is an important step toward closed-loop language-to-action control on real humanoid hardware with a single distilled policy, going beyond hierarchical motion-generation-plus-tracking approaches and open-loop text-to-action systems. The paper's strengths are the clear two-stage pipeline, the real-hardware zero-shot deployment evidence, the inclusion of an MLP baseline, and the explicit ablation list. However, the quantitative support is thin: most headline claims rest on video demos, one t-SNE plot, and a small set of single-number metrics without error bars or significance tests. The load-bearing assumption that the CVAE latent space is smooth, continuous, and semantically action-relevant is asserted more than demonstrated. These issues are fixable with additional experiments and statistical reporting, so the manuscript is a plausible major-revision candidate rather than a rejection.

major comments (5)
  1. [Section IV-C, Table II] The generalization claim to unseen commands is not statistically supported. Table II reports a single Motion Quality value per condition with no seeds, no confidence intervals, and no significance test across only 15 unseen commands. The conclusion that 'CLIP+CV AE consistently produces higher-quality motions' is therefore stronger than the evidence. Please report means and variances over multiple training seeds, specify the exact list of the 15 commands, and run a paired significance test. In addition, the Motion Quality Metric is defined only parenthetically as a weighted sum of keypoint and joint errors normalized by an exponential; since these are the same families of quantities used in the teacher reward (Table I) and the student imitation loss (Eq. 8), the metric partly measures imitation fidelity rather than task success or language alignment, which should be acknowledged and, ideally, supplemented with a task-level or human-evaluation metric.
  2. [Section III-B, IV-B, IV-E] The core structural claim—that the CVAE latent space is smooth and action-relevant, enabling generalization, transitions, and novel-motion synthesis—rests on the KL term in Eq. (8), one t-SNE plot of nine motions (Fig. 5), and one interpolated rollout (Figs. 8 and 9). Those do not establish that interpolated latent codes decode to stable, executable actions or that paraphrase embeddings land near their training counterparts. Please provide quantitative evidence: for example, success rates over many interpolated command pairs, a Lipschitz-style measure of decoding stability under latent perturbations, and multiple t-SNE runs with quantitative cluster-quality metrics. Without such evidence, the transition, generalization, and novel-motion claims are anecdotal.
  3. [Section IV-D, Figs. 1 and 7] The smooth-transition capability is demonstrated only through video stills and qualitative descriptions such as 'seamlessly transitions' and 'without resets.' There is no quantitative transition metric, no success rate over repeated transition trials, and no comparison with a baseline that executes the same command sequence with a hierarchical motion generator plus tracker or with resets between clips. Please define a transition-smoothness or transition-success metric (e.g., joint velocity discontinuities, foot contact stability, or completion rate under time limits) and report it for the proposed policy and at least one baseline, or explicitly reframe this contribution as a qualitative demonstration.
  4. [Section III-B, Eqs. (8) and (9)] The relative-tracking objective in Eq. (9) is not connected to the student loss in Eq. (8). The paper states that 'the robot's tracking objective becomes' minimizing the displacement error, but neither Eq. (8), the DAgger loss, nor the policy update step indicates how this term enters the optimization, how it is weighted, or how it interacts with the imitation loss and KL term. This also makes the 'No-Rel' ablation in Table III hard to interpret, since the reader cannot tell what exactly was removed. Please specify the complete student objective and the optimization procedure.
  5. [Table III, Section IV-F] The ablation metrics are single numbers reported after 10k iterations, without multiple seeds, variance, or confidence intervals. The Stability metric is described as 'success rate over 1000 steps under perturbations' but the perturbation distribution and success criteria are not specified. Since the ablation is used to justify all three core design choices, please report mean and standard deviation across seeds, specify the exact perturbation protocol, and provide a significance test for the pairwise differences claimed in the text.
minor comments (5)
  1. [Fig. 5] t-SNE is stochastic and can create apparent clusters even without semantic structure; please report multiple t-SNE runs with different random seeds or a quantitative cluster-quality measure, rather than relying on a single plot.
  2. [Table II footnote] The Motion Quality Metric is called a weighted sum but the expressions in Table I are exponentials of keypoint and joint errors; please give the exact formula, normalization, and the weights used in Tables II and III.
  3. [Section III-A and Appendix A] The number of future keypoint frames is inconsistent: Section III-A says 'future five-frame keypoint positions' and Appendix A lists '6 frames' for target keypoint positions; please reconcile the notation.
  4. [Section III-B, Eq. (9)] The interval Δt in Δp_t = p_t − p_{t−Δt} is not defined; state whether it is the control timestep, the 10 Hz history sampling interval, or another period.
  5. [Throughout] The paper uses inconsistent spellings 'CV AE' and 'CVAE' (e.g., Fig. 1 caption vs. the text); please standardize the terminology.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the teacher–student derivation is self-contained and the evaluation metric, though related to the teacher reward, is an external kinematic measure.

full rationale

LangWBC's derivation chain is self-contained: a language-agnostic teacher is trained with PPO to track retargeted MoCap (Eqs. 1–3, Table I), and a CVAE student is trained by DAgger behavior cloning to imitate teacher actions from CLIP text embeddings and proprioceptive history (Eqs. 4–8). The paper's capabilities—diverse motions, paraphrase generalization, smooth transitions, and latent interpolation—are emergent properties evaluated by simulation and hardware rollouts, not quantities defined in terms of the training objective. The Motion Quality Metric in Tables II and III is a kinematic tracking score against MoCap references; while it shares terms with the teacher reward, it is an external evaluation measure, not a parameter fitted to the test set, and the student's objective (Eq. 8) is action-matching rather than direct keypoint tracking. No load-bearing self-citation appears: related-work citations to the authors' prior systems are contextual, and the core components (CLIP, CVAE, DAgger, PPO) are externally established. No uniqueness theorem or ansatz is imported from the authors' own prior work. The limitations section explicitly acknowledges the VAE expressiveness gap, further indicating that the claims are empirical rather than definitional. Therefore no circular step is present.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

All central hyperparameters are hand-chosen, and the key behavioral claims rely on the smoothness of the learned latent space and on the fidelity of the simulation-to-real transfer. No new physical entities are introduced; the latent code is a learned representation, not an invented real-world quantity.

free parameters (6)
  • Teacher reward weights (Table I) = e.g., keypoint tracking 1.0, joint tracking 1.0, orientation penalty -10.0, termination -200.0
    Hand-chosen shaping weights determine which MoCap motions are learnable and how the teacher behaves; they indirectly shape all student skills.
  • Symmetry loss weight lambda_sym (Eq. 2) = not stated
    Balances the PPO objective and the mirror-symmetry regularizer; affects teacher sample efficiency and motion naturalness.
  • KL weight lambda_KL (Eq. 8) = not stated
    Balances behavior-cloning fidelity against latent-space regularization; the entire interpolation and smooth-transition claims depend on the latent space being structured, which this weight controls.
  • Latent dimension and encoder/decoder widths = latent 128; MLPs 2048-1024-512 and 512-1024-2048
    Architecture capacity is a hand-chosen hyperparameter that affects how smoothly the latent space can be and whether novel interpolated commands decode to stable actions.
  • Motion Quality metric weights (Tables II, III) = not stated; described as weighted sum of keypoint and joint errors
    The main quantitative evaluation metric is defined ad hoc with unspecified weights, and it closely mirrors the training objective, making the headline numbers hard to interpret.
  • Domain randomization ranges (Table IV) = ranges in table, e.g., friction [0.2,0.6], mass [0.9,1.1]x
    Hand-chosen randomization bounds are the mechanism claimed to enable zero-shot sim-to-real transfer; too narrow and the robot fails on hardware, too wide and the teacher may not learn.
assumptions (6)
  • domain assumption HumanML3D dataset with text annotations, after LM-based retargeting, yields kinematically feasible and physically trackable references for the G1 across the tested command vocabulary.
    Section III-A and IV-A; the entire teacher skill set is trained on this retargeted data, and the diversity of motions claimed depends on it.
  • domain assumption CLIP text encoder embeddings place semantically similar commands close together so the CVAE can map paraphrase variants to nearby latent codes.
    Eq. (4) in Section III-B and the generalization experiment in Section IV-C; the unseen-text generalization claim relies on this property of a frozen pretrained model.
  • domain assumption The CVAE latent space, regularized by a unit Gaussian KL prior, is smooth and semantically structured so that interpolation between latent codes yields stable, executable motions.
    Sections IV-B, IV-D, IV-E; the transition and novel-motion claims rest on this smoothness, which the paper asserts from t-SNE visualization and one interpolation example.
  • domain assumption DAgger-style on-policy data collection keeps the student within the teacher's support, so behavior cloning over teacher actions yields a policy that does not drift.
    Section III-B, training steps 1-5 and Eqs. (8)-(9); the student's quality is wholly dependent on teacher actions being valid for visited states.
  • domain assumption The simulator with domain randomization (Table IV) models the real G1 closely enough for zero-shot sim-to-real transfer of the student policy.
    Section IV-A and Appendix D; the real-world demonstrations validate this for the tested conditions, but it is assumed for all untested conditions.
  • domain assumption The relative displacement tracking objective (Eq. 9) prevents accumulation of drift errors during student training.
    Section III-B step 4; this design choice is asserted to mitigate distribution shift and is ablated in Table III (No-Rel).

how reviews work

0 comments
Cite this review

Pith. "Pith review of LangWBC: Language-directed Humanoid Whole-Body Control via End-to-end Learning." pith.science (2026). https://pith.science/paper/4CRMX2JK

@misc{pith2026250421738,
  author       = {Pith},
  title        = {Pith review of: LangWBC: Language-directed Humanoid Whole-Body Control via End-to-end Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4CRMX2JK}},
  note         = {Machine review of arXiv:2504.21738}
}
read the original abstract

General-purpose humanoid robots are expected to interact intuitively with humans, enabling seamless integration into daily life. Natural language provides the most accessible medium for this purpose. However, translating language into humanoid whole-body motion remains a significant challenge, primarily due to the gap between linguistic understanding and physical actions. In this work, we present an end-to-end, language-directed policy for real-world humanoid whole-body control. Our approach combines reinforcement learning with policy distillation, allowing a single neural network to interpret language commands and execute corresponding physical actions directly. To enhance motion diversity and compositionality, we incorporate a Conditional Variational Autoencoder (CVAE) structure. The resulting policy achieves agile and versatile whole-body behaviors conditioned on language inputs, with smooth transitions between various motions, enabling adaptation to linguistic variations and the emergence of novel motions. We validate the efficacy and generalizability of our method through extensive simulations and real-world experiments, demonstrating robust whole-body control. Please see our website at LangWBC.github.io for more information.

Figures

Figures reproduced from arXiv: 2504.21738 by the authors.

Figure 1
Figure 1. We propose a language-directed humanoid whole-body control framework that translates natural language commands into continuous robot actions [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The Overview of the Training Framework. The training process includes a motion-tracking teacher training phase and a language-directed student training phase. We first retarget the MoCap dataset and train a teacher policy via reinforcement learning. Then, a student policy, leveraging a CVAE architecture, jointly models high-level linguistic instructions and low-level physical actions of the teacher policy in a unifi… view at source ↗
Figure 3
Figure 3. Robustness to External Disturbances. The humanoid robot demonstrates robust stability while executing a hand-waving motion under external perturbations. When subjected to kicks (top row) and pushes (bottom row), the robot maintains balance and continues the commanded motion, showcasing effective disturbance rejection capabilities without interrupting the primary task. We repeat the iterative process, where the stude… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Real World Demonstration. Conditioned on text commands, our framework is able to learn a diverse distribution of whole-body motions in action generation directly, and can be zero-shot deployed on real-world robots. More results are shown in the accompanying video. lang…
Figure 5
Figure 5. Figure 5: t-SNE Analysis of Latent Space. The plot shows 9 motions from 4 categories of motion, as shown in the legend. We see that similar motions (in the same color band) are closer than dissimilar ones. The axes suggest an interpretable structure: lateral symmetry (left/right…
Figure 7
Figure 7. Figure 7: Smooth Transitions between Different Text Commands. The humanoid robot seamlessly executes a sequence of actions: waving its right hand, transitioning into running, coming to a stop, and concluding with another hand wave. The policy demonstrates the ability to handle d…
Figure 8
Figure 8. Figure 8: Interpolation in the Latent Space. The CVAE gives a structured latent space, enabling the policy to generalize to interpolated command. Here, interpolating between walking (Command 1) and side stepping (Command 2) produces walking to the side, a whole-body motion that …
Figure 9
Figure 9. Figure 9: Latent Space Interpolation: CLIP+CVAE vs. CLIP Alone Comparison of motion quality when interpolating between forward and side￾ways walking. The CLIP+CVAE model (left) produces smooth and coherent diagonal walking, while the CLIP-only baseline (right) results in jittery…
Figure 10
Figure 10. Figure 10: LLM-guided Humanoid Motion Sequence. Given the social scenario “There is a friend 3 meters in front”, the LLM decomposes this high￾level instruction into primitive motion commands, which the robot executes by walking forward, stopping, and greeting with a hand wave. G…
Figure 11
Figure 11. Figure 11: Upper-body Motion Examples. Our framework generates diverse upper-body movements including reaching, lifting, and manipulation tasks. The learned motions can be directly deployed on the real robot [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Lower-body Motion Examples. The framework also enables various lower-body movements such as stepping, squatting and balancing. These motions are also successfully transferred to the real robot without additional training [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MRBench: A Comprehensive Benchmark for Human Motion-Text Retrieval

    cs.CV 2026-08 conditional novelty 6.0 of 10

    MRBench is a multi-source, balanced, multi-granular human motion-text retrieval benchmark, and the proposed granularity-aware adapters improve mixed-granularity retrieval without degrading standard-caption retrieval.

  2. ZeroWBC: Learning Natural Whole-Body Humanoid Interaction from Human Egocentric Data

    cs.RO 2026-03 conditional novelty 5.0 of 10

    An open-loop generation-then-tracking system maps one egocentric image plus language into Unitree G1 whole-body interactions using only human egocentric motion data.

Reference graph

Works this paper leans on

44 extracted references · 11 canonical work pages · cited by 2 Pith papers

  1. [1]

    Expressive whole- body control for humanoid robots.arXiv preprint arXiv:2402.16796, 2024

    Xuxin Cheng, Yandong Ji, Junming Chen, Ruihan Yang, Ge Yang, and Xiaolong Wang. Expressive whole- body control for humanoid robots.arXiv preprint arXiv:2402.16796, 2024

    arXiv 2024

  2. [2]

    Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 0(0), 2024

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 0(0), 2024. doi: 10.1177/02783649241273668

    doi:10.1177/02783649241273668 2024

  3. [3]

    Flexible motion in- betweening with diffusion models

    Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, and Michiel van de Panne. Flexible motion in- betweening with diffusion models. InACM SIGGRAPH 2024 Conference Papers, pages 1–9, 2024

    work page 2024

  4. [4]

    Learning vision-based bipedal locomotion for challeng- ing terrain

    Helei Duan, Bikram Pandit, Mohitvishnu S Gadde, Bart Van Marum, Jeremy Dao, Chanho Kim, and Alan Fern. Learning vision-based bipedal locomotion for challeng- ing terrain. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 56–62. IEEE, 2024

    work page 2024

  5. [5]

    Learning multi-modal whole- body control for real-world humanoid robots.arXiv preprint arXiv:2408.07295, 2024

    Pranay Dugar, Aayam Shrestha, Fangzhou Yu, Bart van Marum, and Alan Fern. Learning multi-modal whole- body control for real-world humanoid robots.arXiv preprint arXiv:2408.07295, 2024

    arXiv 2024

  6. [6]

    Humanplus: Humanoid shad- owing and imitation from humans.arXiv preprint arXiv:2406.10454, 2024

    Zipeng Fu, Qingqing Zhao, Qi Wu, Gordon Wet- zstein, and Chelsea Finn. Humanplus: Humanoid shad- owing and imitation from humans.arXiv preprint arXiv:2406.10454, 2024

    arXiv 2024

  7. [7]

    Coohoi: Learning cooperative human- object interaction with manipulated object dynamics

    Jiawei Gao, Ziqin Wang, Zeqi Xiao, Jingbo Wang, Tai Wang, Jinkun Cao, Xiaolin Hu, Si Liu, Jifeng Dai, and Jiangmiao Pang. Coohoi: Learning cooperative human- object interaction with manipulated object dynamics. arXiv preprint arXiv:2406.14558, 2024

    arXiv 2024

  8. [8]

    Advancing humanoid locomotion: Mastering challenging terrains with denoising world model learning.arXiv preprint arXiv:2408.14472, 2024

    Xinyang Gu, Yen-Jen Wang, Xiang Zhu, Chengming Shi, Yanjiang Guo, Yichen Liu, and Jianyu Chen. Advancing humanoid locomotion: Mastering challenging terrains with denoising world model learning.arXiv preprint arXiv:2408.14472, 2024

    arXiv 2024

Show all 44 references
  1. [9]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152–5161, June 2022

    2022

  2. [10]

    Omnih2o: Universal and dexterous human- to-humanoid whole-body teleoperation and learning

    Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous human- to-humanoid whole-body teleoperation and learning. arXiv preprint arXiv:2406.08858, 2024

    2024 arXiv

  3. [11]

    Learning human- to-humanoid real-time whole-body teleoperation

    Tairan He, Zhengyi Luo, Wenli Xiao, Chong Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Learning human- to-humanoid real-time whole-body teleoperation. In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8944–8951. IEEE, 2024

    2024

  4. [12]

    Hover: Versatile neural whole- body controller for humanoid robots.arXiv preprint arXiv:2410.21229, 2024

    Tairan He, Wenli Xiao, Toru Lin, Zhengyi Luo, Zhenjia Xu, Zhenyu Jiang, Jan Kautz, Changliu Liu, Guanya Shi, Xiaolong Wang, et al. Hover: Versatile neural whole- body controller for humanoid robots.arXiv preprint arXiv:2410.21229, 2024

    arXiv

  5. [13]

    Diffuseloco: Real-time legged locomo- tion control with diffusion from offline datasets.arXiv preprint arXiv:2404.19264, 2024

    Xiaoyu Huang, Yufeng Chi, Ruofeng Wang, Zhongyu Li, Xue Bin Peng, Sophia Shao, Borivoje Nikolic, and Koushil Sreenath. Diffuseloco: Real-time legged locomo- tion control with diffusion from offline datasets.arXiv preprint arXiv:2404.19264, 2024

    2024 arXiv

  6. [14]

    Diffuse-cloc: Guided diffusion for physics-based character look-ahead control

    Xiaoyu Huang, Takara Truong, Yunbo Zhang, Fangzhou Yu, Jean Pierre Sleiman, Jessica Hodgins, Koushil Sreenath, and Farbod Farshidian. Diffuse-cloc: Guided diffusion for physics-based character look-ahead control. arXiv preprint arXiv:2503.11801, 2025

    2025 arXiv

  7. [15]

    Ex- body2: Advanced expressive humanoid whole-body con- trol.arXiv preprint arXiv:2412.13196, 2024

    Mazeyu Ji, Xuanbin Peng, Fangchen Liu, Jialong Li, Ge Yang, Xuxin Cheng, and Xiaolong Wang. Ex- body2: Advanced expressive humanoid whole-body con- trol.arXiv preprint arXiv:2412.13196, 2024

    2024 arXiv

  8. [16]

    Superpadl: Scaling language-directed physics- based control with progressive supervised distillation

    Jordan Juravsky, Yunrong Guo, Sanja Fidler, and Xue Bin Peng. Superpadl: Scaling language-directed physics- based control with progressive supervised distillation. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024

    2024

  9. [17]

    Guided motion diffusion for controllable human motion synthesis

    Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakorn, and Siyu Tang. Guided motion diffusion for controllable human motion synthesis. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 2151–2162, 2023

    2023

  10. [18]

    Learning quadrupedal locomotion over challenging terrain.Science robotics, 5 (47):eabc5986, 2020

    Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain.Science robotics, 5 (47):eabc5986, 2020

    2020

  11. [19]

    Reinforcement learning for robust parameterized loco- motion control of bipedal robots

    Zhongyu Li, Xuxin Cheng, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, and Koushil Sreenath. Reinforcement learning for robust parameterized loco- motion control of bipedal robots. In2021 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 2811...

    2021

  12. [20]

    Rein- forcement learning for versatile, dynamic, and robust bipedal locomotion control.The International Jour- nal of Robotics Research, 44(5):840–888, 2024

    Zhongyu Li, Xue Bin Peng, Pieter Abbeel, Sergey Levine, Glen Berseth, and Koushil Sreenath. Rein- forcement learning for versatile, dynamic, and robust bipedal locomotion control.The International Jour- nal of Robotics Research, 44(5):840–888, 2024. doi: 10.1177/02783649241285161

    2024 doi

  13. [21]

    Berkeley hu- manoid: A research platform for learning-based control

    Qiayuan Liao, Bike Zhang, Xuanyu Huang, Xiaoyu Huang, Zhongyu Li, and Koushil Sreenath. Berkeley hu- manoid: A research platform for learning-based control. arXiv preprint arXiv:2407.21781, 2024

    2024 arXiv

  14. [22]

    Learning hu- manoid locomotion with perceptive internal model.arXiv preprint arXiv:2411.14386, 2024

    Junfeng Long, Junli Ren, Moji Shi, Zirui Wang, Tao Huang, Ping Luo, and Jiangmiao Pang. Learning hu- manoid locomotion with perceptive internal model.arXiv preprint arXiv:2411.14386, 2024

    2024 arXiv

  15. [23]

    Perpetual humanoid control for real-time simulated avatars

    Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 10895–10904, 2023

    2023

  16. [24]

    Grasping di- verse objects with simulated humanoids.arXiv preprint arXiv:2407.11385, 2024

    Zhengyi Luo, Jinkun Cao, Sammy Christen, Alexander Winkler, Kris Kitani, and Weipeng Xu. Grasping di- verse objects with simulated humanoids.arXiv preprint arXiv:2407.11385, 2024

    2024 arXiv

  17. [25]

    Learning from massive human videos for universal humanoid pose control.arXiv preprint arXiv:2412.14172, 2024

    Jiageng Mao, Siheng Zhao, Siqi Song, Tianheng Shi, Jun- jie Ye, Mingtong Zhang, Haoran Geng, Jitendra Malik, Vitor Guizilini, and Yue Wang. Learning from massive human videos for universal humanoid pose control.arXiv preprint arXiv:2412.14172, 2024

    2024 arXiv

  18. [26]

    An algorithm for least-squares estimation of nonlinear parameters.Journal of the society for Industrial and Applied Mathematics, 11(2):431–441, 1963

    Donald W Marquardt. An algorithm for least-squares estimation of nonlinear parameters.Journal of the society for Industrial and Applied Mathematics, 11(2):431–441, 1963

    1963

  19. [27]

    Orbit: A unified simulation framework for interactive robot learning environments.IEEE Robotics and Au- tomation Letters, 8(6):3740–3747, 2023

    Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A unified simulation framework for interactive robot learni...

    2023

  20. [28]

    Deepmimic: Example-guided deep re- inforcement learning of physics-based character skills

    Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example-guided deep re- inforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 37(4):1–14, 2018

    2018

  21. [29]

    Amp: Adversarial motion priors for stylized physics-based character control.ACM Transac- tions on Graphics (ToG), 40(4):1–20, 2021

    Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control.ACM Transac- tions on Graphics (ToG), 40(4):1–20, 2021

    2021

  22. [30]

    Learning transferable visual models from natural lan- guage supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural lan- guage supervision. InInternational conference on ma- chine learning,...

    2021

  23. [31]

    Real-world hu- manoid locomotion with reinforcement learning.Science Robotics, 9(89):eadi9579, 2024

    Ilija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, and Koushil Sreenath. Real-world hu- manoid locomotion with reinforcement learning.Science Robotics, 9(89):eadi9579, 2024. doi: 10.1126/scirobotics. adi9579

    2024 doi

  24. [32]

    A reduction of imitation learning and structured prediction to no-regret online learning

    St ´ephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InProceedings of the fourteenth international conference on artificial intelli- gence and statistics, pages 627–635. JMLR Workshop and ...

    2011

  25. [33]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  26. [34]

    Robot motion diffusion model: Motion generation for robotic characters

    Agon Serifi, Ruben Grandia, Espen Knoop, Markus Gross, and Moritz B ¨acher. Robot motion diffusion model: Motion generation for robotic characters. In SIGGRAPH Asia 2024 Conference Papers, pages 1–9, 2024

    2024

  27. [35]

    Interactive character control with auto-regressive motion diffusion models.ACM Transactions on Graphics (TOG), 43(4):1–14, 2024

    Yi Shi, Jingbo Wang, Xuekun Jiang, Bingkun Lin, Bo Dai, and Xue Bin Peng. Interactive character control with auto-regressive motion diffusion models.ACM Transactions on Graphics (TOG), 43(4):1–14, 2024

    2024

  28. [36]

    Learning structured output representation using deep conditional generative models.Advances in neural information processing systems, 28, 2015

    Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models.Advances in neural information processing systems, 28, 2015

    2015

  29. [37]

    Humanmimic: Learning natural loco- motion and transitions for humanoid robot via wasser- stein adversarial imitation

    Annan Tang, Takuma Hiraoka, Naoki Hiraoka, Fan Shi, Kento Kawaharazuka, Kunio Kojima, Kei Okada, and Masayuki Inaba. Humanmimic: Learning natural loco- motion and transitions for humanoid robot via wasser- stein adversarial imitation. In2024 IEEE International Conference on Ro...

    2024

  30. [38]

    Calm: Conditional adversarial latent models for directable virtual charac- ters

    Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, and Xue Bin Peng. Calm: Conditional adversarial latent models for directable virtual charac- ters. InACM SIGGRAPH 2023 Conference Proceedings, pages 1–9, 2023

    2023

  31. [39]

    Maskedmimic: Unified physics- based character control through masked motion inpaint- ing.ACM Transactions on Graphics (TOG), 43(6):1–21, 2024

    Chen Tessler, Yunrong Guo, Ofir Nabati, Gal Chechik, and Xue Bin Peng. Maskedmimic: Unified physics- based character control through masked motion inpaint- ing.ACM Transactions on Graphics (TOG), 43(6):1–21, 2024

    2024

  32. [40]

    Human motion dif- fusion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion dif- fusion model. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  33. [41]

    Equiv- ariant diffusion policy

    Dian Wang, Stephen Hart, David Surovik, Tarik Ke- lestemur, Haojie Huang, Haibo Zhao, Mark Yeatman, Jiuguang Wang, Robin Walters, and Robert Platt. Equiv- ariant diffusion policy. In8th Annual Conference on Robot Learning, 2024

    2024

  34. [42]

    Wococo: Learning whole-body humanoid control with sequential contacts.arXiv preprint arXiv:2406.06005, 2024

    Chong Zhang, Wenli Xiao, Tairan He, and Guanya Shi. Wococo: Learning whole-body humanoid control with sequential contacts.arXiv preprint arXiv:2406.06005, 2024

    2024 arXiv

  35. [43]

    Simulation and retargeting of complex multi-character interactions

    Yunbo Zhang, Deepak Gopinath, Yuting Ye, Jessica Hodgins, Greg Turk, and Jungdam Won. Simulation and retargeting of complex multi-character interactions. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–11, 2023. APPENDIX A. Teacher Policy Input State The input of the teac...

    2023

  36. [44]

    •Target Keypoint Positions(6 frames×7 links×3 dimensions = 126): Future desired positions of keypoints over six time steps, providing a trajectory for the robot to follow

    body link mass (28 dims); 4) external force (3 dims); and 5) external torque (3 dims). •Target Keypoint Positions(6 frames×7 links×3 dimensions = 126): Future desired positions of keypoints over six time steps, providing a trajectory for the robot to follow. •Degree-of-Freedom...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.