{"id":"8cb0c756-0ea5-428d-8cad-53f9fdfa8692","arxiv_id":"2412.14172","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Humanoid-X contributes 163,800 text-annotated motion clips retargeted from human videos into humanoid robot poses, and UH-1 is an autoregressive transformer that maps text instructions to humanoid actions.","lead":"This paper introduces Humanoid-X, a dataset of over 20 million humanoid-robot pose frames derived from internet videos of people, together with UH-1, a transformer model that turns text commands into robot motions. The authors report strong simulated and real-robot results, but the evaluation benchmark is built with their own retargeting pipeline and the data and code are not yet public.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Action labels Arobot come from an RL policy trained only on curated CMU MoCap, with no per-sample tracking validation; if the policy fails on video-derived keypoints, much of the 20M-sample training signal is mislabeled.","rationale":"The paper's direction is plausible and the real-robot demonstrations are encouraging, but the conditionality of the reader's verdict is well placed. My stress-test converges on the same load-bearing point as the reader's weakest assumption: the supervised training signal Arobot is generated by an RL policy whose training distribution is a clean subset of CMU MoCap, not the noisy, diverse video-derived keypoints that constitute Humanoid-X. No per-sample validation or quality filtering is reported, so the 20M action labels are unverified. This is the single point where the central claim could fail even if every other component of the pipeline works, because UH-1 is trained end-to-end on those labels. The HumanoidML3D benchmark is built with the same retargeting method, so it cannot independently audit label quality, and the real-robot set of 12 mostly upper-body commands is too narrow to establish 'universal' generalization. The proposed test—computing per-sample tracking success and retraining on the validated subset—would settle whether the concern lands. Since the reader already recommended conditional acceptance, my analysis does not shift the verdict; it strengthens the conditions under which the paper's central claim would be credible, namely release of the RL policy and per-sample quality statistics.","tokens_in":24950,"tokens_out":4142,"duration_ms":38504,"concrete_test":"Reproduce or obtain the goal-conditioned RL policy described in Appendix B.6, apply it to a stratified random sample of 1,000 Humanoid-X clips spanning all data sources, and compute per-sample tracking success under a fixed threshold: no fall, final upper-body keypoint error below a stated bound, and root tracking reward within the 5th percentile of the policy's held-out CMU MoCap performance. Report the distribution and the fraction of samples that fail. Then retrain UH-1 on the successfully tracked subset versus the full dataset; if the FID and diversity metrics do not change materially, the mislabeled-sample concern is mitigated, but if a substantial fraction (e.g., >5%) of clips fail and removing them changes the scaling results, the central claim requires qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 1: 'training on massive video data enables a level of generalizability ... previously unattainable') depends on Humanoid-X's Arobot being physically deployable actions for all 163,800 samples. Appendix B.6 states that the goal-conditioned RL policy is trained only on a selectively used portion of the CMU MoCap dataset in AMASS, with motions involving interactions, heavy objects, and rough terrain excluded. That policy is then applied to retargeted keypoints from noisy in-the-wild videos spanning Kinetics, YouTube, and other sources. The paper reports no per-sample tracking success, episode return, fall rate, keypoint-error threshold, or other quality filter for the generated Arobot sequences, and Section 3.2 presents the RL labeling step as if it succeeds uniformly. If the MoCap-trained policy tracks poorly on out-of-distribution video-derived motions, the resulting Arobot sequences used as tokenizer and Transformer targets are not valid humanoid actions. UH-1 is trained on those labels, so the scaling gains in Fig. 7 and the HumanoidML3D improvements in Table 2 could reflect learning from a mixture of valid and degenerate actions rather than 'previously unattainable' generalization. The HumanoidML3D benchmark is constructed with the same retargeting pipeline, so it cannot independently validate label quality, and the 12-command real-robot evaluation is too narrow and mostly upper-body to certify the full dataset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents two artifacts: Humanoid-X, a 163,800-sample dataset with over 20 million frames of text-labeled humanoid keypoints and target joint-position actions mined from videos, and UH-1, a VQ-VAE-tokenized autoregressive Transformer that maps text instructions to humanoid keypoints or actions. The data pipeline consists of clip mining, video captioning, SMPL pose estimation, retargeting to a humanoid skeleton, and goal-conditioned RL to produce deployable actions. The experiments report FID, diversity, and retrieval metrics on a retargeted HumanoidML3D benchmark, scaling and ablation curves, and success rates on 12 real-robot commands.","tokens_in":25416,"tokens_out":9742,"duration_ms":80501,"significance":"The scale of Humanoid-X and the attempt to turn unlabeled Internet video into physically deployable humanoid action labels are timely and potentially valuable; the controlled codebook-size and architecture ablations and the real-robot demonstrations are useful contributions. However, the evidence for the central generalization claim is currently incomplete: the RL-generated action labels are not validated per sample, and the main benchmark is generated with the paper's own retargeting pipeline. If the label-quality and benchmark concerns are addressed, this could be an important dataset and model; at present the load-bearing evidence is not yet convincing.","major_comments":[{"comment":"The action labels Arobot are produced by a goal-conditioned RL policy trained only on a curated subset of CMU MoCap from AMASS, with interactions, heavy objects, and rough terrain excluded, and then applied to keypoints retargeted from noisy in-the-wild videos. The paper reports no per-sample tracking success, episode return, fall rate, keypoint-error threshold, or any other quality filter for the resulting 163,800 Arobot sequences. Because Arobot is the training target of the action tokenizer (Eq. 10) and of the autoregressive objective (Eq. 13), a substantial fraction of degenerate labels would contaminate the tokenizer and the transformer, and the scaling gains in Fig. 7 and Table 2 could partly reflect learning from invalid actions. I ask for a quantitative label-quality study, e.g., the fraction of clips whose policy rollout satisfies a motion-tracking reward or fall-avoidance threshold, and a corresponding filtering rule in the dataset release.","section":"§3.2 / Appendix B.6"},{"comment":"The HumanoidML3D benchmark is created by the authors by retargeting HumanML3D with the same motion-retargeting pipeline used to build Humanoid-X. The comparisons against MDM and T2M-GPT therefore measure, at least in part, how well each method matches the authors' retargeting transformation rather than how well it controls a humanoid robot. The paper should add an independent evaluation, for example deploying the generated actions in a physics simulator with fixed balance and tracking rewards, or comparing with a whole-body humanoid control baseline under identical retargeting. Without such validation, the 'level of generalizability ... previously unattainable' claim in Section 1 is not fully supported.","section":"§5.1 / Table 1"},{"comment":"Table 2 claims that pre-training on Humanoid-X 'greatly improves' FID from 0.445 to 0.379. The reported 95% confidence intervals (0.445±0.078 and 0.379±0.046) overlap, so the improvement is not established without a paired test or a confidence interval on the difference. Please report paired bootstrap significance tests, or a repeated-evaluation protocol with fixed seeds, before using this result to support the scaling claim.","section":"§5.2 / Table 2"},{"comment":"The real-robot evaluation covers 12 commands, most of which are upper-body poses, and for text-to-action open-loop control the lower body is handled by a pre-trained locomotion policy (Section 5.3). Success is determined by a majority vote of five human evaluators (Appendix E.1), and each command is tested only 10 times. These results are encouraging evidence that the selected commands are deployable, but they do not certify the full 20M-sample action space or support the 'universal' scope implied by the paper's framing. I recommend either broadening the command set (including whole-body and locomotion tasks) or explicitly limiting the generalizability claim to the tested upper-body distribution.","section":"§5.3 / Table 3"}],"minor_comments":[{"comment":"In the sentence 'we first tokenize each Arobot into Ztoken using Eq. 3', the reference should be to Eq. (10), not Eq. (3), which defines forward kinematics.","section":"§4 (after Eq. 13)"},{"comment":"References [72] and [73] are the same CVPR 2023 paper by Zhang et al.; the duplicate should be removed and the citations disambiguated.","section":"References [72] and [73]"},{"comment":"The caption and surrounding text do not state whether the FID and diversity values are computed on a held-out subset of Humanoid-X or on HumanoidML3D after fine-tuning; please clarify the evaluation protocol so the numbers can be related to Tables 1 and 2.","section":"Figure 7"},{"comment":"The success rates are point estimates from 10 trials; please report binomial confidence intervals or the raw counts, and specify how the five evaluators were recruited or whether they were blinded to the commanded instruction.","section":"Table 3 / Appendix E.1"},{"comment":"The release statement says data and code will be released 'in the future' but gives no timeline or plan; for a dataset-centric paper, a concrete release protocol (including split definitions and source-video handling) should be described.","section":"Appendix C.3"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is well grounded in Appendix B.6 and is the main reason I cannot recommend acceptance. The issue is fixable: the authors can add per-sample RL-label validation, paired significance tests, and an independent benchmark, so major revision rather than rejection is appropriate. I would also ask the editor to require a concrete data-release plan."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this is a serious dataset-and-pipeline paper that should be reviewed, but its central generalization claim outruns the evidence. The paper builds a 20M-frame text-annotated humanoid pose dataset (Humanoid-X) from internet video and trains a text-to-action transformer (UH-1). That is genuinely new at this scale, and the controlled ablations — pretraining gains, scaling curves, tokenizer codebook size, transformer vs. diffusion — are internally consistent. The real-robot results on 12 commands are encouraging, even if narrow and mostly upper-body.\n\nWhat the paper does well: the pipeline is thoughtful. They mine videos, caption them, estimate SMPL via VIBE, retarget to a Unitree H1-2, and train a goal-conditioned RL policy to output deployable DoF actions. The dataset statistics are thorough, the tokenizer with temporal downsampling and a first-order reconstruction loss is a sensible adaptation, and the comparison against MDM and T2M-GPT is fair in design. The pretraining gain on HumanoidML3D (FID 0.445 to 0.379) is credible as an internal result.\n\nThe soft spots are the ones you would expect. First, the main quantitative benchmark (HumanoidML3D) is generated by the same retargeting pipeline used to build Humanoid-X, so the pretraining improvement partly measures how well UH-1 matches the authors' own transformation, not generalizability to independently produced humanoid action data. Second, the stress-test note is right about the RL labeling: Appendix B.6 says the goal-conditioned policy is trained only on a curated subset of CMU MoCap from AMASS, with interactions and heavy objects excluded, and it is then applied to noisy in-the-wild keypoints. No per-sample tracking success, episode return, or quality filter is reported for the 163,800 samples. If the policy tracks poorly on video-derived keypoints, a chunk of the 20M samples are mislabeled as deployable actions. That is a load-bearing gap for the 'universal' claim.\n\nThe real-robot experiments partially answer this — 90 to 100 percent success on 12 commands is real evidence that at least some generated actions are deployable. But 12 mostly upper-body commands do not certify the full dataset, and the \"previously unattainable\" phrasing in Section 1 overclaims.\n\nBottom line: the dataset and pipeline are a legitimate contribution worth refereeing. The authors should release data and code, report action-label coverage statistics from the RL policy, and evaluate against an independent humanoid-action target. If they do that, the central claims could become well supported. I would send it to review with a request for major revision rather than desk reject.","headline":"A serious dataset-and-pipeline paper with a strong real-robot demo, but the main benchmark is built by the same retargeting pipeline and the RL action labels lack per-sample validation, so the 'universal' claim needs support.","tokens_in":25834,"tokens_out":3347,"would_cite":true,"duration_ms":26084,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that massive human-video data, automatically converted into text-action pairs, gives a single model text-conditioned humanoid pose control that generalizes to new commands and deploys on a real robot at near-100% success.","keywords":["humanoid robot control","text-conditioned motion generation","large-scale video dataset","motion retargeting","reinforcement learning","action tokenization","Transformer","real-robot deployment"],"falsifier":"Run the goal-conditioned policy of Appendix B.6 on a random sample of Humanoid-X keypoint sequences and record the fraction of episodes that keep the robot stable and within a defined tracking error (for example, root position error below a threshold with no fall); if a substantial fraction of sequences cannot be tracked, then a large share of the dataset's physically deployable action labels are wrong, which would invalidate the claim that UH-1 learns deployable actions from massive video data.","tokens_in":24748,"feed_emoji":"🤖","tokens_out":6703,"duration_ms":54530,"temperature":0.7,"pith_summary":"This paper sets out to show that the bottleneck in humanoid robotics is data, not architecture. It builds Humanoid-X, a dataset of 163,800 video clips (over 20 million frames, about 240 hours) that are automatically converted into text-action pairs: each clip is captioned, reconstructed as a 3D human pose, retargeted onto a humanoid skeleton, and labeled with physically deployable joint actions by a goal-conditioned reinforcement-learning policy. On top of this dataset it trains UH-1, a Transformer that takes a text instruction and auto-regressively generates action tokens. The paper's central assertion is that training on this massive video-derived corpus achieves a level of generalization in humanoid control that was previously unattainable, supporting this with improved FID on the HumanoidML3D benchmark and near-100 percent success on 12 real-robot commands.","feed_headline":"20M video poses teach a humanoid robot to obey text commands","feed_subtitle":"A text-to-action model trained on internet video beats motion-generation baselines and completes real-robot commands at near 100 percent.","key_machinery":"The load-bearing machinery has two parts. First, the Humanoid-X annotation pipeline converts raw Internet video into five aligned modalities (video, text, SMPL human pose, humanoid keypoints, and humanoid actions), with the key transfer step being motion retargeting that fits human shape parameters to the robot and solves inverse kinematics for joint angles, followed by a goal-conditioned RL policy that turns keypoints into physically deployable target joint positions. Second, UH-1 uses a VQ-VAE action tokenizer that maps short action clips to discrete motion-primitive tokens and an autoregressive Transformer conditioned on a CLIP text embedding to generate those tokens, with a first-order reconstruction loss to keep generated motions temporally smooth. This design is what lets the model be trained on 20 million action frames and then decoded into PD-controller torques for a real robot.","core_discovery":"The central claim is that massive, automatically labeled human-video data is a viable substitute for expensive teleoperation and narrow simulation when learning whole-body humanoid control. Concretely, the paper claims that UH-1 maps natural-language instructions directly to either humanoid keypoints (closed-loop, via a goal-conditioned policy) or joint target positions (open-loop), and that this mapping generalizes across a vocabulary of thousands of action words. The evidence reported is that UH-1 beats two-stage text-to-motion-plus-retargeting baselines on the retargeted HumanoidML3D benchmark, that pre-training on Humanoid-X improves FID from 0.445 to 0.379, that scaling the training set from 1% to 100% monotonically improves FID and diversity, and that on a real humanoid robot the model completes 12 distinct text commands at roughly 90-100% success in both control modes.","pith_inferences":["An implication the paper leaves implicit: if the RL-labeling assumption holds, the same pipeline should scale further by simply adding more video, since captioning, pose estimation, and retargeting are all automated; only scaling within a fixed 163,800-clip pool is demonstrated.","A testable extension of the central claim is full-body lower-limb control from video: the real-robot experiments mostly evaluate upper-body gestures with a separate locomotion policy, so whether massive video data improves legs, balance, and locomotion in the wild remains open.","The comparison against text-to-motion-plus-retargeting baselines suggests the intermediate human-motion generation step is a bottleneck; if UH-1's one-stage mapping is the cause of its gain, future work should focus on direct action-space modeling rather than improving generated 3D human motion quality."],"forward_implications":["Text-conditioned humanoid control can scale with data: pre-training on Humanoid-X improves FID from 0.445 to 0.379 on HumanoidML3D, and scaling the training set from 1% to 100% improves FID from 0.689 to 0.463.","One model can cover a broad action vocabulary with a single text-to-action mapping, without an explicit intermediate human-motion generation step.","Open-loop text-to-action control is sufficient for many upper-body tasks when paired with a separate locomotion policy, while closed-loop text-to-keypoint control is more robust in simulation (mean success 94.33% versus 89.83%).","On the tested commands, the model transfers to a real humanoid robot with roughly 90-100% success in both control modes.","The first-order reconstruction loss in the action tokenizer improves the quality and temporal smoothness of generated humanoid actions."],"supporting_citations":[{"why":"Supplies the video captioning model that generates the action-focused text descriptions in Humanoid-X.","marker":"[10]"},{"why":"Provides the HumanML3D benchmark that the paper retargets into HumanoidML3D for evaluation.","marker":"[18]"},{"why":"Supplies the motion-retargeting approach for fitting human shapes to the humanoid robot before keypoint extraction.","marker":"[20]"},{"why":"Supplies the video-based 3D human pose and shape estimator (SMPL parameters) used on all video clips.","marker":"[27]"},{"why":"Supplies the AMASS/CMU MoCap corpus used to train the goal-conditioned RL labeling policy.","marker":"[37]"},{"why":"Supplies the CLIP text encoder that embeds instructions for the UH-1 Transformer.","marker":"[45]"},{"why":"Supplies the PPO algorithm used to train the goal-conditioned humanoid control policy.","marker":"[52]"},{"why":"Serves as the motion-diffusion baseline that UH-1 must beat on HumanoidML3D.","marker":"[59]"},{"why":"Supplies the VQ-VAE discrete representation learning method behind the action tokenizer.","marker":"[62]"},{"why":"Serves as the text-to-motion GPT baseline that UH-1 must beat on HumanoidML3D.","marker":"[73]"}],"fun_headline_variants":["20M human videos teach humanoid robots text commands","Text commands control a real humanoid from 20M video poses","Humanoid-X: 20M poses for universal text-to-action control","Scaling video data to universal humanoid text control","From 20M human videos to text-driven humanoid control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's action labels come from a goal-conditioned RL policy trained on a curated subset of CMU MoCap from AMASS, not on the Humanoid-X videos themselves, and the paper reports no per-sample check that this policy successfully tracks the noisy, video-derived keypoints for all 163,800 samples; if the policy silently fails on motions outside its MoCap training distribution, much of the 20-million-sample training signal would be mislabeled.","fun_headline_variants_meta":{"raw":{"variants":["20M human videos teach humanoid robots text commands","Text commands control a real humanoid from 20M video poses","Humanoid-X: 20M poses for universal text-to-action control","Scaling video data to universal humanoid text control","From 20M human videos to text-driven humanoid control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000817,"raw_usage":{"total_tokens":3575,"prompt_tokens":936,"completion_tokens":2639,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":2554}},"tokens_in":552,"tokens_out":2639,"duration_ms":18262,"temperature":1.0,"reasoning_tokens":2554,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:24:28.272960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the goal-conditioned policy of Appendix B.6 on a random sample of Humanoid-X keypoint sequences and record the fraction of episodes that keep the robot stable and within a defined tracking error (for example, root position error below a threshold with no fall); if a substantial fraction of sequences cannot be tracked, then a large share of the dataset's physically deployable action labels are wrong, which would invalidate the claim that UH-1 learns deployable actions from massive video data.","supporting_citations":[{"cited_title":"Generating diverse and natural 3d human motions from text","cited_arxiv_id":null,"evidence_quote":"Provides the HumanML3D benchmark that the paper retargets into HumanoidML3D for evaluation."},{"cited_title":"Vibe: Video inference for human body pose and shape estimation","cited_arxiv_id":null,"evidence_quote":"Supplies the video-based 3D human pose and shape estimator (SMPL parameters) used on all video clips."},{"cited_title":"Troje, Ger- ard Pons-Moll, and Michael J","cited_arxiv_id":null,"evidence_quote":"Supplies the AMASS/CMU MoCap corpus used to train the goal-conditioned RL labeling policy."},{"cited_title":"Learn- ing transferable visual models from natural language super- vision","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP text encoder that embeds instructions for the UH-1 Transformer."},{"cited_title":"Human motion diffu- sion model","cited_arxiv_id":null,"evidence_quote":"Serves as the motion-diffusion baseline that UH-1 must beat on HumanoidML3D."},{"cited_title":"Neural discrete representation learning","cited_arxiv_id":null,"evidence_quote":"Supplies the VQ-VAE discrete representation learning method behind the action tokenizer."},{"cited_title":"Generating human motion from textual descriptions with discrete representations","cited_arxiv_id":null,"evidence_quote":"Serves as the text-to-motion GPT baseline that UH-1 must beat on HumanoidML3D."}],"review_version":1}