Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

Humanoid World Models: Open World Foundation Models for Humanoid Robotics

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a lightweight, open-source world model can forecast humanoid egocentric video from control tokens, trained and run on one or two GPUs using 100 hours of demonstrations.

desk verdict Solid lightweight video-prediction engineering, but the world-model claim is unproven because nothing shows the action tokens actually steer the generated video. read the letter →

arxiv 2506.01182 v2 pith:USYXPLR2 submitted 2025-06-01 cs.RO cs.AI

Muhammad Qasim Ali , Aditya Sridhar , Shahbuland Matiana , Alex Wong , Mohammad Al-Sharman This is my paper
classification cs.ROcs.AI
keywords humanoidroboticsworldmodelsegocentricvideopredictionmaskedmodelingflowmatchingparametersharingaction-conditionedgenerationlightweightfoundation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a physically plausible, humanoid-specific world model can be trained and deployed on hardware as modest as one or two workstation GPUs, not just in large compute centers. It presents Humanoid World Models (HWM), a family of action-conditioned video generators trained on 100 hours of egocentric demonstrations, and reports that masked-transformer models forecast future frames better than flow-matching models on the same data. The paper's key efficiency claim is that sharing transformer parameters across the video and action token streams reduces model size by 33–53% with only small changes in quality metrics. If the claim holds, open-source predictive simulators for humanoids become practical for small labs, giving robots a way to imagine the consequences of actions before executing them.

What carries the argument

The load-bearing mechanism is masked video modelling in a discrete latent space: a spatiotemporal VQ-VAE (a tokenizer that maps short video clips into compact grids of discrete tokens) compresses video 8× in space and time, and a bidirectional masked transformer learns to fill in masked future tokens from past tokens and embedded action vectors. The block-level design is factorized spatial/temporal attention with rotary position embeddings, organized into four token streams (past video, future video, past actions, future actions), with variants that use joint attention, split self/cross-attention, or shared parameters across streams. For the flow-matching counterpart, a continuous VAE latent is used and the same block variants are trained with a velocity-matching objective and classifier-free guidance. These masking and parameter-sharing choices carry the argument: cosine masking plus token corruption teaches the transformer to reconstruct future structure, and sharing QKV projections, MLPs, and modulation parameters across streams is what yields the 33–53% size reduction with near-parity metrics.

What would settle it

Condition the trained model on mismatched actions for held-out clips, for example by permuting or zeroing the future action vectors while keeping the same past frames; a genuinely action-conditioned world model should produce measurably worse FID and PSNR than with the true actions, and if the gap is negligible the scoring is dominated by appearance priors rather than action-driven prediction. A second check is multi-step closed-loop rollouts, where generated frames are fed back as past context and errors are tracked over repeated predictions.

Watch

Extended reading notes

Core claim

The central claim is that lightweight open-source models can serve as humanoid world models: given nine past egocentric frames and a sequence of control tokens, HWM generates eight future 256×256 RGB frames that match recorded video well enough to be useful for planning and synthetic data. On 100 hours of humanoid demonstrations, the masked-transformer variant reaches FID 10.13 and PSNR 29.02 dB, while flow-matching's best FID is 110.73 and its PSNR tops out near 20.50 dB even with more parameters and longer training. The paper further claims that sharing parameters across token streams is nearly free: full sharing cuts the masked model from 0.321 to 0.195 billion parameters and peak GPU memory from 2.63 to 2.12 GB, with FID moving from 10.13 to 14.21 and PSNR from 29.02 to 28.66 dB. The intended upshot is that predictive simulation for humanoid robots does not require the multi-billion-parameter, multi-GPU setups of current video foundation models.

Load-bearing premise

The load-bearing premise is that the 25-dimensional action records are accurately synchronized with the 30 Hz egocentric frames and are sufficient to determine what the camera sees next; if the alignment is noisy or the actions do not capture the disturbances and contacts that change the scene, the model's good frame scores would not transfer to planning or new environments.

Editorial extensions

If this is right

  • A humanoid world model with 0.2–0.3 billion parameters can be trained on a single workstation GPU and used for egocentric video prediction, placing predictive simulation within reach of small labs.
  • Masked transformers are the better generative backbone for this dataset and compute budget, outperforming flow matching on both FID and PSNR while sampling faster.
  • Full parameter sharing across video and action streams cuts model size by up to 53% and peak memory by about 20%, with FID rising from 10.13 to 14.21 and PSNR dropping from 29.02 to 28.66 dB, a mild degradation that favors efficiency-focused deployments.
  • Because the model is action-conditioned, it can serve as a dynamics model for long-horizon planning and as a generator of synthetic rollouts for policy learning, without requiring real-world trial and error.
  • The open-source release lets other groups finetune HWM on their own embodiment data rather than starting from closed, compute-heavy foundation models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the reported FID and PSNR are computed on VAE-reconstructed ground truth, so they measure how well the model reproduces the tokenizer's output, not whether predicted motion is physically consistent; a closed-loop rollout test with multi-step feedback would be a sharper validation.
  • Beyond the paper: because actions are the only conditioning signal besides past frames, the same architecture should expose whether a model is truly action-driven: swapping in a mismatched action sequence should degrade forecasts, and this test could be run on the released checkpoints.
  • Beyond the paper: the parameter-sharing result suggests that large video diffusion or vision-language-action models may be able to drop a third to half their parameters by sharing weights across context streams, a transferable design lesson for embodied foundation models.
  • Beyond the paper: training on more diverse humanoid embodiments and adding closed-loop fine-tuning on model errors would test whether the 100-hour dataset is the bottleneck or the architecture, a question the paper leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces Humanoid World Models (HWM), a family of lightweight generative world models for forecasting egocentric videos from humanoid demonstrations. Two generative paradigms are explored: masked transformers operating on discrete VQ-VAE tokens, and flow-matching models operating on continuous latents. The authors compare architectural variants along joint vs. split attention and parameter-sharing axes, and evaluate them using FID and PSNR on 21,000 generated frames from a held-out validation set, together with model size, GPU memory, and sampling speed. The main claims are that masked transformers outperform flow-matching models, and that parameter-sharing reduces model size by 33-53% with minimal impact on visual quality or prediction fidelity.

Significance. If the action-conditioning premise holds, HWM would be a useful open-source contribution: it demonstrates that a humanoid-specific world model can be trained on modest hardware (1-2 GPUs), and the systematic comparison of attention and parameter-sharing designs provides practical guidance for small labs. The paper is transparent about resource usage and evaluates against VAE-reconstructed ground truth, which is methodologically sound for isolating generative-model quality. However, the central claim that these are action-conditioned world models is not currently supported: the reported FID/PSNR metrics cannot distinguish an action-conditioned predictor from a model that merely learns a marginal video prior, and the masked-vs-flow comparison is confounded by different tokenizers. The engineering results are plausible, but the evidence needs to be strengthened before the scientific claims can be accepted.

major comments (3)
  1. [Section 2.2, 2.3, and Section 3 (Evaluation)] The paper provides no evidence that the generated future video is actually sensitive to the future action tokens. In the Masked-HWM training objective (Section 2.2), only the masked future video tokens are supervised, and the action streams are never corrupted or predicted; in Flow-HWM (Eq. 3), the loss supervises the velocity field, but nothing prevents the model from ignoring the conditioning tokens ap and af. The only quantitative evaluations are frame-level FID and PSNR over 21,000 generated frames (Section 3), which can be matched by a model that outputs a generic but visually plausible future independent of the action sequence. I ask for an action-consistency experiment: hold the past frames fixed, generate videos under two different future action sequences (e.g., reaching vs. not reaching, or moving left vs. right), and quantify whether the outputs diverge, using a distance metric or a downstream action classifier; alternatively, train an ablated model that receives no action tokens and show the current model outperforms it. Without such a control, the central claim that HWM functions as an action-conditioned world model is unverified.
  2. [Section 3.1 vs. Section 3.2] The headline comparison between masked and flow-matching models is confounded by tokenizer choice: Masked-HWM uses the Cosmos DV8x8x8 tokenizer (Section 3.1) while Flow-HWM uses the Cosmos Continuous 8x16x16 tokenizer (Section 3.2). These tokenizers differ in both quantization type (discrete vs. continuous) and spatial/temporal compression, so the large FID gap (10.13 vs. 110.73) cannot be attributed to the generative model family. To support the conclusion 'Masked Transformers consistently outperformed Flow-Matching,' the comparison should be repeated with a common tokenizer or the claim should be explicitly restricted to the specific tokenizer/model configuration.
  3. [Tables 1 and 2] All quantitative results are reported as single point estimates with no standard deviations, number of seeds, or significance tests. This is particularly problematic for the parameter-sharing claim: the FID difference between Base Block (10.13) and Modality Sharing (11.67) or Full Sharing (14.21) is reported as 'minimal impact', and the PSNR differences (29.02 vs. 28.66) are very small, but without variance estimates these could be within training noise. Please report mean ± std over at least three seeds (or equivalent bootstrap intervals) for the main variants, and state the number of seeds used for each table.
minor comments (7)
  1. [Section 2.2, first paragraph] The phrase 'After passing passing vp, vf' contains a duplicated word 'passing' and should read 'After passing vp, vf'.
  2. [Section 2.4, Parameter Sharing] The notation for the number of layers is inconsistent: the text refers to 'the remaining l − 4 transformer layers' but the models are described as having 'd transformer blocks'; please unify the notation.
  3. [Section 2.4, Flow-HWM base block] The description of 3D rotary position embeddings for video tokens and 1D RoPE for action tokens would benefit from a precise statement of how the embeddings are applied (e.g., to queries and keys, with what spatial/temporal axes) so the implementation is reproducible.
  4. [Section 3.2, Experimental Setup] The use of classifier-free guidance with scale 3.0 is reported, but the paper does not explain how conditioning is dropped during training (e.g., masking of condition tokens with a certain probability); please add this detail.
  5. [Section 4, Conclusion] There is a typo in the final sentence: 'very benefitial' should be 'very beneficial'.
  6. [Section 3, Evaluation] Please clarify whether the 21,000 evaluated frames are non-overlapping and whether the held-out validation split is separated by episode to avoid temporal leakage between training and evaluation clips.
  7. [Abstract and Section 4] The models are described as open-source, but no code or model release link is provided; please include a repository or checkpoint URL in the final version.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: all reported results are empirical metrics on held-out data, and the parameter-sharing efficiency claims are structural rather than fitted outcomes.

full rationale

This paper does not exhibit any circular derivation. The central claims are (1) that masked-transformer and flow-matching world models can be trained on the 1xGPT dataset to forecast future egocentric video, and (2) that parameter-sharing variants reduce model size with modest quality changes. Neither claim reduces to the paper's inputs by construction. The FID and PSNR numbers in Tables 1 and 2 are measured on 21,000 generated frames against a held-out validation set, with ground truth passed through the same VAE to isolate the generative model from tokenizer reconstruction quality (Section 3, Evaluation paragraph). This is a standard evaluation protocol, not a fitted quantity renamed as a prediction. The parameter-sharing size reductions (33-53%) follow directly from the architectural description in Section 2.4: shared QKV projections, modulation parameters, and MLPs across token streams mechanically reduce parameter count; they are not learned or fitted outputs. The comparative claim that Masked-HWM outperforms Flow-HWM is an empirical observation, not a consequence of the definitions. The paper cites prior work such as MaskGIT, MAGVIT, Copilot-4D, SD3, Cosmos, and fal.ai for architectural inspiration, but none of these citations is load-bearing in a way that forbids alternatives or imports a preferred conclusion from the authors' own prior work; no uniqueness theorem is invoked. The reviewer-flagged concerns about the lack of an action-consistency control and the tokenizer confound between Masked-HWM (Cosmos DV8x8x8) and Flow-HWM (Cosmos 8x16x16) are legitimate threats to the strength of the empirical conclusions, but they are not circularity. A confounded comparison is a validity issue, not a derivation that is equivalent to its inputs. Because there is no self-definitional step, no fitted input called a prediction, and no load-bearing self-citation chain, the appropriate circularity score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claims rest on hand-set hyperparameters and on assumptions about data synchronization, metric validity, and pretrained tokenizers. No new physical entities are introduced.

free parameters (6)
  • Maximum corruption rate rho_max = 0.2
    Hand-set for Masked-HWM training (Section 3.1); affects robustness of masked reconstruction and final video quality.
  • Decoding iterations K = 2
    Hand-set Masked-HWM inference schedule (Section 3.1); lower K speeds sampling but may reduce refinement quality.
  • Classifier-free guidance scale = 3.0
    Hand-set for Flow-HWM inference (Section 3.2); trades sample fidelity against diversity and affects FID/PSNR.
  • Flow denoising steps = 50
    Hand-set ODE solver steps for Flow-HWM (Section 3.2); more steps generally improve quality at higher compute.
  • Number of shared transformer layers = last l-4 layers
    Hybrid sharing scheme: first 4 blocks keep separate parameters, remaining l-4 share (Section 2.4); this choice directly sets the 33-53% size reductions reported.
  • Flow-matching sigma_min = not specified
    Small positive constant in Eq. (1) ensuring non-zero support at t=1; value is never given, though it affects the velocity target and sampling trajectory.
assumptions (4)
  • domain assumption The 1xGPT dataset provides accurately synchronized 25-d action vectors for every 30Hz egocentric frame (Section 3, Dataset).
    All action-conditioned video prediction depends on this alignment; errors systematically corrupt the learned dynamics.
  • domain assumption FID and PSNR computed on VAE-reconstructed ground truth are meaningful proxies for world-model quality (Section 3, Evaluation).
    The paper's quantitative claims rest entirely on these metrics; they do not measure physical plausibility, planning success, or generalization.
  • domain assumption The Cosmos tokenizers preserve sufficient spatiotemporal information for generative training (Sections 3.1 and 3.2).
    Both models operate in deep latent spaces; if the tokenizer discards critical dynamics, no generative model can recover them.
  • standard math Flow matching linear interpolation (Eqs. 1-2) defines a valid probability path for video latents (Section 2.3).
    The framework is adopted from Lipman et al. (2023) without re-derivation; standard assumptions of the method are inherited.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Humanoid World Models: Open World Foundation Models for Humanoid Robotics." pith.science (2026). https://pith.science/paper/USYXPLR2

@misc{pith2026250601182,
  author       = {Pith},
  title        = {Pith review of: Humanoid World Models: Open World Foundation Models for Humanoid Robotics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/USYXPLR2}},
  note         = {Machine review of arXiv:2506.01182}
}
read the original abstract

Humanoid robots, with their human-like form, are uniquely suited for interacting in environments built for people. However, enabling humanoids to reason, plan, and act in complex open-world settings remains a challenge. World models, models that predict the future outcome of a given action, can support these capabilities by serving as a dynamics model in long-horizon planning and generating synthetic data for policy learning. We introduce Humanoid World Models (HWM), a family of lightweight, open-source models that forecast future egocentric video conditioned on humanoid control tokens. We train two types of generative models, Masked Transformers and Flow-Matching, on 100 hours of humanoid demonstrations. Additionally, we explore architectural variants with different attention mechanisms and parameter-sharing strategies. Our parameter-sharing techniques reduce model size by 33-53% with minimal impact on performance or visual fidelity. HWMs are designed to be trained and deployed in practical academic and small-lab settings, such as 1-2 GPUs.

Figures

Figures reproduced from arXiv: 2506.01182 by the authors.

Figure 1
Figure 1. Overview of Humanoid World Models (HWM). Given past video observations and humanoid control tokens (joint angles, velocities, etc.), it predicts future video observations. One avenue for improving humanoid intelligence and con￾trol is through World Models—predictive models trained to forecast future outcomes based on past observations and actions (Ha & Schmidhuber, 2018). In our setting, these function as action-con… view at source ↗
Figure 2
Figure 2. Architecture of a single transformer block in Masked￾HWM (Base Block variant). Video and action streams are pro￾cessed independently, with video streams also receiving Spatial Attention. All streams interact via joint Temporal Attention. RoPE is applied per attention type (2D for spatial and 1D for temporal). Each stream uses distinct MLP weights in the feedforward stage. 2024), we adopt a factorized or separate spa… view at source ↗
Figure 3
Figure 3. Architecture of a single transformer block in Flow-HWM (Base Block variant). Each token stream (past/future video and actions) uses separate weights for timestep modulation, QKV pro￾jection, and feedforward MLPs. Joint Attention integrates all streams. RoPE is applied by modality: 3D for video tokens, 1D for action tokens. beddings (RoPE) to the video tokens, as done in Cosmos (NVIDIA et al., 2025), and apply 1D RoP… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The model learns both structural elements of the scene, such as furniture and small objects that the robot is manipulating, and overall textures. Larger parts of the robot’s appendage, including arms and wheels, are generally modeled accurately. However, precision elem…
Figure 5
Figure 5. Figure 5: Sample videos from the Base Variant of Flow-HWM. Top row: generated frames; bottom row: ground truth. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Foundation Model Driven Robotics: A Comprehensive Review

    cs.RO 2025-07 conditional novelty 2.0 of 10

    A review of foundation-model-driven robotics that synthesizes recent work across perception, planning, control, HRI, simulation, and sim-to-real transfer, and highlights open challenges.

Reference graph

Works this paper leans on

50 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    URL https:// doi.org/10.1007/978-3-540-27833-7_1 . Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lee, A., Vyas, A., Shi, B., Ma, C.-Y ., Chuang, C.-Y ., Yan, D., Choudhary, D., Wang, D., Sethi, G., Pang, G., Ma, H., Misra, I., Hou, J., Wang, J., Jagadeesh, K., Li, K., Zhang, L., Singh, M., Williamson, M., Le, M., Yu, M., Singh, M. K., Zhang, P...

    work page doi:10.1007/978-3-540-27833-7_1

  2. [4]

    Bommasani, R., Hudson, D

    URL https://arxiv.org/abs/2410.24164. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., ...

    arXiv

  3. [6]

    Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11315–11325, 2022a. Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer, 2022b. URL https://arxiv.org/abs/...

    arXiv

  4. [7]

    Du, Y ., Yang, M., Florence, P., Xia, F., Wahid, A., Ichter, B., Sermanet, P., Yu, T., Abbeel, P., Tenenbaum, J

    URL https: //arxiv.org/abs/2503.10618. Du, Y ., Yang, M., Florence, P., Xia, F., Wahid, A., Ichter, B., Sermanet, P., Yu, T., Abbeel, P., Tenenbaum, J. B., Kaelbling, L., Zeng, A., and Tompson, J. Video language planning,

    arXiv

  5. [8]

    Duan, J., Pumacay, W., Kumar, N., Wang, Y

    URL https://arxiv.org/abs/ 2310.10625. Duan, J., Pumacay, W., Kumar, N., Wang, Y . R., Tian, S., Yuan, W., Krishna, R., Fox, D., Mandlekar, A., and Guo, Y . Aha: A vision-language-model for detecting and rea- soning over failures in robotic manipulation,

    arXiv

  6. [9]

    URL https://arxiv.org/abs/2410.00371. Esser, P., Kulal, S., Blattmann, A., Entezari, R., M ¨uller, J., Saini, H., Levi, Y ., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., Lacey, K., Goodwin, A., Marek, Y ., and Rombach, R. Scaling recti- fied flow transformers for high-resolution image synthe- sis,

    arXiv

  7. [12]

    Conditional Image Generation by Conditioning Variational Auto-Encoders

    URL https://arxiv.org/abs/ 2102.12037. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    work page Pith review arXiv

  8. [15]

    Ho, J., Jain, A., and Abbeel, P

    URL https://arxiv.org/abs/ 2207.12598. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models,

    arXiv

Show all 50 references
  1. [17]

    Hor´e, A

    URL https://arxiv.org/abs/2204.03458. Hor´e, A. and Ziou, D. Image quality metrics: Psnr vs. ssim. In 2010 20th International Conference on Pattern Recognition, pp. 2366–2369,

    2010 arXiv

  2. [19]

    org/abs/2401.02105

    URLhttps://arxiv. org/abs/2401.02105. Jin, Y ., Sun, Z., Li, N., Xu, K., Xu, K., Jiang, H., Zhuang, N., Huang, Q., Song, Y ., Mu, Y ., and Lin, Z. Pyramidal flow matching for efficient video generative modeling,

    arXiv

  3. [20]

    URL https://arxiv.org/abs/2410.05954. Kim, M., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., Vuong, Q., Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., and Finn, C. Open- vla: An o...

  4. [21]

    Langley, P

    URL https://arxiv.org/abs/2412.03603. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,

    2000 arXiv

  5. [22]

    Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A

    URL https://arxiv.org/ abs/2408.12093. Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Lan- guage model programs for embodied control,

    arXiv

  6. [23]

    Lipman, Y ., Chen, R

    URL https://arxiv.org/abs/2209.07753. Lipman, Y ., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling,

    arXiv

  7. [24]

    Liu, X., Gong, C., and Liu, Q

    URL https://arxiv.org/abs/2210.02747. Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learn- ing to generate and transfer data with rectified flow,

    arXiv

  8. [25]

    Liu, Y ., Zhang, K., Li, Y ., Yan, Z., Gao, C., Chen, R., Yuan, Z., Huang, Y ., Sun, H., Gao, J., He, L., and Sun, L

    URL https://arxiv.org/abs/2209.03003. Liu, Y ., Zhang, K., Li, Y ., Yan, Z., Gao, C., Chen, R., Yuan, Z., Huang, Y ., Sun, H., Gao, J., He, L., and Sun, L. Sora: A review on background, technology, limitations, and opportunities of large vision models,

    arXiv

  9. [26]

    Luo, Z., Shi, F., Ge, Y ., Yang, Y ., Wang, L., and Shan, Y

    URL https://arxiv.org/abs/2402.17177. Luo, Z., Shi, F., Ge, Y ., Yang, Y ., Wang, L., and Shan, Y . Open-magvit2: An open-source project toward democ- ratizing auto-regressive visual generation,

    arXiv

  10. [27]

    URL https://arxiv.org/abs/2409.04410. NVIDIA, :, Agarwal, N., Ali, A., Bala, M., Balaji, Y ., Barker, E., Cai, T., Chattopadhyay, P., Chen, Y ., Cui, Y ., Ding, Y ., Dworakowski, D., Fan, J., Fenzi, M., Fer- roni, F., Fidler, S., Fox, D., Ge, S., Ge, Y ., Gu, J., Gu- rurani, S...

    arXiv

  11. [28]

    Peebles, W

    URL https://arxiv.org/abs/2501.03575. Peebles, W. and Xie, S. Scalable diffusion models with trans- formers,

    arXiv

  12. [29]

    Pfeifer, R

    URL https://arxiv.org/abs/ 2212.09748. Pfeifer, R. and Iida, F. Embodied Artificial Intelligence: Trends and Challenges, pp. 1–26. Springer Berlin Heidel- berg, Berlin, Heidelberg,

    arXiv

  13. [32]

    Ramesh, A., Pavlov, M., Goh, G., Gray, S., V oss, C., Rad- ford, A., Chen, M., and Sutskever, I

    URL https://arxiv.org/abs/2410.13720. Ramesh, A., Pavlov, M., Goh, G., Gray, S., V oss, C., Rad- ford, A., Chen, M., and Sutskever, I. Zero-shot text-to- image generation,

    arXiv

  14. [34]

    Sohl-Dickstein, J., Weiss, E

    URL https://arxiv.org/ abs/2112.10752. Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics,

    arXiv

  15. [36]

    Su, J., Lu, Y ., Pan, S., Murtadha, A., Wen, B., and Liu, Y

    URL https://arxiv.org/abs/2011.13456. Su, J., Lu, Y ., Pan, S., Murtadha, A., Wen, B., and Liu, Y . Roformer: Enhanced transformer with rotary position embedding,

    2011 arXiv

  16. [37]

    Tong, S., Liu, Z., Zhai, Y ., Ma, Y ., LeCun, Y ., and Xie, S

    URL https://arxiv.org/abs/ 2104.09864. Tong, S., Liu, Z., Zhai, Y ., Ma, Y ., LeCun, Y ., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal llms,

    arXiv

  17. [38]

    van den Oord, A., Vinyals, O., and Kavukcuoglu, K

    URL https://arxiv.org/ abs/2401.06209. van den Oord, A., Vinyals, O., and Kavukcuoglu, K. Neural discrete representation learning,

    arXiv

  18. [39]

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A

    URL https: //arxiv.org/abs/1711.00937. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need,

    arXiv

  19. [40]

    Vianello, L., Penco, L., Gomes, W., You, Y ., Anzalone, S

    URL https://arxiv.org/ abs/1706.03762. Vianello, L., Penco, L., Gomes, W., You, Y ., Anzalone, S. M., Maurice, P., Thomas, V ., and Ivaldi, S. Human- humanoid interaction and cooperation: a review. Current Robotics Reports, 2(4):441–454, December

    arXiv

  20. [41]

    URL https://doi

    doi: 10.1007/s43154-021-00068-z. URL https://doi. org/10.1007/s43154-021-00068-z . Wang, S., Han, M., Jiao, Z., Zhang, Z., Wu, Y . N., Zhu, S.-C., and Liu, H. Llm3:large language model-based task and motion planning with motion failure reasoning,

    doi

  21. [42]

    Wu, J., Yin, S., Feng, N., He, X., Li, D., Hao, J., and Long, M

    URL https://arxiv.org/abs/2403.11552. Wu, J., Yin, S., Feng, N., He, X., Li, D., Hao, J., and Long, M. ivideogpt: Interactive videogpts are scalable world models. In Advances in Neural Information Processing Systems,

    arXiv

  22. [43]

    Xing, Z., Feng, Q., Chen, H., Dai, Q., Hu, H., Xu, H., Wu, Z., and Jiang, Y .-G

    URL https://arxiv.org/abs/2406.09455. Xing, Z., Feng, Q., Chen, H., Dai, Q., Hu, H., Xu, H., Wu, Z., and Jiang, Y .-G. A survey on video diffusion mod- els,

    arXiv

  23. [44]

    URL https://arxiv.org/abs/2310. 10647. Yang, M., Du, Y ., Ghasemipour, K., Tompson, J., Schuur- mans, D., and Abbeel, P. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114,

    arXiv

  24. [45]

    URL https://arxiv.org/abs/2402.17139. Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y ., Hong, W., Zhang, X., Feng, G., Yin, D., 11 Humanoid World Models Zhang, Y ., Wang, W., Cheng, Y ., Xu, B., Gu, X., Dong, Y ., and Tang, J. Cogvideox: Text-to-video diff...

    arXiv

  25. [46]

    Yu, L., Cheng, Y ., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A

    URL https: //arxiv.org/abs/2408.06072. Yu, L., Cheng, Y ., Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M.-H., Hao, Y ., Essa, I., and Jiang, L. Magvit: Masked generative video trans- former,

    arXiv

  26. [47]

    Yu, L., Lezama, J., Gundavarapu, N

    URL https://arxiv.org/abs/ 2212.05199. Yu, L., Lezama, J., Gundavarapu, N. B., Versari, L., Sohn, K., Minnen, D., Cheng, Y ., Birodkar, V ., Gupta, A., Gu, X., Hauptmann, A. G., Gong, B., Yang, M.-H., Essa, I., Ross, D. A., and Jiang, L. Language model beats diffusion – tokeni...

    arXiv

  27. [48]

    Zhang, J., Huang, J., Jin, S., and Lu, S

    URL https://arxiv.org/abs/2310.05737. Zhang, J., Huang, J., Jin, S., and Lu, S. Vision-language models for vision tasks: A survey,

    arXiv

  28. [49]

    Zhang, L., Xiong, Y ., Yang, Z., Casas, S., Hu, R., and Urtasun, R

    URL https: //arxiv.org/abs/2304.00685. Zhang, L., Xiong, Y ., Yang, Z., Casas, S., Hu, R., and Urtasun, R. Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion. arXiv preprint arXiv:2311.01017,

    arXiv

  29. [50]

    Zhu, F., Wu, H., Guo, S., Liu, Y ., Cheang, C., and Kong, T

    URL https://arxiv.org/abs/ 2304.13705. Zhu, F., Wu, H., Guo, S., Liu, Y ., Cheang, C., and Kong, T. Irasim: Learning interactive real-robot action simulators. arXiv:2406.12802,

    arXiv

  30. [2004]

    doi: 10.1007/978-3-540-27833-7

    ISBN 978-3-540-27833-7. doi: 10.1007/978-3-540-27833-7

    doi

  31. [2007]

    URL https://royalsocietypublishing.org/ doi/10.1098/rsta.2006.1917

    doi: 10.1098/rsta.2006.1917. URL https://royalsocietypublishing.org/ doi/10.1098/rsta.2006.1917. Ho, J. and Salimans, T. Classifier-free diffusion guid- ance,

    2006

  32. [2010]

    2010.579

    doi: 10.1109/ICPR. 2010.579. Imtiaz, R. and Khan, A. Perceptions of humanoid robots in caregiving: A study of skilled nursing home and long term care administrators,

    2010 doi

  33. [2014]

    Goswami, A

    URLhttps: //arxiv.org/abs/1406.2661. Goswami, A. and Vadakkepat, P. (eds.). Hu- manoid Robotics: A Reference . Springer Dor- drecht,

    arXiv

  34. [2015]

    Song, Y ., Sohl-Dickstein, J., Kingma, D

    URL https: //arxiv.org/abs/1503.03585. Song, Y ., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative model- ing through stochastic differential equations,

    arXiv

  35. [2018]

    Hirose, M

    URL https://arxiv.org/abs/1706.08500. Hirose, M. and Ogawa, K. Honda humanoid robots develop- ment. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 365 (1850):11–19,

    arXiv

  36. [2019]

    doi: 10.1007/978-94-007-6046-2

    ISBN 978-94-007-6046-2. doi: 10.1007/978-94-007-6046-2. URL https: //link.springer.com/referencework/ 10.1007/978-94-007-6046-2 . 9 Humanoid World Models Ha, D. and Schmidhuber, J. World mod- els. CoRR, abs/1803.10122,

    arXiv

  37. [2020]

    Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D

    URL https://arxiv.org/ abs/2006.11239. Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models,

    2006 arXiv

  38. [2021]

    Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B

    URL https://arxiv.org/ abs/2102.12092. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models,

    arXiv

  39. [2022]

    Bruce, J., Dennis, M

    URL https://arxiv.org/abs/2108.07258. Bruce, J., Dennis, M. D., Edwards, A., Parker-Holder, J., Shi, Y ., Hughes, E., Lai, M., Mavalankar, A., Steiger- wald, R., Apps, C., et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning,

    arXiv

  40. [2023]

    Bar, A., Zhou, G., Tran, D., Darrell, T., and LeCun, Y

    URL https://arxiv.org/ abs/2303.08797. Bar, A., Zhou, G., Tran, D., Darrell, T., and LeCun, Y . Nav- igation world models,

    arXiv

  41. [2024]

    URL https://arxiv.org/abs/2401.12945. Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Shi, L. X., Tanner, J., Vuong...

    arXiv

  42. [2025]

    org/abs/2412.03572

    URL https://arxiv. org/abs/2412.03572. Bar-Tal, O., Chefer, H., Tov, O., Herrmann, C., Paiss, R., Zada, S., Ephrat, A., Hur, J., Liu, G., Raj, A., Li, Y ., Rubinstein, M., Michaeli, T., Wang, O., Sun, D., Dekel, T., and Mosseri, I. Lumiere: A space- time diffusion model for vi...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.