Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

From Virtual Games to Real-World Play

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A video diffusion model trained only on labeled car-racing game data and unlabeled real-world videos can control real vehicles, bicycles, and pedestrians in photorealistic generated video, without any real-world action labels.

desk verdict RealPlay shows a promising result—game-only action labels transferring to real-world control—but thin human-only evaluation and a loose CFG analogy leave the central claim under-supported. read the letter →

arxiv 2506.18901 v1 pith:CHGEMSCG submitted 2025-06-23 cs.CV

Wenqiang Sun , Fangyun Wei , Jinjing Zhao , Xi Chen , Zilong Chen , Hongyang Zhang , Jun Zhang , Yan Lu This is my paper · ORCID
classification cs.CV
keywords interactivevideogenerationdiffusionmodelsworldgame-to-realtransferentitychunk-wiseclassifier-freeguidanceadaptiveLayerNorm
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces RealPlay, an interactive video-generation system that acts like a playable engine for real-world footage. Its central claim is that a diffusion-based video generator can learn, from labeled car-racing game data plus unlabeled real-world videos, to respond to user commands—move forward, turn left, turn right—by producing short photorealistic video chunks, and that these commands transfer to real vehicles, bicycles, and pedestrians without ever being labeled on real data. The result matters because it suggests that high-fidelity, controllable simulation of the real world can be built from game supervision plus unlabeled video, avoiding the bottleneck of manual action annotation. RealPlay reports a 90 percent human-judged control success rate averaged across entities, with per-entity rates between 83 and 95 percent.

What carries the argument

The central object is an action-conditioned chunk-wise video diffusion generator, adapted from a pre-trained image-to-video diffusion transformer. The adaptation replaces single-frame conditioning with chunk-level conditioning, uses an attention mask so the conditioning chunk only sees itself and the language tokens, reduces the generated horizon from 13 to 4 video latents per chunk, and adds noise to conditioning chunks during training so the model tolerates its own imperfect past outputs. A control module injects a one-hot action (move forward, turn left, turn right) through an adaptive LayerNorm: an MLP projects the action into a 512-dimensional feature, which is added to the timestep embedding and converted into scale and shift parameters that modulate the network activations. The unlabeled real-world videos are presented with the zero action vector, which the paper argues plays the role of condition-dropped samples in classifier-free guidance, teaching the model to associate game action labels with real-world motion.

What would settle it

A direct test is to retrain RealPlay with unlabeled real-world data whose motion does not resemble cars—say, fish or birds—or to replace the zero action vector with a fixed random vector; if control success on those entities collapses to chance, the claimed game-to-real transfer is not a general mechanism but a property of motion similarity or of the zero-vector trick.

Watch

Extended reading notes

Core claim

RealPlay is a neural network-based real-world game engine: it takes a user's control command, observes the current frame, and returns a short photorealistic video chunk that follows the command, then repeats the loop on the newly generated chunk. The paper's central discovery is that this interactive control can be learned with labeled game data alone as the action supervision: the model is trained on a car-racing game with recorded three-command actions together with unlabeled real-world clips of vehicles, bicycles, and pedestrians, using a zero action vector for the real clips. At inference, the game-origin commands steer the real-world entities with a human-judged success rate of 90 percent overall (83.3 for vehicles, 91.7 for bicycles, 95.0 for pedestrians), and the generated footage stays temporally consistent across iterations. The paper also reports that when no clear entity is centered, the camera itself follows the command, so the system learns a combined entity-and-camera control policy.

Load-bearing premise

The recipe assumes that showing a real-world video with a zero action vector teaches the model the same thing as dropping the action condition in classifier-free guidance; if that equivalence is false, the reported transfer would need another explanation and may not extend to other game/real pairings.

Editorial extensions

If this is right

  • A single source of labeled simulation data plus unlabeled real footage is enough to build an interactive photorealistic simulator, removing the need for expensive real-world action annotation.
  • Control accuracy tracks motion amplitude: entities with larger, more distinctive motions are easier to steer, so future real-world game engines should favor high-motion footage in their unlabeled training sets.
  • Adding unlabeled real-world data from other entities improves control of each individual entity, because the shared motion dynamics transfer across entity types.
  • Quality degrades as the number of generated chunks grows, and shrinks when each chunk is only one or a few video latents, so there is a latency-vs-quality tradeoff in interactive generation.
  • More labeled game samples consistently improve real-world control transfer, indicating that the practical bottleneck is the quantity of labeled simulation data, not the real-world footage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the zero-vector/classifier-free-guidance equivalence is the true mechanism, the recipe should transfer to any pair of labeled simulator and unlabeled real footage with roughly matched motion distributions; a direct test would be swapping the racing game for a legged-robot simulator and checking whether control transfers to humanoid walking.
  • The entity transfer likely decomposes into game data supplying the action-to-motion mapping and unlabeled real data supplying appearance and plausible dynamics; if so, controlling a brand-new entity should require only a few unlabeled clips of that entity, not new labels.
  • The camera-follows-entity behavior suggests the control signal is a steering command to the whole observation (camera and subject) rather than an object-level action, so future designs might separate camera control from entity control to make the interaction more predictable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. RealPlay adapts CogVideoX-5B into a chunk-wise interactive video generator. In stage 1 it trains a chunk-conditioned model on general-domain video data; in stage 2 it fine-tunes on labeled game data from Forza Horizon 5 (three actions: forward, left, right) plus unlabeled real-world videos of vehicles, bicycles, and pedestrians, using a zero action vector for real clips and injecting actions via adaptive LayerNorm. The paper claims two transfer phenomena: control transfer from game commands to real-world entities, and entity transfer to bicycles and pedestrians, with a 90% human-judged control success rate and an Elo of 1184. It also reports ablations on per-entity data, game data quantity, action injection strategy, and the number of video latents per chunk.

Significance. If the empirical claim holds, this is a meaningful result for interactive video generation: it suggests that game-derived action labels can be combined with unlabeled real footage to build controllable photorealistic video, avoiding expensive real-world action annotation. The paper includes useful comparisons against text-prompted video generators, a human-labeled real-data baseline, and a LAPA pseudo-label baseline, and the per-entity and cross-entity ablations are informative. I see no circularity in the evaluation: control success is judged on generated videos and is not fitted into the model. However, the central quantitative evidence is a single human-judged number without error bars or objective corroboration, and the proposed CFG-based explanation for transfer is not substantiated; these gaps are fixable but currently limit confidence in the generalization claim.

major comments (4)
  1. [Section 4, Table 1] The paper reports the headline 90.0% control success rate as the primary evidence for its central claim, but the evaluation is described only as 'human evaluation' with no number of evaluators, no number of judged clips, no confidence interval or error bar, no inter-annotator agreement, and no per-action confusion matrix. Because the action set has only three labels and the real training videos are unlabeled, the measured accuracy could in part reflect a strong forward-motion prior and a generic 'move' bias rather than a faithful per-command mapping. The authors should report the full evaluation protocol, per-action results for forward/left/right, confidence intervals, agreement statistics, and at least one objective trajectory metric, such as optical-flow direction of the segmented entity or camera motion, computed on held-out prompts.
  2. [Section 3.2] The claimed equivalence to classifier-free guidance is not established. Standard CFG trains conditional and unconditional predictions on the same data distribution with a random drop probability and interpolates at inference with a guidance scale; here the zero action vector is used for every unlabeled real sample, the conditional branch is trained only on game scenes, and no action-drop probability or CFG scale is reported in Section 4. For a real-world input, the conditional prediction is therefore out-of-distribution, so the proposed 'CFG interpolation' story does not define a steering mechanism. The passage is an analogy, not a derivation; it should be labeled as speculation or supported by experiments that vary the training drop probability and guidance scale and that measure whether the real-branch behavior depends on them.
  3. [Section 4.2, Table 3] The bicycle ablation shows that control success jumps from 0% with game data alone to 72.5% after adding unlabeled real bicycle videos, and to 91.7% after adding all real entity types. This is consistent with an alternative explanation: the unlabeled real data supplies a real-scene motion prior and entity appearance, and the apparent action compliance is a generic response of the camera/entity to the action embedding rather than a game-to-real transfer of per-command semantics. To isolate the role of the game labels, the authors should compare against a control trained with the same real data but with shuffled or random game action labels, and/or evaluate on real videos with human-annotated ground-truth directions and report per-action accuracy. Without such a control, the central claim that the game-provided labels are what transfers is not uniquely supported.
  4. [Section 4.1, CogVideoX-5B (Human) baseline] The human-labeled real-data baseline is trained on only about 20% of the real dataset, namely the 'clear' samples, while RealPlay is trained on all unlabeled real data plus 80K labeled game samples. The comparison therefore confounds annotation quality with data quantity and with the presence of game data. The authors should add a controlled comparison in which the human-labeled baseline also receives the game data, or in which RealPlay uses a matched subset of real data, so that the 90% versus 58.9% gap can be attributed to the proposed recipe rather than to dataset size.
minor comments (4)
  1. [Section 3.2] The notation C_k + a_k -> C_{k+1} is figurative: the action is injected via adaptive LayerNorm, not added to the frame latents. Please clarify this to avoid a literal reading.
  2. [Abstract and Section 5] The phrasing 'trained only with car-based game supervision' is imprecise: the action labels are car-based, but the training data also include unlabeled real-world videos of bicycles and pedestrians. Please restate this to distinguish action-label supervision from visual data.
  3. [Section 4] The paper does not describe how evaluation prompts and videos are selected or whether the evaluated scenes are disjoint from the training sequences; a sentence on the evaluation split would strengthen the generalization claim.
  4. [Section 3.1 and Figure 3] No ablation is provided for the noise augmentation component even though it is presented as a key remedy for iterative inconsistency; a with/without comparison would support the causal claim in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported control-transfer and entity-transfer results are empirical outcomes measured on generated videos, not quantities fitted into the training objective or derived from a self-citation chain.

full rationale

RealPlay's derivation chain is: (1) adapt CogVideoX into a chunk-wise generator on a general-domain video corpus; (2) fine-tune on labeled Forza Horizon 5 game clips with action one-hot vectors plus unlabeled real-world clips with an all-zero action vector; (3) measure control success via human evaluation, Elo scores, VBench metrics, and ablations. The central claim—game-to-real control transfer and transfer to bicycles/pedestrians—is an empirical result, not a parameter fitted to the headline numbers. The all-zero action convention for real clips is a training choice; it does not by construction entail the reported 90% control rate. The Section 3.2 CFG discussion is explicitly labeled 'an intuitive explanation' and is post-hoc; even if the analogy is imperfect or contested, it is not used as a formal derivation, so it does not constitute a circular reduction. Self-citations appear (e.g., The Matrix [9] as the source of game data, and background video-generation works [24, 37]), but none is invoked as an authority that determines RealPlay's transfer capability; the dataset citation supplies training clips, not the conclusion. The appendix limitations are candid and do not reveal any fitted-input-called-prediction pattern. No circular step meets the required standard of quoting a specific equation or construction that makes the output equivalent to the input.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The core claims are empirical and do not rest on a closed-form derivation, so the ledger records the design choices and background assumptions the reported transfer depends on. No invented entities are introduced; RealPlay is a trained model, not a new physical object.

free parameters (3)
  • Temporal chunk size = 4 video latents (16 frames)
    Hand-picked trade-off between response latency and visual quality; Figure 4 shows that 1 latent per chunk degrades quality, so the choice of 4 is an empirical design decision.
  • Training data mixture = 80K game samples + 18K per real entity class
    The ratio of labeled game data to unlabeled real data is chosen without a stated optimization. Figure 6 shows control rate grows with game data amount, so the exact mixture affects the headline 90% control rate.
  • Noise augmentation level = not reported
    The strength of noise injected into conditioning chunks is adopted from Diffusion Forcing [4] but not specified numerically; it directly affects the train-inference gap the method addresses.
assumptions (4)
  • domain assumption CogVideoX-5B provides a strong base prior for real-world dynamics and generalization.
    Invoked in Section 3.2 and Section 5; the transfer result is credited to the pre-trained model's generalization capability.
  • domain assumption Forza Horizon 5 visuals are photorealistic enough to narrow the synthetic-to-real domain gap.
    Section 3.2: 'the game data used in our setup are derived from Forza Horizon 5, a photorealistic AAA game with high visual realism, which significantly narrows the domain gap between synthetic and real-world data.'
  • ad hoc to paper Zero action vector for unlabeled data behaves like condition dropout in classifier-free guidance.
    Section 3.2 presents this as an 'intuitive explanation' rather than a derivation; the control transfer mechanism depends on this asserted equivalence.
  • domain assumption Noise augmentation on conditioning chunks closes the train-inference distribution gap.
    Section 3.1 item 4 adopts this from Diffusion Forcing [4]; the paper does not independently validate the noise schedule.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Virtual Games to Real-World Play." pith.science (2026). https://pith.science/paper/CHGEMSCG

@misc{pith2026250618901,
  author       = {Pith},
  title        = {Pith review of: From Virtual Games to Real-World Play},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CHGEMSCG}},
  note         = {Machine review of arXiv:2506.18901}
}
read the original abstract

We introduce RealPlay, a neural network-based real-world game engine that enables interactive video generation from user control signals. Unlike prior works focused on game-style visuals, RealPlay aims to produce photorealistic, temporally consistent video sequences that resemble real-world footage. It operates in an interactive loop: users observe a generated scene, issue a control command, and receive a short video chunk in response. To enable such realistic and responsive generation, we address key challenges including iterative chunk-wise prediction for low-latency feedback, temporal consistency across iterations, and accurate control response. RealPlay is trained on a combination of labeled game data and unlabeled real-world videos, without requiring real-world action annotations. Notably, we observe two forms of generalization: (1) control transfer-RealPlay effectively maps control signals from virtual to real-world scenarios; and (2) entity transfer-although training labels originate solely from a car racing game, RealPlay generalizes to control diverse real-world entities, including bicycles and pedestrians, beyond vehicles. Project page can be found: https://wenqsun.github.io/RealPlay/

Figures

Figures reproduced from arXiv: 2506.18901 by the authors.

Figure 1
Figure 1. RealPlay is a neural-network-driven real-world game engine with three key characteristics: (1) It supports iterative interaction—at each iteration, users observe the current visual scene, provide control signals, and receive control-accurate, temporally consistent, and realistic video chunks in response. (2) It eliminates the need for annotated real-world data while exhibiting strong control transfer capabilities, e… view at source ↗
Figure 2
Figure 2. RealPlay involves a two-stage training process. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visual quality degrades in both game and real-world settings, but the image quality when controlling a game entity consistently re￾mains higher than that of a real-world entity (e.g., the bicycle in this study), highlighting the greater challenge of modeling real-world entities. 0 1 2 3 4 5 6 Video Chunk ID 64 65 66 67 68 69 70 71 Imaging Quality 1 Latent 2 Latents 4 Latents [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Both the control success rate and Elo scores steadily improve as training progresses. The evaluation is performed on the bicycle entity. 500 1,000 2,000 4,000 6,000 Number of Game Samples 0 20 40 60 80 Control Rate (%) 59.8 60.1 73.1 81.1 83.3 [PITH_FULL_IMAGE:figures…
Figure 7
Figure 7. Figure 7: Visualizations of videos generated by the chunk-wise generation model. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Visualization of RealPlay’s capability to control cars in diverse game environments. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: RealPlay’s effectiveness in controlling diverse real-world entities. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Long-horizon videos generated by RealPlay demonstrating sustained control over real [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: RealPlay learns both entity and camera control; when no focused entity is present, it [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Qualitative comparison with baseline methods. The target action is [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Qualitative comparison with baseline methods. The target action is [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CustomX: Unified Character, Action, and Scene Customization in Video World Models

    cs.CV 2025-12 conditional novelty 6.0 of 10

    AniX generates controllable videos of a user-supplied character performing typed actions inside a user-supplied 3D scene by fine-tuning a pre-trained video generator on small locomotion datasets.

  2. WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling

    cs.CV 2025-12 unverdicted novelty 6.0 of 10

    WorldPlay uses dual action representation, reconstituted context memory, and context forcing distillation to produce consistent 720p streaming video at 24 FPS for interactive world modeling.

  3. From Pixels to States: Rethinking Interactive World Models as Game Engines

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Interactive world models are reorganized around the game-engine action-state-observation loop, and a 90-hour Black Myth: Wukong dataset with frame-aligned actions, ground-truth states, and observations is introduced.

Reference graph

Works this paper leans on

54 extracted references · 14 canonical work pages · cited by 3 Pith papers

  1. [1]

    Alonso, A

    E. Alonso, A. Jelley, V . Micheli, A. Kanervisto, A. J. Storkey, T. Pearce, and F. Fleuret. Diffusion for world modeling: Visual details matter in atari.Advances in Neural Information Processing Systems, 37: 58757–58791, 2024

    work page 2024

  2. [2]

    Bruce, M

    J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y . Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, et al. Genie: Generative interactive environments. InForty-first International Conference on Machine Learning, 2024

    2024

  3. [3]

    H. Che, X. He, Q. Liu, C. Jin, and H. Chen. Gamegen-x: Interactive open-world game video generation. arXiv preprint arXiv:2411.00769, 2024

    arXiv 2024

  4. [4]

    B. Chen, D. Martí Monsó, Y . Du, M. Simchowitz, R. Tedrake, and V . Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion.Advances in Neural Information Processing Systems, 37:24081–24125, 2024

    work page 2024

  5. [5]

    G. Chen, D. Lin, J. Yang, C. Lin, J. Zhu, M. Fan, H. Zhang, S. Chen, Z. Chen, C. Ma, et al. Skyreels-v2: Infinite-length film generative model.arXiv preprint arXiv:2504.13074, 2025

    arXiv 2025

  6. [6]

    H. Chen, M. Xia, Y . He, Y . Zhang, X. Cun, S. Yang, J. Xing, Y . Liu, Q. Chen, X. Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation.arXiv preprint arXiv:2310.19512, 2023

    arXiv 2023

  7. [7]

    H. Chen, Y . Zhang, X. Cun, M. Xia, X. Wang, C. Weng, and Y . Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7310–7320, 2024

    2024

  8. [8]

    Quevedo, Q

    Decart, J. Quevedo, Q. McIntyre, S. Campbell, X. Chen, and R. Wachen. Oasis: A universe in a transformer

    work page

Show all 54 references
  1. [9]

    R. Feng, H. Zhang, Z. Yang, J. Xiao, Z. Shu, Z. Liu, A. Zheng, Y . Huang, Y . Liu, and H. Zhang. The matrix: Infinite-horizon world generation with real-time moving control.arXiv preprint arXiv:2412.03568, 2024

    2024 arXiv

  2. [10]

    Frans, D

    K. Frans, D. Hafner, S. Levine, and P. Abbeel. One step diffusion via shortcut models.arXiv preprint arXiv:2410.12557, 2024

    2024 arXiv

  3. [11]

    K. Gao, J. Shi, H. Zhang, C. Wang, and J. Xiao. Vid-gpt: Introducing gpt-style autoregressive generation in video diffusion models.arXiv preprint arXiv:2406.10981, 2024

    2024 arXiv

  4. [12]

    Y . Gu, W. Mao, and M. Z. Shou. Long-context autoregressive video modeling with next-frame prediction. arXiv preprint arXiv:2503.19325, 2025

    2025 arXiv

  5. [13]

    Y . Guo, C. Yang, A. Rao, Z. Liang, Y . Wang, Y . Qiao, M. Agrawala, D. Lin, and B. Dai. Animated- iff: Animate your personalized text-to-image diffusion models without specific tuning.arXiv preprint arXiv:2307.04725, 2023

    2023 arXiv

  6. [14]

    Henschel, L

    R. Henschel, L. Khachatryan, H. Poghosyan, D. Hayrapetyan, V . Tadevosyan, Z. Wang, S. Navasardyan, and H. Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text.arXiv preprint arXiv:2403.14773, 2024

    2024 arXiv

  7. [15]

    Ho and T

    J. Ho and T. Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  8. [16]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  9. [17]

    J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet. Video diffusion models.Advances in Neural Information Processing Systems, 35:8633–8646, 2022

    2022

  10. [18]

    W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868, 2022

    2022 arXiv

  11. [19]

    Huang, Y

    Z. Huang, Y . He, J. Yu, F. Zhang, C. Si, Y . Jiang, Y . Zhang, T. Wu, Q. Jin, N. Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21807–21818, 2024

    2024

  12. [20]

    Y . Jin, Z. Sun, N. Li, K. Xu, H. Jiang, N. Zhuang, Q. Huang, Y . Song, Y . Mu, and Z. Lin. Pyramidal flow matching for efficient video generative modeling.arXiv preprint arXiv:2410.05954, 2024. 20

    2024

  13. [21]

    J. Kim, J. Kang, J. Choi, and B. Han. Fifo-diffusion: Generating infinite videos from text without training. arXiv preprint arXiv:2405.11473, 2024

    2024 arXiv

  14. [22]

    W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models.arXiv preprint arXiv:2412.03603, 2024

    2024 arXiv

  15. [23]

    B. Lin, Y . Ge, X. Cheng, Z. Li, B. Zhu, S. Wang, X. He, Y . Ye, S. Yuan, L. Chen, et al. Open-sora plan: Open-source large video generation model.arXiv preprint arXiv:2412.00131, 2024

    2024 arXiv

  16. [24]

    F. Liu, W. Sun, H. Wang, Y . Wang, H. Sun, J. Ye, J. Zhang, and Y . Duan. Reconx: Reconstruct any scene from sparse views with video diffusion model.arXiv preprint arXiv:2408.16767, 2024

    2024 arXiv

  17. [25]

    G. Ma, H. Huang, K. Yan, L. Chen, N. Duan, S. Yin, C. Wan, R. Ming, X. Song, X. Chen, et al. Step- video-t2v technical report: The practice, challenges, and future of video foundation model.arXiv preprint arXiv:2502.10248, 2025

    2025 arXiv

  18. [26]

    K. Nan, R. Xie, P. Zhou, T. Fan, Z. Yang, Z. Chen, X. Li, J. Yang, and Y . Tai. Openvid-1m: A large-scale high-quality dataset for text-to-video generation.arXiv preprint arXiv:2407.02371, 2024

    2024 arXiv

  19. [27]

    Parker-Holder, P

    J. Parker-Holder, P. Ball, J. Bruce, V . Dasagi, K. Holsheimer, C. Kaplanis, A. Moufarek, G. Scully, J. Shar, J. Shi, S. Spencer, J. Yung, M. Dennis, S. Kenjeyev, S. Long, V . Mnih, H. Chan, M. Gazeau, B. Li, F. Pardo, L. Wang, L. Zhang, F. Besse, T. Harley, A. Mitenkova, J. W...

    2024

  20. [28]

    Peebles and S

    W. Peebles and S. Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  21. [29]

    X. Peng, Z. Zheng, C. Shen, T. Young, X. Guo, B. Wang, H. Xu, H. Liu, M. Jiang, W. Li, et al. Open-sora 2.0: Training a commercial-level video generation model in $200 k.arXiv preprint arXiv:2503.09642, 2025

    2025 arXiv

  22. [30]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of Machine Learning Research, 21(140):1–67, 2020. URLhttp://jmlr.org/papers/v21/20-074.html

    2020

  23. [31]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  24. [32]

    Magi-1: Autoregressive video generation at scale, 2025

    Sand-AI. Magi-1: Autoregressive video generation at scale, 2025. URL https://static.magi.world/ static/files/MAGI_1.pdf

    2025

  25. [33]

    Siarohin, S

    A. Siarohin, S. Lathuilière, S. Tulyakov, E. Ricci, and N. Sebe. First order motion model for image animation.Advances in neural information processing systems, 32, 2019

    2019

  26. [34]

    Singer, A

    U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni, et al. Make-a-video: Text-to-video generation without text-video data.arXiv preprint arXiv:2209.14792, 2022

    2022 arXiv

  27. [35]

    J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  28. [36]

    K. Song, B. Chen, M. Simchowitz, Y . Du, R. Tedrake, and V . Sitzmann. History-guided video diffusion. arXiv preprint arXiv:2502.06764, 2025

    2025 arXiv

  29. [37]

    W. Sun, S. Chen, F. Liu, Z. Chen, Y . Duan, J. Zhang, and Y . Wang. Dimensionx: Create any 3d and 4d scenes from a single image with controllable video diffusion.arXiv preprint arXiv:2411.04928, 2024

    2024 arXiv

  30. [38]

    Teed and J

    Z. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020

    2020

  31. [39]

    Valevski, Y

    D. Valevski, Y . Leviathan, M. Arar, and S. Fruchter. Diffusion models are real-time game engines.arXiv preprint arXiv:2408.14837, 2024

    2024 arXiv

  32. [40]

    Van Den Oord, O

    A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning.Advances in neural information processing systems, 30, 2017. 21

    2017

  33. [41]

    A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, J. Wang, J. Zhang, J. Zhou, J. Wang, J. Chen, K. Zhu, K. Zhao, K. Yan, L. Huang, M. Feng, N. Zhang, P. Li, P. Wu, R. Chu, R. Feng, S. Zhang, S. Sun, T. Fang, T. Wang, T. Gui, T. Weng, T. Shen...

    2025 arXiv

  34. [42]

    C. Wu, J. Liang, X. Hu, Z. Gan, J. Wang, L. Wang, Z. Liu, Y . Fang, and N. Duan. Nuwa-infinity: Autore- gressive over autoregressive generation for infinite visual synthesis.arXiv preprint arXiv:2207.09814, 2022

    2022 arXiv

  35. [43]

    T. Wu, Z. Fan, X. Liu, H.-T. Zheng, Y . Gong, J. Jiao, J. Li, J. Guo, N. Duan, W. Chen, et al. Ar-diffusion: Auto-regressive diffusion model for text generation.Advances in Neural Information Processing Systems, 36:39957–39974, 2023

    2023

  36. [44]

    Z. Xiao, Y . Lan, Y . Zhou, W. Ouyang, S. Yang, Y . Zeng, and X. Pan. Worldmem: Long-term consistent world simulation with memory.arXiv preprint arXiv:2504.12369, 2025

    2025

  37. [45]

    D. Xie, Z. Xu, Y . Hong, H. Tan, D. Liu, F. Liu, A. Kaufman, and Y . Zhou. Progressive autoregressive video diffusion models.arXiv preprint arXiv:2410.08151, 2024

    2024 arXiv

  38. [46]

    J. Xing, M. Xia, Y . Zhang, H. Chen, W. Yu, H. Liu, G. Liu, X. Wang, Y . Shan, and T.-T. Wong. Dynami- crafter: Animating open-domain images with video diffusion priors. InEuropean Conference on Computer Vision, pages 399–417. Springer, 2024

    2024

  39. [47]

    M. Yang, J. Li, Z. Fang, S. Chen, Y . Yu, Q. Fu, W. Yang, and D. Ye. Playable game generation.arXiv preprint arXiv:2412.00887, 2024

    2024 arXiv

  40. [48]

    Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y . Yang, W. Hong, X. Zhang, G. Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer.arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  41. [49]

    S. Ye, J. Jang, B. Jeon, S. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y .-W. Chao, B. Y . Lin, et al. Latent action pretraining from videos.arXiv preprint arXiv:2410.11758, 2024

    2024 arXiv

  42. [50]

    T. Yin, Q. Zhang, R. Zhang, W. T. Freeman, F. Durand, E. Shechtman, and X. Huang. From slow bidirectional to fast autoregressive video diffusion models.arXiv preprint arXiv:2412.07772, 2, 2024

    2024

  43. [51]

    J. Yu, Y . Qin, X. Wang, P. Wan, D. Zhang, and X. Liu. Gamefactory: Creating new games with generative interactive videos.arXiv preprint arXiv:2501.08325, 2025

    2025

  44. [52]

    Zhang, C

    Y . Zhang, C. Peng, B. Wang, P. Wang, Q. Zhu, Z. Gao, E. Li, Y . Liu, and Y . Zhou. Matrix-game: Interactive world foundation model.arXiv, 2025

    2025

  45. [53]

    Zheng, X

    Z. Zheng, X. Peng, T. Yang, C. Shen, S. Li, H. Liu, Y . Zhou, T. Li, and Y . You. Open-sora: Democratizing efficient video production for all.arXiv preprint arXiv:2412.20404, 2024. 22

    2024 arXiv

  46. [2024]

    URLhttps://oasis-model.github.io/

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.