{"id":"d0c90563-fe2c-489c-9735-0549e78466da","arxiv_id":"2506.18901","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A chunk-wise video diffusion model trained on labeled game data plus unlabeled real footage transfers game-style control commands to real-world entities.","lead":"RealPlay turns video-generating AI into an interactive engine, letting a user steer a photoreal video scene with simple commands like \"turn left\". It learns control from a car-racing game and applies that control to real-world videos of vehicles, bicycles, and pedestrians, without needing action labels on the real footage.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 90% control-transfer claim rests on an unverified CFG-style equivalence and a human-only evaluation; without objective action discrimination or inter-annotator agreement, the observed accuracy could reflect motion priors rather than genuine game-to-real label transfer.","rationale":"The paper makes a useful engineering contribution: the chunk-wise adaptation of CogVideoX, noisy conditioning, and the mixed game/real training recipe are concrete and the ablations in Tables 2–4 and Figures 3–6 give some internal support. I am not recommending rejection. However, the central empirical claim is 'control transfer without real-world action labels,' and the only quantitative evidence for it is a human-judged success rate. The Section 3.2 CFG explanation is explicitly 'intuitive' and has a distribution mismatch: conditional and unconditional branches are trained on different domains, so it cannot by itself validate the observed transfer. The proposed objective trajectory test would disambiguate genuine action-specific control from a forward-motion prior or evaluator expectation. The reader's conditional verdict already captures the appropriate level of caution, and my concern is essentially the same one identified in the reader's weakest assumption, so I do not move the verdict.","tokens_in":16260,"tokens_out":10668,"duration_ms":123881,"concrete_test":"Run a decisive objective-action test: fix 30 real initial frames (10 per entity), generate 10 videos per action with RealPlay-AdaLN using the same initial frame for forward, left, and right, then classify the dominant entity/camera trajectory using SAM-2 segmentation plus RAFT optical flow as forward/left/right. Separately, have two independent annotators label each generated video and report per-action accuracy plus Cohen's kappa. If the three-condition trajectory distributions are not clearly separable (per-action classification not well above the forward-motion base rate) or annotator agreement is below 0.6, the 90% control rate does not establish that game labels, rather than the real-scene motion prior, drive the output.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RealPlay's central claim — that game-only action labels transfer to real-world vehicles, bicycles, and pedestrians — depends on two things that are not established. First, Section 3.2 explains transfer by treating unlabeled real videos (zero action) as CFG condition-dropped samples and labeled game videos as conditional samples. But standard CFG trains conditional and unconditional branches on the same data distribution with a random drop; here the conditional branch only sees game scenes while the zero-action branch only sees real scenes, and no action-drop probability or CFG scale is reported in Section 4. For a real input, the conditional prediction is out-of-distribution, so CFG interpolation is not a well-defined steering mechanism. Second, the headline evidence is a 90% human-judged control success rate with no error bars, no evaluator count, no inter-annotator agreement, and no objective trajectory metric. The real training videos are predominantly forward-moving and the action set has only three directions; Table 3 shows bicycle control jumps from 0% with game data alone to 72.5% once real bicycle videos are added, which is consistent with the model learning a strong real-scene motion prior plus a generic action bias rather than a faithful mapping from game commands to real-world motion. If the CFG equivalence fails and the evaluation is confounded, the central claim is unsupported, and the recipe might not generalize beyond the specific game/real pairing and entities tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"RealPlay adapts CogVideoX-5B into a chunk-wise interactive video generator. In stage 1 it trains a chunk-conditioned model on general-domain video data; in stage 2 it fine-tunes on labeled game data from Forza Horizon 5 (three actions: forward, left, right) plus unlabeled real-world videos of vehicles, bicycles, and pedestrians, using a zero action vector for real clips and injecting actions via adaptive LayerNorm. The paper claims two transfer phenomena: control transfer from game commands to real-world entities, and entity transfer to bicycles and pedestrians, with a 90% human-judged control success rate and an Elo of 1184. It also reports ablations on per-entity data, game data quantity, action injection strategy, and the number of video latents per chunk.","tokens_in":16556,"tokens_out":7582,"duration_ms":80693,"significance":"If the empirical claim holds, this is a meaningful result for interactive video generation: it suggests that game-derived action labels can be combined with unlabeled real footage to build controllable photorealistic video, avoiding expensive real-world action annotation. The paper includes useful comparisons against text-prompted video generators, a human-labeled real-data baseline, and a LAPA pseudo-label baseline, and the per-entity and cross-entity ablations are informative. I see no circularity in the evaluation: control success is judged on generated videos and is not fitted into the model. However, the central quantitative evidence is a single human-judged number without error bars or objective corroboration, and the proposed CFG-based explanation for transfer is not substantiated; these gaps are fixable but currently limit confidence in the generalization claim.","major_comments":[{"comment":"The paper reports the headline 90.0% control success rate as the primary evidence for its central claim, but the evaluation is described only as 'human evaluation' with no number of evaluators, no number of judged clips, no confidence interval or error bar, no inter-annotator agreement, and no per-action confusion matrix. Because the action set has only three labels and the real training videos are unlabeled, the measured accuracy could in part reflect a strong forward-motion prior and a generic 'move' bias rather than a faithful per-command mapping. The authors should report the full evaluation protocol, per-action results for forward/left/right, confidence intervals, agreement statistics, and at least one objective trajectory metric, such as optical-flow direction of the segmented entity or camera motion, computed on held-out prompts.","section":"Section 4, Table 1"},{"comment":"The claimed equivalence to classifier-free guidance is not established. Standard CFG trains conditional and unconditional predictions on the same data distribution with a random drop probability and interpolates at inference with a guidance scale; here the zero action vector is used for every unlabeled real sample, the conditional branch is trained only on game scenes, and no action-drop probability or CFG scale is reported in Section 4. For a real-world input, the conditional prediction is therefore out-of-distribution, so the proposed 'CFG interpolation' story does not define a steering mechanism. The passage is an analogy, not a derivation; it should be labeled as speculation or supported by experiments that vary the training drop probability and guidance scale and that measure whether the real-branch behavior depends on them.","section":"Section 3.2"},{"comment":"The bicycle ablation shows that control success jumps from 0% with game data alone to 72.5% after adding unlabeled real bicycle videos, and to 91.7% after adding all real entity types. This is consistent with an alternative explanation: the unlabeled real data supplies a real-scene motion prior and entity appearance, and the apparent action compliance is a generic response of the camera/entity to the action embedding rather than a game-to-real transfer of per-command semantics. To isolate the role of the game labels, the authors should compare against a control trained with the same real data but with shuffled or random game action labels, and/or evaluate on real videos with human-annotated ground-truth directions and report per-action accuracy. Without such a control, the central claim that the game-provided labels are what transfers is not uniquely supported.","section":"Section 4.2, Table 3"},{"comment":"The human-labeled real-data baseline is trained on only about 20% of the real dataset, namely the 'clear' samples, while RealPlay is trained on all unlabeled real data plus 80K labeled game samples. The comparison therefore confounds annotation quality with data quantity and with the presence of game data. The authors should add a controlled comparison in which the human-labeled baseline also receives the game data, or in which RealPlay uses a matched subset of real data, so that the 90% versus 58.9% gap can be attributed to the proposed recipe rather than to dataset size.","section":"Section 4.1, CogVideoX-5B (Human) baseline"}],"minor_comments":[{"comment":"The notation C_k + a_k -> C_{k+1} is figurative: the action is injected via adaptive LayerNorm, not added to the frame latents. Please clarify this to avoid a literal reading.","section":"Section 3.2"},{"comment":"The phrasing 'trained only with car-based game supervision' is imprecise: the action labels are car-based, but the training data also include unlabeled real-world videos of bicycles and pedestrians. Please restate this to distinguish action-label supervision from visual data.","section":"Abstract and Section 5"},{"comment":"The paper does not describe how evaluation prompts and videos are selected or whether the evaluated scenes are disjoint from the training sequences; a sentence on the evaluation split would strengthen the generalization claim.","section":"Section 4"},{"comment":"No ablation is provided for the noise augmentation component even though it is presented as a key remedy for iterative inconsistency; a with/without comparison would support the causal claim in the text.","section":"Section 3.1 and Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the paper is a reasonable fit for a vision venue. The main risk is that the headline 90% control rate is a single human-evaluation number with no objective validation, and the CFG-style explanation in Section 3.2 is an analogy rather than a tested mechanism. I would want the revision to add the quantitative evaluation and the controlled ablations described in the major comments before considering acceptance. I do not see a circularity problem in the current evaluation protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe thing to know about this paper is that the core empirical result is more interesting than the presentation. RealPlay trains a video diffusion model on labeled game clips from Forza Horizon 5 plus unlabeled real-world clips of vehicles, bicycles, and pedestrians, using a zero action vector for the real clips. At inference it steers those entities left/right/forward at about 90% human-judged success, beating a chunk-wise model trained on a subset of human-labeled real data (58.9%). The ablation in Table 3 is the strongest evidence: with game data only, the model cannot control a bicycle at all (0%, and the bicycle morphs into a game-style car); adding unlabeled real bicycle videos jumps to 72.5%. That is a genuine effect, and it is not present in the cited GameNGen/GameFactory line of work.\n\nWhat the paper does well is the recipe itself and the chunk-wise adaptation. The noise augmentation for iterative conditioning is sensible, and the cross-entity training analysis is honest about how data composition helps. The related-work coverage is appropriate, including the Matrix dataset and LAPA baseline.\n\nThe soft spots are real but not fatal. First, the headline metric is a human judgment with no reported number of evaluators, no inter-annotator agreement, no error bars, and no per-action breakdown. For a claim about control transfer, you want at least one objective trajectory metric, even on a subset. Second, the CFG-style explanation in Section 3.2 is not how the model is actually trained or used. No action-drop probability or CFG scale is reported, and the two 'branches' see different distributions (game with actions, real with zeros), so standard CFG theory does not apply. The authors should present this as an intuition, not a mechanism. Third, the entity transfer framing oversells: the appendix shows the model also controls the camera, and success is higher for entities with larger motion amplitude, which is consistent with a strong real-motion prior plus a generic action bias rather than a precise game-to-real mapping. The 0% game-only result weakens that alternative, but it does not eliminate it.\n\nWho is this for: people working on interactive video generation, world models, and sim-to-real transfer. It deserves a serious referee, but I would ask for a major revision: tighten the evaluation, add objective measures, and rewrite the transfer explanation.\n\nBest,","headline":"RealPlay shows a promising result—game-only action labels transferring to real-world control—but thin human-only evaluation and a loose CFG analogy leave the central claim under-supported.","tokens_in":17079,"tokens_out":4263,"would_cite":false,"duration_ms":43250,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A video diffusion model trained only on labeled car-racing game data and unlabeled real-world videos can control real vehicles, bicycles, and pedestrians in photorealistic generated video, without any real-world action labels.","keywords":["interactive video generation","video diffusion models","world models","game-to-real transfer","entity transfer","chunk-wise generation","classifier-free guidance","adaptive LayerNorm"],"falsifier":"A direct test is to retrain RealPlay with unlabeled real-world data whose motion does not resemble cars—say, fish or birds—or to replace the zero action vector with a fixed random vector; if control success on those entities collapses to chance, the claimed game-to-real transfer is not a general mechanism but a property of motion similarity or of the zero-vector trick.","tokens_in":16067,"feed_emoji":"🎮","tokens_out":8411,"duration_ms":74862,"temperature":0.7,"pith_summary":"The paper introduces RealPlay, an interactive video-generation system that acts like a playable engine for real-world footage. Its central claim is that a diffusion-based video generator can learn, from labeled car-racing game data plus unlabeled real-world videos, to respond to user commands—move forward, turn left, turn right—by producing short photorealistic video chunks, and that these commands transfer to real vehicles, bicycles, and pedestrians without ever being labeled on real data. The result matters because it suggests that high-fidelity, controllable simulation of the real world can be built from game supervision plus unlabeled video, avoiding the bottleneck of manual action annotation. RealPlay reports a 90 percent human-judged control success rate averaged across entities, with per-entity rates between 83 and 95 percent.","feed_headline":"Game-trained controls steer real-world bikes and pedestrians at 90%","feed_subtitle":"A video model trained on racing-game labels plus unlabeled footage transfers turn commands to real scenes.","key_machinery":"The central object is an action-conditioned chunk-wise video diffusion generator, adapted from a pre-trained image-to-video diffusion transformer. The adaptation replaces single-frame conditioning with chunk-level conditioning, uses an attention mask so the conditioning chunk only sees itself and the language tokens, reduces the generated horizon from 13 to 4 video latents per chunk, and adds noise to conditioning chunks during training so the model tolerates its own imperfect past outputs. A control module injects a one-hot action (move forward, turn left, turn right) through an adaptive LayerNorm: an MLP projects the action into a 512-dimensional feature, which is added to the timestep embedding and converted into scale and shift parameters that modulate the network activations. The unlabeled real-world videos are presented with the zero action vector, which the paper argues plays the role of condition-dropped samples in classifier-free guidance, teaching the model to associate game action labels with real-world motion.","core_discovery":"RealPlay is a neural network-based real-world game engine: it takes a user's control command, observes the current frame, and returns a short photorealistic video chunk that follows the command, then repeats the loop on the newly generated chunk. The paper's central discovery is that this interactive control can be learned with labeled game data alone as the action supervision: the model is trained on a car-racing game with recorded three-command actions together with unlabeled real-world clips of vehicles, bicycles, and pedestrians, using a zero action vector for the real clips. At inference, the game-origin commands steer the real-world entities with a human-judged success rate of 90 percent overall (83.3 for vehicles, 91.7 for bicycles, 95.0 for pedestrians), and the generated footage stays temporally consistent across iterations. The paper also reports that when no clear entity is centered, the camera itself follows the command, so the system learns a combined entity-and-camera control policy.","pith_inferences":["If the zero-vector/classifier-free-guidance equivalence is the true mechanism, the recipe should transfer to any pair of labeled simulator and unlabeled real footage with roughly matched motion distributions; a direct test would be swapping the racing game for a legged-robot simulator and checking whether control transfers to humanoid walking.","The entity transfer likely decomposes into game data supplying the action-to-motion mapping and unlabeled real data supplying appearance and plausible dynamics; if so, controlling a brand-new entity should require only a few unlabeled clips of that entity, not new labels.","The camera-follows-entity behavior suggests the control signal is a steering command to the whole observation (camera and subject) rather than an object-level action, so future designs might separate camera control from entity control to make the interaction more predictable."],"forward_implications":["A single source of labeled simulation data plus unlabeled real footage is enough to build an interactive photorealistic simulator, removing the need for expensive real-world action annotation.","Control accuracy tracks motion amplitude: entities with larger, more distinctive motions are easier to steer, so future real-world game engines should favor high-motion footage in their unlabeled training sets.","Adding unlabeled real-world data from other entities improves control of each individual entity, because the shared motion dynamics transfer across entity types.","Quality degrades as the number of generated chunks grows, and shrinks when each chunk is only one or a few video latents, so there is a latency-vs-quality tradeoff in interactive generation.","More labeled game samples consistently improve real-world control transfer, indicating that the practical bottleneck is the quantity of labeled simulation data, not the real-world footage."],"supporting_citations":[{"why":"Supplies the pre-trained image-to-video generator that RealPlay adapts into a chunk-wise generator.","marker":"[48]"},{"why":"Supplies the noise-augmentation strategy used to close the gap between clean training conditions and the model's own generated conditions.","marker":"[4]"},{"why":"Provides the labeled car-racing game dataset with recorded action annotations.","marker":"[9]"},{"why":"Provides the general-domain video corpus used to adapt the generator to chunk-wise generation.","marker":"[26]"},{"why":"Provides the classifier-free guidance view that the paper uses to explain why zero-action unlabeled real videos teach action transfer.","marker":"[15]"},{"why":"Provides the pseudo-labeling baseline that RealPlay's unlabeled-data training is compared against.","marker":"[49]"},{"why":"Supplies the benchmark metrics used to evaluate visual quality of generated chunks.","marker":"[19]"}],"fun_headline_variants":["Game-trained controls steer real-world scenes at 90% success","RealPlay: game-only action labels generalize to real-world control","From racing game to real-world entities: 90% control success","Interactive video engine: game controls drive bikes, pedestrians"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The recipe assumes that showing a real-world video with a zero action vector teaches the model the same thing as dropping the action condition in classifier-free guidance; if that equivalence is false, the reported transfer would need another explanation and may not extend to other game/real pairings.","fun_headline_variants_meta":{"raw":{"variants":["Game-trained controls steer real-world scenes at 90% success","RealPlay: game-only action labels generalize to real-world control","From racing game to real-world entities: 90% control success","Interactive video engine: game controls drive bikes, pedestrians"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1588,"prompt_tokens":924,"completion_tokens":664,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":594}},"tokens_in":540,"tokens_out":664,"duration_ms":6630,"temperature":1.0,"reasoning_tokens":594,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:40:48.253556+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to retrain RealPlay with unlabeled real-world data whose motion does not resemble cars—say, fish or birds—or to replace the zero action vector with a fixed random vector; if control success on those entities collapses to chance, the claimed game-to-real transfer is not a general mechanism but a property of motion similarity or of the zero-vector trick.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the noise-augmentation strategy used to close the gap between clean training conditions and the model's own generated conditions."}],"review_version":1}