{"id":"aa955077-eb47-4973-bf55-fbfc7108e5f3","arxiv_id":"2411.09153","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A two-stage robot manipulation policy that pre-trains a video diffusion transformer on Open X-Embodiment videos and then adapts it with a layer-wise self-attention adapter for action prediction, outperforming GR-1 on CALVIN.","lead":"VidMan trains a video diffusion model to predict future robot camera frames, then reuses that model with a lightweight adapter to predict robot actions. On the CALVIN benchmark it reports an 11.7% relative improvement over the strong GR-1 baseline, and larger offline gains on small robot datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5) feeds pure noise into the future-frame slots at action time, so the policy sees only history; the paper's ablations do not isolate whether the gains come from future-prediction dynamics or from backbone scale, the diffusion action head, and extra inputs.","rationale":"The paper is a serious engineering contribution: it reports a two-stage recipe, ablates several design choices, includes a GPT-style control, and provides offline numbers on multiple OXE subsets. The reader's CONDITIONAL verdict is fair. My stress-test targets the causal attribution in the central claim. Eq. (5) feeds pure noise into the future-frame slots at action time; the only information the policy receives is history, language, and pretrained weights. The pure_noise choice in Table 5 is an empirical input-format finding, not evidence that the features are dynamics-bearing. The most convincing evidence would be a pretraining-target ablation that keeps architecture, data, and compute fixed and varies only whether the first stage predicts future frames. Without that, the 11.7% CALVIN margin and the OXE gains could be attributed to the large STDiT-XL/2 backbone, the diffusion action head, or additional modalities (wrist camera, proprioception), not to the 'implicit dynamics' story. This does not invalidate the method; it narrows what the paper has established. The reader already made the verdict conditional on code, error bars, and in-distribution clarification; I would add the target-ablation and multi-seed reporting as acceptance conditions, so the verdict remains CONDITIONAL/UNCHANGED.","tokens_in":18129,"tokens_out":6644,"duration_ms":86810,"concrete_test":"Run a matched pretraining-target ablation: fix the Open-Sora backbone, the layer-wise adapter, the diffusion action head, the OXE data, training budget, and hyperparameters, and change only the stage-1 objective from future-frame denoising (Eq. 4) to masked reconstruction of the two observed historical frames using the same video tokenizer. Evaluate on CALVIN ABC->D with at least 3 seeds and report mean and standard deviation. If average task length stays within ~0.1 of 3.42, the central role of future-dynamics pretraining is not supported; if it drops toward the w/o-Pre-train level (~2.89) or below, the implicit-dynamics story is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that video-diffusion pretraining on future-frame prediction provides 'implicit dynamics' that improve action prediction. In the second stage, however, Eq. (5) replaces the future-frame tokens with pure Gaussian noise: Vaction = eps(theta,phi_ada)(V^K_c, y, K, Qaction), where V^K_c contains only historical frames and noise in the future slots. The only observation signal available to the policy is therefore history, language, and the pretrained weights. The paper's supporting evidence is empirical: Table 5 shows pure_noise outperforms no_noise and pure_zero on Bridge, and Table 2b shows OXE pretraining helps over no pretraining. But Table 5 tests the input placeholder format, not whether the intermediate features actually encode future-state information, and Table 2b does not isolate the future-prediction objective from large-scale visual pretraining on the same backbone. VidMan-GPT controls for the diffusion objective only partially and still uses a video-prediction objective. The reported CALVIN gain over GR-1 (3.42 vs 3.06) and the OXE offline gains could therefore come from the larger STDiT-XL/2 backbone, the diffusion action head, or the extra modalities (wrist camera, proprioception) rather than from the 'implicit dynamics' mechanism that the paper claims. Without a matched control that varies only the first-stage prediction target, the causal attribution at the center of the paper is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VidMan, a two-stage framework for robot manipulation. In the first stage, a video diffusion transformer (initialized from Open-Sora) is pretrained on the Open X-Embodiment dataset to predict future frames from history and language instructions. In the second stage, the same transformer is adapted with layer-wise self-attention adapters to output actions directly, with future slots filled by pure Gaussian noise instead of denoised frames. The authors report an 11.7% relative improvement over GR-1 on CALVIN average task length and over 9% precision gains over Octo-base on several OXE offline sub-datasets, with additional RLBench results. The main claim is that video-diffusion pretraining provides 'implicit dynamics' that improve action prediction precision.","tokens_in":18481,"tokens_out":4326,"duration_ms":42384,"significance":"If the central claim is established, the paper advances the use of video generative models for robot policy learning and provides a practical two-stage recipe (video-pretrain, then adapt with light-weight adapters). The paper is commendable for shipping extensive ablations: two-stage vs. co-training (Table 2a), pretraining data (Table 2b), adapter design (Table 2c), frame interval sensitivity (Table 3), placeholder type (Table 5), and history/future lengths (Table 6). It also compares against a GPT-style video-prediction baseline (VidMan-GPT), which is a useful control. However, the causal attribution of the gains to 'implicit dynamics' is not fully demonstrated, and some offline evaluations are in-distribution with respect to the pretraining data, which tempers the significance of the reported gains.","major_comments":[{"comment":"The action stage feeds pure Gaussian noise into the future-frame slots, so the policy at inference time has access only to historical frames, language, and the pretrained weights. Eq. (5) defines Vaction = epsilon_{theta, phi_ada}(V^K_c, y, K, Qaction) with V^K_c containing noise in the future slots. This means the 'implicit dynamics' from future-frame prediction cannot be directly used at action time; any benefit must come from the pretrained representations, not from observed future information. Table 5 tests the placeholder format (no_noise vs. pure_noise vs. pure_zero) but does not test whether the intermediate features actually encode future-state information. To support the central claim, the authors should provide a matched control that keeps the backbone and data fixed while varying only the first-stage objective (e.g., masked image modeling or direct action prediction without future-frame prediction), or demonstrate through probing that the adapted features contain future information that correlates with the performance gains.","section":"Section 4.2, Eq. (5)"},{"comment":"The offline evaluation is performed on four OXE sub-datasets (Bridge, Taco Play, Cable Routing, AUTOLab UR5) that are part of the stage-1 pretraining distribution. The paper explicitly states 'we evaluate performance on in-distribution tasks' (Section 5.2). Since the first stage is trained on OXE (Section 5.1.1), the evaluation subsets are potentially seen during pretraining, making the reported '9% precision gains' a measure of fit to the training distribution rather than generalization to novel embodiments or scenes. To support the claim of improved data utilization efficiency, the authors should either hold out the evaluation sub-datasets from stage-1 pretraining or provide results on truly held-out datasets beyond CALVIN. If the sub-datasets are included in pretraining, this should be stated clearly and the claim should be tempered accordingly.","section":"Section 5.2 and Figure 3 / Table 7"},{"comment":"The CALVIN results in Table 1 are reported as single numbers without variance across evaluation seeds. CALVIN rollouts are stochastic (initial state, environment variations), and the 11.7% relative improvement over GR-1 (3.42 vs. 3.06) could lie within run-to-run variance; previous works typically report mean and standard deviation over multiple evaluation seeds. Moreover, Section 5.1.1 states that VidMan additionally uses wrist-camera observations and proprioceptive state data in the second stage, while the comparison protocol for GR-1 and other baselines may not include these modalities. Please report variance over at least three seeds for VidMan and, if possible, for the baselines, and provide an ablation that removes the proprioception input or matches the exact input set of GR-1 to ensure the comparison is controlled.","section":"Section 5.1.2 and Table 1"}],"minor_comments":[{"comment":"The theoretical framing in Section 3 (Eqs. 1-3) is not used to derive the two-stage design; the connection between the posterior P(S|O) and the noise-in-future-slots stage 2 is left implicit. Consider clarifying how the theory motivates the specific design choices, or trim the theory to avoid overclaiming derivational value.","section":"Section 3"},{"comment":"The notation for epsilon is overloaded: it denotes both the noise variable in Eq. (4) and the network function (with subscripts) in Eq. (5). The subscript (theta, phi_ada) on epsilon in Eq. (5) is not defined in the text. Please use distinct symbols for the noise and the network to avoid confusion.","section":"Eq. (4) and Eq. (5)"},{"comment":"The frame sampling interval is selected based on performance on Bridge and CALVIN, which are also evaluation benchmarks. This constitutes tuning on the test distribution and should be acknowledged; the sensitivity of the main CALVIN and offline results to this hyperparameter should be discussed to assess robustness.","section":"Section 5.3 and Table 3"},{"comment":"The 'no_noise' condition in Table 5 is unclear: does it feed the ground-truth clean future frames at training and test time? If clean future frames are provided at test time, that would be an unrealistic but easier setting; please specify what inputs are used in each row.","section":"Appendix A.4, Table 5"},{"comment":"The abstract states 'over 9% precision gains on the OXE small-scale dataset', but Figure 3 shows gains of 9.9% on Cable Routing and 9.0% on Autolab UR5 relative to Octo-base. Please state the baseline explicitly in the abstract and clarify that these are relative improvements on in-distribution sub-datasets.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper has strong experimental breadth and a clear two-stage recipe, but the central attribution of gains to video-diffusion 'implicit dynamics' is not yet established, and the offline evaluation protocol raises questions about in-distribution evaluation. The CALVIN comparison may also be confounded by additional input modalities. I recommend a major revision that adds a matched pretraining-objective control, reports CALVIN variance, and clarifies or rectifies the in-distribution offline evaluation. These are feasible within the scope of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, practical recipe. The two-stage design—Open-Sora video diffusion pretraining on OXE for future-frame prediction, then a layer-wise adapter plus diffusion action head for action prediction—is genuinely new. The headline CALVIN gain over GR-1 (3.42 vs 3.06 average length) is plausible, though the paper reports no error bars. The OXE offline gains are real but partly in-distribution, since both stages train on OXE and evaluation is on the same sub-datasets.\n\nWhat the paper does well: the ablations are thoughtful. The two-stage vs co-training comparison, the pretraining-type comparison (none/Ego4d/OXE), and the adapter on/off study all support the design choices. The VidMan-GPT baseline is the most valuable control: it uses the same STDiT-XL/2 backbone and a GPT-style next-frame prediction objective, so beating it on all four OXE sets suggests the diffusion multi-frame objective adds something beyond parameter count or action head. The pure-noise placeholder in stage 2 is a neat trick and Table 5 shows it clearly beats zero or no-noise alternatives.\n\nSoft spots, in proportion: the missing variance on CALVIN is a real weakness—for a benchmark where a few seeds can move the average by 0.1, readers need to know the spread. The OXE offline evaluation is explicitly described as in-distribution; the small-dataset gains (Cable Routing, Autolab UR5) are expected if you simply train longer on those sets, though the Bridge result is more convincing. The mechanism claim is undersupported: the paper never shows what the adapter extracts or why the future-prediction target helps beyond providing a large-scale visual pretraining signal. Section 3's Bayes/inverse-dynamics framing is heuristic, not a derivation.\n\nOn the stress-test note: the worry that gains could come from backbone scale, the action head, or extra inputs is partially unfair because VidMan-GPT controls for those factors. However, it does not isolate the future-prediction target from the diffusion schedule itself, and the paper does not demonstrate that the intermediate features encode future-state information. So the central attribution to 'implicit dynamics' remains an inference, not a demonstrated fact.\n\nThis is for the robot imitation learning community. It is a useful recipe, not a paradigm shift. I would send it to peer review. The revision needs: error bars on CALVIN, code release, a clear statement about the in-distribution OXE evaluation, and ideally a control that varies only the first-stage prediction target (e.g., predicting the same future frames with an MSE loss instead of diffusion). If those are addressed, it is a solid accept.","headline":"A credible two-stage video-diffusion recipe for robot manipulation that beats GR-1 on CALVIN, but the 'implicit dynamics' claim is not fully proven and the offline OXE evidence is partly in-distribution.","tokens_in":761,"tokens_out":842,"would_cite":true,"duration_ms":49549,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VidMan pretrains a video diffusion transformer on robot video to forecast future frames, then reads actions directly from that transformer with a layer-wise adapter, beating GR-1 by 11.7% on CALVIN and improving offline accuracy by up to…","keywords":["robot manipulation","video diffusion model","world model","inverse dynamics","imitation learning","self-attention adapter","Open X-Embodiment","CALVIN benchmark"],"falsifier":"Train the exact same layer-wise adapter and diffusion action head on a same-size transformer that was pretrained to predict the next frame without a diffusion objective (or on a randomly initialized transformer), keeping all data and hyperparameters fixed; if offline action accuracy on Taco Play or Cable Routing does not drop materially, then the diffusion-based dynamics pretraining is not the causal ingredient.","tokens_in":17941,"feed_emoji":"🤖","tokens_out":8624,"duration_ms":87593,"temperature":0.7,"pith_summary":"The paper tries to show that a video diffusion model trained to predict future frames from robot trajectories carries reusable knowledge of how scenes change, and that this knowledge can be turned into a manipulation policy without running the slow denoising process at deployment. VidMan first pretrains a video generation transformer on the Open X-Embodiment dataset to forecast future camera views, then inserts a lightweight layer-wise self-attention adapter that reads actions out of the same transformer while the future-frame slots are filled with pure noise. On the CALVIN benchmark the adapted policy completes an average of 3.42 sequential tasks versus 3.06 for the GR-1 baseline, a relative gain of 11.7%, and on small OXE subtasks it improves offline action accuracy by roughly 9 to 10 percentage points over Octo-base. If the claim holds, video world models can serve as data-efficient initialization for robot policies, especially when target-robot data is scarce.","feed_headline":"Video diffusion pretraining lifts robot action accuracy 11.7%","feed_subtitle":"VidMan turns a video diffusion model into a fast action policy, beating GR-1 on CALVIN and raising offline accuracy.","key_machinery":"The load-bearing object is the layer-wise self-attention adapter: after each transformer block, a small module with multi-head self-attention and a gated feed-forward network receives the visual tokens plus a set of learnable action queries, and fuses each layer's features into action embeddings. In the second stage the diffusion step is fixed at the maximum $k=K$, so the future-frame tokens entering the transformer are pure noise rather than a noised video, which makes the adapted model an inverse dynamics model $P(a \\mid s_t, s_{t+1})$ over implicit states. These action embeddings are decoded by a small diffusion-based action head, so the expensive video denoising loop never runs at inference time.","core_discovery":"VidMan's central claim is that a single video diffusion transformer can be used twice: first as a dynamics-aware visionary model that learns long-horizon state transitions from heterogeneous robot video, and then as an implicit inverse dynamics model that maps observations and language to actions. The second use is enabled by replacing the denoising input for future frames with pure Gaussian noise, so the transformer's iterative generation is disabled, and by attaching a layer-wise self-attention adapter that pools action-relevant information from every transformer layer into learnable action tokens. The paper argues that this preserves the pre-trained dynamics knowledge while avoiding the cost and bias of generating full future images, and the ablation evidence shows that co-training action and video losses at the same time hurts performance, while a single action-only fine-tuning stage works better.","pith_inferences":["The paper does not show what the adapter's action queries attend to; a natural probe would be to visualize attention between action tokens and spatial positions, testing whether the 'implicit dynamics' are object motion, arm kinematics, or scene layout.","Because the frame-interval ablation shows a sweet spot at interval 3, one testable extension is a learned or per-task sampling interval; faster tasks might want shorter horizons and slower tasks longer ones.","A cleaner causal check than the paper's ablations would replace the video diffusion transformer with a same-capacity non-diffusion video predictor; if gains persist, the diffusion denoising objective may not be the operative ingredient."],"forward_implications":["The same pretrained video transformer can be converted into a manipulation policy that completes longer instruction chains: on CALVIN, average length rises from 3.06 (GR-1) to 3.42, a relative 11.7% gain.","Two-stage training with action supervision alone beats co-training video and action losses: co-training drops average length from 3.42 to 2.70, so the recipe is not just a bigger model with more losses.","The policy runs at one forward pass with no iterative denoising, so exploiting video-diffusion dynamics need not sacrifice control frequency.","Robot-specific video pretraining, not general web video, is the productive source: OXE pretraining adds 0.53 to CALVIN average length over no pretraining, while Ego4d pretraining slightly hurts.","The layer-wise adapter is necessary: removing it drops CALVIN average length from 3.42 to 1.54, and unfreezing the backbone gives a further gain over freezing (3.42 vs 2.98)."],"supporting_citations":[{"why":"provides the heterogeneous robot trajectories used for stage-1 future-frame pretraining.","marker":"[8]"},{"why":"supplies the video diffusion transformer backbone and its internet-video initialization.","marker":"[21]"},{"why":"is the CALVIN baseline whose 3.06 average length VidMan improves to 3.42.","marker":"[9]"},{"why":"defines the offline evaluation protocol on the four OXE subtasks and serves as the main generalist-policy baseline.","marker":"[7]"},{"why":"is the source of the layer-wise self-attention adapter with gated feed-forward network.","marker":"[22]"},{"why":"provides the diffusion-based action head that decodes action embeddings into control signals.","marker":"[51]"},{"why":"supplies the CALVIN benchmark and its language-labelled training data for the action stage.","marker":"[14]"},{"why":"provides the pretrained video tokenizer that encodes image trajectories into video tokens.","marker":"[49]"}],"fun_headline_variants":["Video diffusion model improves robot action prediction by 11.7%","VidMan turns video dynamics into robot manipulation skills","Pretrained video diffusion boosts robot accuracy on CALVIN","Implicit dynamics from video diffusion enhance robot control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method works only if the intermediate features of a video diffusion transformer, with the future-frame slots replaced by pure noise, genuinely encode reusable implicit dynamics rather than merely providing a large network whose extra capacity carries the gains.","fun_headline_variants_meta":{"raw":{"variants":["Video diffusion model improves robot action prediction by 11.7%","VidMan turns video dynamics into robot manipulation skills","Pretrained video diffusion boosts robot accuracy on CALVIN","Implicit dynamics from video diffusion enhance robot control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00026,"raw_usage":{"total_tokens":1599,"prompt_tokens":968,"completion_tokens":631,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":565}},"tokens_in":584,"tokens_out":631,"duration_ms":6469,"temperature":1.0,"reasoning_tokens":565,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:57:46.185018+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the exact same layer-wise adapter and diffusion action head on a same-size transformer that was pretrained to predict the next frame without a diffusion objective (or on a randomly initialized transformer), keeping all data and hyperparameters fixed; if offline action accuracy on Taco Play or Cable Routing does not drop materially, then the diffusion-based dynamics pretraining is not the causal ingredient.","supporting_citations":[{"cited_title":"Open-sora: Democratizing efficient video production for all, March 2024","cited_arxiv_id":null,"evidence_quote":"supplies the video diffusion transformer backbone and its internet-video initialization."}],"review_version":1}