Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VidMan pretrains a video diffusion transformer on robot video to forecast future frames, then reads actions directly from that transformer with a layer-wise adapter, beating GR-1 by 11.7% on CALVIN and improving offline accuracy by up to…

desk verdict A credible two-stage video-diffusion recipe for robot manipulation that beats GR-1 on CALVIN, but the 'implicit dynamics' claim is not fully proven and the offline OXE evidence is partly in-distribution. read the letter →

arxiv 2411.09153 v1 pith:T6EWCNPG submitted 2024-11-14 cs.CV cs.RO

Youpeng Wen , Junfan Lin , Yi Zhu , Jianhua Han , Hang Xu , Shen Zhao , Xiaodan Liang This is my paper · ORCID
classification cs.CVcs.RO
keywords robotmanipulationvideodiffusionmodelworldinversedynamicsimitationlearningself-attentionadapterOpenX-EmbodimentCALVINbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a video diffusion model trained to predict future frames from robot trajectories carries reusable knowledge of how scenes change, and that this knowledge can be turned into a manipulation policy without running the slow denoising process at deployment. VidMan first pretrains a video generation transformer on the Open X-Embodiment dataset to forecast future camera views, then inserts a lightweight layer-wise self-attention adapter that reads actions out of the same transformer while the future-frame slots are filled with pure noise. On the CALVIN benchmark the adapted policy completes an average of 3.42 sequential tasks versus 3.06 for the GR-1 baseline, a relative gain of 11.7%, and on small OXE subtasks it improves offline action accuracy by roughly 9 to 10 percentage points over Octo-base. If the claim holds, video world models can serve as data-efficient initialization for robot policies, especially when target-robot data is scarce.

What carries the argument

The load-bearing object is the layer-wise self-attention adapter: after each transformer block, a small module with multi-head self-attention and a gated feed-forward network receives the visual tokens plus a set of learnable action queries, and fuses each layer's features into action embeddings. In the second stage the diffusion step is fixed at the maximum $k=K$, so the future-frame tokens entering the transformer are pure noise rather than a noised video, which makes the adapted model an inverse dynamics model $P(a \mid s_t, s_{t+1})$ over implicit states. These action embeddings are decoded by a small diffusion-based action head, so the expensive video denoising loop never runs at inference time.

What would settle it

Train the exact same layer-wise adapter and diffusion action head on a same-size transformer that was pretrained to predict the next frame without a diffusion objective (or on a randomly initialized transformer), keeping all data and hyperparameters fixed; if offline action accuracy on Taco Play or Cable Routing does not drop materially, then the diffusion-based dynamics pretraining is not the causal ingredient.

Watch

Extended reading notes

Core claim

VidMan's central claim is that a single video diffusion transformer can be used twice: first as a dynamics-aware visionary model that learns long-horizon state transitions from heterogeneous robot video, and then as an implicit inverse dynamics model that maps observations and language to actions. The second use is enabled by replacing the denoising input for future frames with pure Gaussian noise, so the transformer's iterative generation is disabled, and by attaching a layer-wise self-attention adapter that pools action-relevant information from every transformer layer into learnable action tokens. The paper argues that this preserves the pre-trained dynamics knowledge while avoiding the cost and bias of generating full future images, and the ablation evidence shows that co-training action and video losses at the same time hurts performance, while a single action-only fine-tuning stage works better.

Load-bearing premise

The method works only if the intermediate features of a video diffusion transformer, with the future-frame slots replaced by pure noise, genuinely encode reusable implicit dynamics rather than merely providing a large network whose extra capacity carries the gains.

Editorial extensions

If this is right

  • The same pretrained video transformer can be converted into a manipulation policy that completes longer instruction chains: on CALVIN, average length rises from 3.06 (GR-1) to 3.42, a relative 11.7% gain.
  • Two-stage training with action supervision alone beats co-training video and action losses: co-training drops average length from 3.42 to 2.70, so the recipe is not just a bigger model with more losses.
  • The policy runs at one forward pass with no iterative denoising, so exploiting video-diffusion dynamics need not sacrifice control frequency.
  • Robot-specific video pretraining, not general web video, is the productive source: OXE pretraining adds 0.53 to CALVIN average length over no pretraining, while Ego4d pretraining slightly hurts.
  • The layer-wise adapter is necessary: removing it drops CALVIN average length from 3.42 to 1.54, and unfreezing the backbone gives a further gain over freezing (3.42 vs 2.98).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not show what the adapter's action queries attend to; a natural probe would be to visualize attention between action tokens and spatial positions, testing whether the 'implicit dynamics' are object motion, arm kinematics, or scene layout.
  • Because the frame-interval ablation shows a sweet spot at interval 3, one testable extension is a learned or per-task sampling interval; faster tasks might want shorter horizons and slower tasks longer ones.
  • A cleaner causal check than the paper's ablations would replace the video diffusion transformer with a same-capacity non-diffusion video predictor; if gains persist, the diffusion denoising objective may not be the operative ingredient.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes VidMan, a two-stage framework for robot manipulation. In the first stage, a video diffusion transformer (initialized from Open-Sora) is pretrained on the Open X-Embodiment dataset to predict future frames from history and language instructions. In the second stage, the same transformer is adapted with layer-wise self-attention adapters to output actions directly, with future slots filled by pure Gaussian noise instead of denoised frames. The authors report an 11.7% relative improvement over GR-1 on CALVIN average task length and over 9% precision gains over Octo-base on several OXE offline sub-datasets, with additional RLBench results. The main claim is that video-diffusion pretraining provides 'implicit dynamics' that improve action prediction precision.

Significance. If the central claim is established, the paper advances the use of video generative models for robot policy learning and provides a practical two-stage recipe (video-pretrain, then adapt with light-weight adapters). The paper is commendable for shipping extensive ablations: two-stage vs. co-training (Table 2a), pretraining data (Table 2b), adapter design (Table 2c), frame interval sensitivity (Table 3), placeholder type (Table 5), and history/future lengths (Table 6). It also compares against a GPT-style video-prediction baseline (VidMan-GPT), which is a useful control. However, the causal attribution of the gains to 'implicit dynamics' is not fully demonstrated, and some offline evaluations are in-distribution with respect to the pretraining data, which tempers the significance of the reported gains.

major comments (3)
  1. [Section 4.2, Eq. (5)] The action stage feeds pure Gaussian noise into the future-frame slots, so the policy at inference time has access only to historical frames, language, and the pretrained weights. Eq. (5) defines Vaction = epsilon_{theta, phi_ada}(V^K_c, y, K, Qaction) with V^K_c containing noise in the future slots. This means the 'implicit dynamics' from future-frame prediction cannot be directly used at action time; any benefit must come from the pretrained representations, not from observed future information. Table 5 tests the placeholder format (no_noise vs. pure_noise vs. pure_zero) but does not test whether the intermediate features actually encode future-state information. To support the central claim, the authors should provide a matched control that keeps the backbone and data fixed while varying only the first-stage objective (e.g., masked image modeling or direct action prediction without future-frame prediction), or demonstrate through probing that the adapted features contain future information that correlates with the performance gains.
  2. [Section 5.2 and Figure 3 / Table 7] The offline evaluation is performed on four OXE sub-datasets (Bridge, Taco Play, Cable Routing, AUTOLab UR5) that are part of the stage-1 pretraining distribution. The paper explicitly states 'we evaluate performance on in-distribution tasks' (Section 5.2). Since the first stage is trained on OXE (Section 5.1.1), the evaluation subsets are potentially seen during pretraining, making the reported '9% precision gains' a measure of fit to the training distribution rather than generalization to novel embodiments or scenes. To support the claim of improved data utilization efficiency, the authors should either hold out the evaluation sub-datasets from stage-1 pretraining or provide results on truly held-out datasets beyond CALVIN. If the sub-datasets are included in pretraining, this should be stated clearly and the claim should be tempered accordingly.
  3. [Section 5.1.2 and Table 1] The CALVIN results in Table 1 are reported as single numbers without variance across evaluation seeds. CALVIN rollouts are stochastic (initial state, environment variations), and the 11.7% relative improvement over GR-1 (3.42 vs. 3.06) could lie within run-to-run variance; previous works typically report mean and standard deviation over multiple evaluation seeds. Moreover, Section 5.1.1 states that VidMan additionally uses wrist-camera observations and proprioceptive state data in the second stage, while the comparison protocol for GR-1 and other baselines may not include these modalities. Please report variance over at least three seeds for VidMan and, if possible, for the baselines, and provide an ablation that removes the proprioception input or matches the exact input set of GR-1 to ensure the comparison is controlled.
minor comments (5)
  1. [Section 3] The theoretical framing in Section 3 (Eqs. 1-3) is not used to derive the two-stage design; the connection between the posterior P(S|O) and the noise-in-future-slots stage 2 is left implicit. Consider clarifying how the theory motivates the specific design choices, or trim the theory to avoid overclaiming derivational value.
  2. [Eq. (4) and Eq. (5)] The notation for epsilon is overloaded: it denotes both the noise variable in Eq. (4) and the network function (with subscripts) in Eq. (5). The subscript (theta, phi_ada) on epsilon in Eq. (5) is not defined in the text. Please use distinct symbols for the noise and the network to avoid confusion.
  3. [Section 5.3 and Table 3] The frame sampling interval is selected based on performance on Bridge and CALVIN, which are also evaluation benchmarks. This constitutes tuning on the test distribution and should be acknowledged; the sensitivity of the main CALVIN and offline results to this hyperparameter should be discussed to assess robustness.
  4. [Appendix A.4, Table 5] The 'no_noise' condition in Table 5 is unclear: does it feed the ground-truth clean future frames at training and test time? If clean future frames are provided at test time, that would be an unrealistic but easier setting; please specify what inputs are used in each row.
  5. [Abstract] The abstract states 'over 9% precision gains on the OXE small-scale dataset', but Figure 3 shows gains of 9.9% on Cable Routing and 9.0% on Autolab UR5 relative to Octo-base. Please state the baseline explicitly in the abstract and clarify that these are relative improvements on in-distribution sub-datasets.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central claims are anchored to external benchmarks and matched ablations; the pure-noise design and in-distribution OXE evaluation raise internal-validity concerns but do not make any reported result equivalent to its inputs by construction.

full rationale

The paper's main CALVIN result (Table 1) is an external benchmark comparison against published baselines, especially GR-1 and 3D Diffuser Actor, under a fixed zero-shot evaluation protocol; this is not a fitted quantity renamed as a prediction. The supporting ablations are matched controls within the paper: Table 2b compares w/o pretrain, w/ Ego4d, and w/ OXE under the same second-stage training; Table 2c removes the adapter; Table 2a compares co-training against action-only; Table 5 compares pure noise, zero embeddings, and no noise in the future slots. None of these controls defines the target metric in terms of the fitted parameters. The offline OXE evaluation is described by the paper as in-distribution, and the pretraining data overlap with the evaluation domains is a legitimate generalization concern, but the comparison is against Octo and RT-1-X trained on the same OXE data, so the reported margins are not forced by construction. Equation (5) indeed replaces future-frame tokens with pure Gaussian noise, meaning the policy observes only history plus the pretrained weights; this weakens the causal attribution of gains to 'implicit dynamics' and is a real internal-validity confound (backbone scale, diffusion action head, and extra input modalities are not fully controlled), but it is a correctness and experimental-design issue rather than circularity: no equation, parameter, or benchmark metric is defined in terms of the claimed conclusion. The paper also does not rely on load-bearing self-citations: Open-Sora, Flamingo, Diffusion Policy, GR-1, and OXE are all external prior work, and no uniqueness or force argument is imported from the authors' own prior results. Therefore the derivation chain is not circular, and the appropriate score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method rests on the transferability of video prediction pretraining and several hand-chosen design decisions; none of these are derived from first principles.

free parameters (6)
  • Frame sampling interval = 3
    Chosen via ablation in Table 3; affects both video prediction and action performance.
  • Number of history frames m = 2
    Selected based on experiments in Table 6; more history frames improve action prediction.
  • Number of future frames n = 2
    Selected based on experiments in Table 6; future frames matter less than history frames.
  • Fixed diffusion step K in stage 2 = 1000 (pure noise)
    Chosen via ablation in Table 5; pure noise input works best for the action stage.
  • Action prediction horizon = 12 (OXE), 10 (CALVIN)
    Set per dataset, not justified by derivation.
  • Diffusion action head steps L = 100
    Hyperparameter for the action diffusion head.
assumptions (5)
  • standard math Bayes' theorem and the probabilistic state-space formulation in Section 3
    Used to motivate the inverse dynamics view, but not used computationally.
  • domain assumption Open-Sora transformer pretrained on internet video transfers to robot video dynamics
    Initialization relies on this transfer (Section 5.1.1).
  • domain assumption OXE video pretraining is an effective source of dynamics knowledge
    The central claim of the first stage; supported only by ablations on CALVIN and in-distribution OXE.
  • domain assumption The video tokenizer and CLIP text encoder provide adequate representations
    Used without finetuning for video and language encoding.
  • ad hoc to paper Using pure noise instead of denoising in stage 2 activates the relevant features
    Empirically chosen (Table 5), no theoretical justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation." pith.science (2026). https://pith.science/paper/T6EWCNPG

@misc{pith2026241109153,
  author       = {Pith},
  title        = {Pith review of: VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T6EWCNPG}},
  note         = {Machine review of arXiv:2411.09153}
}
read the original abstract

Recent advancements utilizing large-scale video data for learning video generation models demonstrate significant potential in understanding complex physical dynamics. It suggests the feasibility of leveraging diverse robot trajectory data to develop a unified, dynamics-aware model to enhance robot manipulation. However, given the relatively small amount of available robot data, directly fitting data without considering the relationship between visual observations and actions could lead to suboptimal data utilization. To this end, we propose VidMan (Video Diffusion for Robot Manipulation), a novel framework that employs a two-stage training mechanism inspired by dual-process theory from neuroscience to enhance stability and improve data utilization efficiency. Specifically, in the first stage, VidMan is pre-trained on the Open X-Embodiment dataset (OXE) for predicting future visual trajectories in a video denoising diffusion manner, enabling the model to develop a long horizontal awareness of the environment's dynamics. In the second stage, a flexible yet effective layer-wise self-attention adapter is introduced to transform VidMan into an efficient inverse dynamics model that predicts action modulated by the implicit dynamics knowledge via parameter sharing. Our VidMan framework outperforms state-of-the-art baseline model GR-1 on the CALVIN benchmark, achieving a 11.7% relative improvement, and demonstrates over 9% precision gains on the OXE small-scale dataset. These results provide compelling evidence that world models can significantly enhance the precision of robot action prediction. Codes and models will be public.

Figures

Figures reproduced from arXiv: 2411.09153 by the authors.

Figure 1
Figure 1. VidMan’s two-stage training paradigm mirrors dual process theory: its first stage (like [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of VidMan. (a) We use Video Tokenizer to tokenize the uniform sampled robot visual trajectory Os to video tokens Vs. (b) In the 1st Stage, we concatenate the video tokens processed through the diffusion process with the historical tokens along the channel dimension to form V k c . V k c along with the language tokens and diffusion step k are fed into Open-Sora for video prediction training. In the 2nd Stage… view at source ↗
Figure 3
Figure 3. Offline Performance. The average accuracy (Avg xyz ang) of xyz accuracy and angle accuracy and MSE correspond to the left and right y-axes of the graph respectively. All models were trained on OXE and validated on offline performance across four datasets. VidMan outperformed Octo-base [7] by 5.6% on Bridge, 2.6% on Taco Play, 9.9% on Cable Routing, and 9.0% on Autolab UR5. Additionally, Our method also shows improve… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Efficiency comparison be￾tween two types of training. Effect of layer-wise adapter. To measure the effect of the layer-wise adapter, we removed it (w/o adapter) and directly extracted the observed information into the action policy head by concatenating learnable actio…
Figure 5
Figure 5. Figure 5: Our model utilizes a layer-wise adapter, which includes a self-attention layer and a feed [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Video prediction results on OXE. The images in yellow boxes are ground-truth images; the images in blue boxes are predicted images. The language instruction is placed below the image. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Offline action prediction results on OXE. The upper part of each group of images shows subsampled frames from an episode, while the lower part displays the true and predicted 7D pose results, including x, y, yaw, pitch, roll, and grasp over time. 19 [PITH_FULL_IMAGE:f…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SANTS: A State-Adaptive Scheduler for World Action Models

    cs.RO 2026-05 unverdicted novelty 6.0 of 10

    SANTS adaptively chooses denoising depth in video-based robot action diffusion policies using a state-dependent stopping hazard and noise ratio, trained via downstream action reward to reduce latency.

  2. AMPLIFY: Actionless Motion Priors for Robot Learning from Videos

    cs.RO 2025-06 conditional novelty 6.0 of 10

    A three-stage pipeline that turns keypoint tracks into discrete motion tokens, predicts them from action-free video, and decodes them into actions yields large few-shot and zero-shot policy improvements in robot manipulation.

  3. Native Video-Action Pretraining for Generalizable Robot Control

    cs.RO 2026-07 conditional novelty 5.0 of 10

    A video-action foundation model pretrained natively with a causal diffusion transformer and semantic visual-action tokenizer reports improved few-shot robot manipulation and 225 Hz asynchronous closed-loop control.

Reference graph

Works this paper leans on

67 extracted references · 18 canonical work pages · cited by 3 Pith papers

  1. [1]

    Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge

    Andy Zeng, Kuan-Ting Yu, Shuran Song, Daniel Suo, Ed Walker, Alberto Rodriguez, and Jianxiong Xiao. Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge. In 2017 IEEE international conference on robotics and automation (ICRA) , pages 1386–1383. IEEE, 2017

    work page 2017

  2. [2]

    Single image 3d object detection and pose estimation for grasping

    Menglong Zhu, Konstantinos G Derpanis, Yinfei Yang, Samarth Brahmbhatt, Mabel Zhang, Cody Phillips, Matthieu Lecce, and Kostas Daniilidis. Single image 3d object detection and pose estimation for grasping. In 2014 IEEE International Conference on Robotics and Automation (ICRA) , pages 3936–3943. IEEE, 2014

    work page 2014

  3. [3]

    Vision for robust robot manipulation

    Ester Martinez-Martin and Angel P Del Pobil. Vision for robust robot manipulation. Sensors, 19(7):1648, 2019

    work page 2019

  4. [4]

    Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching

    Andy Zeng, Shuran Song, Kuan-Ting Yu, Elliott Donlon, Francois R Hogan, Maria Bauza, Daolin Ma, Orion Taylor, Melody Liu, Eudald Romo, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. The International Journal of Robotics Research, 41(7):690–705, 2022

    work page 2022

  5. [5]

    Scalable deep reinforcement learning for vision-based robotic manipulation

    Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, pages 651–673. PMLR, 2018

    2018

  6. [6]

    Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, Peter David Fagan, Joey Hejna, Masha Itkina, Marion Lepert, Yecheng Jason Ma, Patrick Tree Miller, Jimmy Wu, Suneel Belkhale, Shivin Dass, Huy Ha, Arhan Jain, Abraham Lee, You...

    work page 2024

  7. [7]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source generalist robot policy. https://octo-models.github.io, 2023

    2023

  8. [8]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023

    arXiv

Show all 67 references
  1. [9]

    Unleashing large-scale video generative pre-training for visual robot manipulation

    Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint arXiv:2312.13139, 2023

    2023 arXiv

  2. [10]

    Large-scale actionless video pre-training via discrete diffusion for efficient policy learning

    Haoran He, Chenjia Bai, Ling Pan, Weinan Zhang, Bin Zhao, and Xuelong Li. Large-scale actionless video pre-training via discrete diffusion for efficient policy learning. arXiv preprint arXiv:2402.14407, 2024

    2024 arXiv

  3. [11]

    R3m: A universal visual representation for robot manipulation

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601, 2022. 10

    2022 arXiv

  4. [12]

    Gaia-1: A generative world model for autonomous driving

    Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080, 2023

    2023 arXiv

  5. [13]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision ...

    2022

  6. [14]

    Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks

    Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022

    2022

  7. [15]

    Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning

    Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956, 2019

    1910 arXiv

  8. [16]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR, 2020

    2020

  9. [17]

    Habituation: a dual-process theory

    Philip M Groves and Richard F Thompson. Habituation: a dual-process theory. Psychological review, 77 (5):419, 1970

    1970

  10. [18]

    Dual process theory: Perspectives and problems

    Jonathan St BT Evans. Dual process theory: Perspectives and problems. Dual process theory 2.0, pages 137–155, 2017

    2017

  11. [19]

    Dual process theories 14

    Bertram Gawronski and Laura A Creighton. Dual process theories 14. The Oxford handbook of social cognition, page 282, 2013

    2013

  12. [20]

    Who is rational?: Studies of individual differences in reasoning

    Keith E Stanovich. Who is rational?: Studies of individual differences in reasoning . Psychology Press, 1999

    1999

  13. [21]

    Open-sora: Democratizing efficient video production for all, March 2024

    Zangwei Zheng, Xiangyu Peng, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024. URL https://github.com/hpcaitech/Open-Sora

    2024

  14. [22]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35:2371...

    2022

  15. [23]

    Rlbench: The robot learning benchmark & learning environment

    Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters , 5(2):3019–3026, 2020

    2020

  16. [24]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  17. [25]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023

    2023 arXiv

  18. [26]

    Vision-language foundation models as effective robot imitators

    Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378, 2023

    2023 arXiv

  19. [27]

    Learning universal policies via text-guided video generation

    Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024

    2024

  20. [28]

    Bc-z: Zero-shot task generalization with robotic imitation learning

    Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991–1002. PMLR, 2022

    2022

  21. [29]

    Vima: General robot manipulation with multimodal prompts

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In F ortieth International Conference on Machine Learning, 2023

    2023

  22. [30]

    Making large language models better planners with reasoning-decision alignment

    Zhijian Huang, Tao Tang, Shaoxiang Chen, Sihao Lin, Zequn Jie, Lin Ma, Guangrun Wang, and Xiaodan Liang. Making large language models better planners with reasoning-decision alignment. arXiv preprint arXiv:2408.13890, 2024. 11

    2024 arXiv

  23. [31]

    Language conditioned imitation learning over unstructured data

    Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07648, 2020

    2005 arXiv

  24. [32]

    What matters in language conditioned robotic imitation learning over unstructured data

    Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters , 7(4):11205–11212, 2022

    2022

  25. [33]

    Zero-shot robotic manipulation with pretrained image-editing diffusion models

    Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310.10639, 2023

    2023 arXiv

  26. [34]

    3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations

    Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. In ICRA 2024 Workshop on 3D Visual Representations for Robot Manipulation, 2024

    2024

  27. [35]

    3d diffuser actor: Policy diffusion with 3d scene representations

    Tsung-Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint arXiv:2402.10885, 2024

    2024 arXiv

  28. [36]

    Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. arXiv preprint arXiv:2409.01652, 2024

    2024 arXiv

  29. [37]

    V oxposer: Composable 3d value maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.05973, 2023

    2023 arXiv

  30. [38]

    Graspnet-1billion: A large-scale benchmark for general object grasping

    Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. Graspnet-1billion: A large-scale benchmark for general object grasping. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11444–11453, 2020

    2020

  31. [39]

    Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection

    Yang Cao, Yihan Zeng, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. In NeurIPS, 2023

    2023

  32. [40]

    Collaborative novel object discovery and box-guided cross-modal alignment for open-vocabulary 3d object detection

    Yang Cao, Yihan Zeng, Hang Xu, and Dan Xu. Collaborative novel object discovery and box-guided cross-modal alignment for open-vocabulary 3d object detection. arXiv preprint arXiv:2406.00830, 2024

    2024 arXiv

  33. [41]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015–4026, 2023

    2023

  34. [42]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023

    2023 arXiv

  35. [43]

    Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection

    Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection. Advances in Neural Information Processing Systems , 35:9125–9138, 2022

    2022

  36. [44]

    Interactive language: Talking to robots in real time

    Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023

    2023

  37. [45]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    2023 arXiv

  38. [46]

    Robocat: A self-improving foundation agent for robotic manipulation

    Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, et al. Robocat: A self-improving foundation agent for robotic manipulation. arXiv preprint arXiv:2306.11706, 2023

    2023 arXiv

  39. [47]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022

    2022 arXiv

  40. [48]

    Robohive: A unified framework for robot learning

    Vikash Kumar, Rutav Shah, Gaoyue Zhou, Vincent Moens, Vittorio Caggiano, Abhishek Gupta, and Aravind Rajeswaran. Robohive: A unified framework for robot learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  41. [49]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 12

    2023 arXiv

  42. [50]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140):1–67, 2020. URL http:...

    2020

  43. [51]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023

    2023 arXiv

  44. [52]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. arXiv preprint arXiv:240...

    2024 arXiv

  45. [53]

    Adapool: Exponential adaptive pooling for information-retaining downsampling

    Alexandros Stergiou and Ronald Poppe. Adapool: Exponential adaptive pooling for information-retaining downsampling. 2021

    2021

  46. [54]

    Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation

    Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation. arXiv preprint arXiv:2305.10874, 2023

    2023 arXiv

  47. [55]

    Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016

    Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning, 2016

    2016

  48. [56]

    Bridgedata v2: A dataset for robot learning at scale

    Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, and Sergey Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (...

    2023

  49. [57]

    Latent plans for task agnostic offline reinforcement learning

    Erick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task agnostic offline reinforcement learning. 2022

    2022

  50. [58]

    Multi-stage cable routing through hierarchical imitation learning

    Jianlan Luo, Charles Xu, Xinyang Geng, Gilbert Feng, Kuan Fang, Liam Tan, Stefan Schaal, and Sergey Levine. Multi-stage cable routing through hierarchical imitation learning. arXiv pre-print, 2023. URL https://arxiv.org/abs/2307.08927

    2023 arXiv

  51. [59]

    Berkeley UR5 demonstration dataset

    Lawrence Yunliang Chen, Simeon Adebola, and Ken Goldberg. Berkeley UR5 demonstration dataset. https://sites.google.com/view/berkeley-ur5/home

  52. [60]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022

    2022

  53. [61]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  54. [62]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , p...

    2021

  55. [63]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  56. [64]

    Scaling vision transformers

    Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12104–12113, 2022

    2022

  57. [65]

    Perceiver-actor: A multi-task transformer for robotic manipulation

    Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pages 785–799. PMLR, 2023

    2023

  58. [66]

    Rvt: Robotic view transformer for 3d object manipulation

    Ankit Goyal, Jie Xu, Yijie Guo, Valts Blukis, Yu-Wei Chao, and Dieter Fox. Rvt: Robotic view transformer for 3d object manipulation. In Conference on Robot Learning, pages 694–710. PMLR, 2023

    2023

  59. [67]

    more diverse

    Theophile Gervet, Zhou Xian, Nikolaos Gkanatsios, and Katerina Fragkiadaki. Act3d: 3d feature field transformers for multi-task robotic manipulation. In 7th Annual Conference on Robot Learning , 2023. 13 A Appendix The outline of the Appendix is as follows: • Negative impacts ...

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.