Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

EC-Flow: Enabling Versatile Robotic Manipulation from Action-Unlabeled Videos via Embodiment-Centric Flow

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A robot can learn manipulation from ordinary videos by predicting the motion of its own body instead of tracking the object.

desk verdict Clever idea to predict flow on the robot body instead of the object, but the action-calculation math as printed does not close the loop, and the headline numbers overstate what the tables show. read the letter →

arxiv 2507.06224 v1 pith:JB6IJBNU submitted 2025-07-08 cs.RO cs.AI

Yixiang Chen , Peiyan Li , Yan Huang , Jiabing Yang , Kehan Chen , Liang Wang This is my paper · ORCID
classification cs.ROcs.AI
keywords embodiment-centricflowaction-unlabeledvideosrobotmanipulationdiffusionmodelURDFkinematicsobject-centricgoalimagepredictionreprojectionoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EC-Flow aims to let a robot learn a manipulation policy from demonstration videos that carry no action labels, by predicting the future pixel motion of the robot's own body—embodiment-centric flow—rather than the motion of the manipulated object. The authors' central claim is that tracking the body instead of the object removes three failure modes of object-centric flow methods: deformable objects that do not move rigidly, objects hidden by occlusion, and tasks like pressing a button or rotating a switch where the object barely moves. They report success improvements of 62%, 45%, and 80% over object-centric flow baselines in occluded, deformable, and non-displacement tasks respectively, and also beat behavior cloning that trains on action-labeled data when demonstrations are scarce. The predicted flow is converted into executable arm poses using only the robot's URDF model, live depth, and camera calibration, so no action supervision is needed at any stage.

What carries the argument

The central object is the embodiment-centric flow itself: a set of point trajectories over the robot's own links, predicted over a fixed horizon by a diffusion model conditioned on the initial image, the language instruction, and the starting point positions. The flow predictor is trained on point tracks extracted from action-unlabeled videos, with the robot's pixels masked by an off-the-shelf segmentation pipeline. The second piece of machinery is the kinematic-aware action calculation: predicted 2D points are filtered, assigned to joints via bounding boxes derived from the robot's URDF, lifted to 3D with live depth, and then used to optimize the end-effector pose by minimizing reprojection error against forward-kinematic transforms from a candidate pose, with the URDF's joint limits as constraints. The auxiliary goal-image branch, co-trained on synchronized noise timesteps, supplies the alignment between body motion, object state, and language that the flow alone lacks.

What would settle it

Run the trained policy on a task where a joint visible in the first frame is deliberately occluded from the second frame onward: the paper's own supplementary analysis predicts failure, so a task completion there would falsify the stated limitation, while swapping the depth camera for a higher-accuracy sensor and observing whether fold-clothes success rises would isolate whether depth error, not flow error, is the binding constraint.

Watch

Extended reading notes

Core claim

The paper's discovery is that shifting the flow prediction target from the object to the embodiment makes action-unlabeled video usable across a much wider class of manipulation tasks. A diffusion model conditioned on the initial frame, the language instruction, and the starting coordinates of sampled body points generates an eight-frame trajectory of 2D flow for points spread across the robot's visible links. An auxiliary goal-image prediction branch, trained with matching diffusion timesteps, forces the predicted flow to be consistent with a final scene that satisfies the instruction, aligning body motion with object interaction even though the object itself is never tracked. To execute, the method filters the predicted points, assigns them to joints through the URDF's geometry, lifts them to 3D with the depth camera, and optimizes the end-effector pose by minimizing the difference between the predicted flow and the reprojection of URDF-based joint transformations, subject to joint limits. The authors validate this in simulation and on a real arm, where it handles deformable objects, occlusions, and non-displacement tasks that object-centric flow cannot.

Load-bearing premise

The load-bearing premise is that the predicted eight-frame flow, together with live depth and camera calibration, is accurate enough for the reprojection optimization to recover a correct end-effector pose; the paper itself notes that depth noise degrades the fold-clothes task and that a joint visible at the start but occluded later can cause execution failure.

Editorial extensions

If this is right

  • Policies can be trained from RGB videos and language instructions alone, so demonstration data becomes as easy to collect as filming a human or a teleoperated robot performing a task.
  • The approach extends video-only manipulation learning to deformable objects, occluded scenes, and actions that do not displace objects, the three settings where object-centric flow breaks down.
  • Deploying a trained flow predictor on a new robot requires only that robot's URDF plus calibrated depth, not a new action-labeled dataset.
  • In the low-data regime of five demonstrations per task, the method outperforms behavior cloning trained on the same number of action-labeled demonstrations in real-world tests.
  • The eight-frame flow prediction runs once at the start of a trajectory, making the current system best suited to coarse open-loop planning rather than dense closed-loop control.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If depth error is the binding constraint, as the fold-clothes results suggest, upgrading the depth sensor or fusing multiple views could raise real-world success without retraining the flow predictor.
  • Because the goal-image branch couples predicted flow to a language-grounded final frame, it may act as a transfer bridge: motion priors from human videos could pre-train the flow predictor, with the URDF-based calculation adapting those priors to a specific robot at deployment.
  • The success on non-object-displacement tasks suggests that body-centric flow is a more general action interface than object state; a testable extension is applying the same framework to mobile manipulators or humanoids whose bodies remain visible during whole-body motions.
  • Replacing the diffusion sampler with a faster generative model would bring the inference time from seconds toward real time, which could turn EC-Flow from a per-trajectory planner into a closed-loop policy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes EC-Flow, a framework for learning robotic manipulation policies from action-unlabeled RGB videos by predicting 'embodiment-centric flow' (pixel trajectories of points on the robot body) rather than object-centric flow. A diffusion model conditioned on an initial image, language instruction, and initial point coordinates predicts an 8-frame flow trajectory, with an auxiliary goal-image prediction branch that is co-trained to align motion with language and object interaction. Actions are then computed from the predicted flow via a URDF-based kinematic optimization: sampled points are assigned to joints, lifted to 3D with depth, and an end-effector pose is optimized so that reprojected joint points match the predicted next-frame 2D locations. The authors evaluate on nine Meta-World tasks and seven real-world tasks with a Franka Research 3 arm, reporting large improvements over object-centric flow baselines, particularly in occluded, deformable, and non-object-displacement tasks.

Significance. If the method works as claimed, the idea of predicting flow on the embodiment rather than the object is a genuinely useful shift: it avoids rigidity assumptions, handles occlusion of both object and arm, and covers tasks where the object does not move. The paper is also refreshing in that it avoids circularity: the flow targets come from CoTracker tracks of the same videos and the success metric is task completion, not the predicted flow itself. The framework is clearly described at a high level, and the real-world evaluation across seven tasks is a strength. However, the central flow-to-action optimization is not correctly specified in the manuscript, and the reported headline improvements are absolute success-rate differences against zero baselines without statistical quantification. These issues currently prevent the reader from reproducing or fully trusting the central claim.

major comments (3)
  1. [Section 3.3.2 / Algorithm 1, step 7] The reprojection objective is not a valid rigid-body reprojection as written. Let x_t = T_ee(t) · ee_j T(q_t) · p_j be the world position at time t of a point with constant link-frame coordinate p_j. The world position at time t+1 is x_{t+1} = T_ee(t+1) · ee_j T(q_{t+1}) · (T_ee(t) · ee_j T(q_t))^{-1} · x_t, and the correct residual is ||π(x_{t+1}) − P^{(t+1)2D}_{ji}||². The printed objective instead evaluates π(T_ee · ee_j T · P^{(t)3D}_{ji}) with P^{(t)3D}_{ji} obtained by lifting the depth at time t, i.e., it applies a candidate transform to an already-world-frame point, and the previous end-effector pose T_ee(t) never enters the loss. Even with perfect flow, perfect depth, and perfect calibration, this objective does not correspond to the motion of a rigid body attached to link j and cannot be expected to recover the correct end-effector pose. Please correct the derivation, state whether P^{(t)3D}_{ji} is in camera, world, or link frame, and specify which quantities are candidate poses versus known previous poses.
  2. [Supplementary B.1 vs. Section 3.3.2] The manuscript describes the deployed action calculation as full-joint optimization (0.37 s in Table 4), but Algorithm 1 and the surrounding text optimize only the end-effector pose T_ee, with joint transforms computed 'via IK'. No objective, variable set, constraints, or initialization for the full-joint optimization is given. This matters for two reasons: the IK step is not a unique mapping for a redundant 7-DoF arm, and Table 4 reports that full-joint optimization outperforms EEF-only optimization by 5.3% in simulation and 7.1% in the real world. The reader cannot reproduce the reported method without knowing the full-joint objective. Please align Algorithm 1 with the deployed method or provide the full-joint formulation explicitly.
  3. [Abstract and Section 4.2] The reported 'improvements' of 62%, 45%, and 80% are absolute success-rate differences, and in several real-world tasks the object-centric baseline is 0/10. For example, Table 3 shows Track2Act at 0/10 on fold-towel, press-mouse, and rotate-switch, while EC-Flow achieves 6/10, 9/10, and 7/10; the '80% improvement' is the absolute success rate of EC-Flow, not a relative improvement over a nonzero baseline. With only 25 simulation trials per task and 10 real-world trials per task, and no confidence intervals or error bars, the current wording overstates the comparison. Please report per-task counts, define how 'improvement' is computed, and provide uncertainty quantification where feasible.
minor comments (5)
  1. [Section 4.2] The sentence 'The results of the real-world evaluation are presented in Table 3.' appears twice in immediate succession; please remove the duplicate.
  2. [Supplementary Figure 7] The caption 'The real-world setup of VERM' appears to use the wrong acronym; it should refer to EC-Flow.
  3. [Supplementary B.1, Table 4] The success-rate column '66.7% / 70.0% 72.0% / 77.1%' is visually ambiguous; please separate the EEF-only and full-joint values clearly, e.g., with two columns or a row label.
  4. [Section 3.2.1] The dataset is first defined as D = {ζ_i}_{i=1}^n with ζ_i = ({o_t}, l_i), and later as D = {(p_i, o_i0, l_i)}_{i=1}^n; the notation should be made consistent.
  5. [Section 1 and Section 3.3] The text emphasizes that EC-Flow uses 'only RGB videos' for training, but the action calculation requires live depth maps and camera calibration; please clarify in the introduction that depth is needed at execution time.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: EC-Flow's flow targets come from external CoTracker tracks, and actions are computed by URDF-based optimization from predicted flow; no fitted parameter is relabeled as a prediction.

full rationale

I walked the derivation chain: RGB videos to CoTracker point tracks (Sec. 3.2.1), diffusion flow prediction conditioned on the initial frame, language, and start points (Eq. 3), auxiliary goal-image prediction (Limage), and URDF-based reprojection optimization (Algorithm 1) to recover the end-effector pose. The flow training target is the tracked point trajectory, not the task-success signal or the output action; the action is recovered from predicted flow plus live depth and camera/URDF kinematics by optimization, with no fitted parameter that is later reported as a prediction. The goal-image branch is an auxiliary loss and shared-conditioning regularizer, not a recycled target of the flow branch or of the success metric. Reliance on CoTracker, GroundingDINO, SAM2, CLIP, and the URDF is reliance on external models and priors, not on the paper's own conclusions or on self-citation. The paper's self-citations ([17], [18]) appear only in related-work enumerations and are not load-bearing. Supplementary limitations (D435i depth inaccuracy in Sec. 4.2; visible-joint consistency in Supp. B.3) describe robustness boundaries rather than circular dependencies. The noted geometric inconsistency in Algorithm 1's printed reprojection objective is a reproducibility and correctness concern—if the formula is taken literally it is not a valid rigid-body reprojection—but it does not make the derivation equivalent to its inputs by construction. Accordingly, no circular step was found.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the availability of accurate CoTracker tracks, a correct URDF and calibrated depth camera, and the ability of a diffusion model trained on 5 demonstrations per task to generalize to new object positions. These are not proven by the paper; they are assumptions about the experimental setup.

free parameters (6)
  • Number of sampled embodiment points Np = 400
    Hand-chosen; ablation shows success rate ranges from 0.591 (Np=100) to 0.733 (Np=800), so the central result depends on this choice.
  • Prediction horizon T = 8
    Ablation: success rate 0.244 at T=2, 0.720 at T=8, 0.707 at T=16. The paper selects T=8 as a balance between performance and cost.
  • Goal-image loss weight lambda = 0.4
    Hand-chosen balance between flow and image losses; affects the learned flow and thus downstream actions.
  • Minimum motion/displacement threshold for point filtering = unspecified
    Filtering requires a minimum displacement threshold, but its value is never reported, so the filtering step is not fully specified.
  • Joint-specific weights in reprojection error = manual
    Supplementary B.1 states these weights 'can be manually tuned based on joint visibility or task requirements', introducing a free knob that affects the action calculation.
  • DDIM sampling steps = 250
    Inference-time hyperparameter; no ablation is shown, but it affects flow quality and inference latency.
assumptions (5)
  • domain assumption CoTracker provides ground-truth-quality point tracks for training the flow prediction model
    Section 3.2.1 uses CoTracker point tracks as supervision; if CoTracker is inaccurate on a robot arm, the learned flow will be biased.
  • domain assumption The robot's URDF correctly reflects its kinematics and the camera is calibrated
    Section 3.3 projects joint positions and runs IK based on URDF and camera intrinsics/extrinsics; errors here directly corrupt action computation.
  • domain assumption The diffusion model generalizes from 5 demonstrations per task to novel object positions
    Section 4.1 trains on 45 videos and evaluates on random object initializations; there is no test on unseen tasks.
  • domain assumption The language instruction and initial frame contain sufficient information to predict the full embodiment flow
    Section 3.2.2 conditions on CLIP text and ResNet visual features; ambiguous instructions would make flow prediction ambiguous.
  • domain assumption Visible joints remain consistent between the initial frame and execution
    Supplementary B.3 states that if a joint visible at the start becomes occluded later, the system may fail; this is an acknowledged limitation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EC-Flow: Enabling Versatile Robotic Manipulation from Action-Unlabeled Videos via Embodiment-Centric Flow." pith.science (2026). https://pith.science/paper/JB6IJBNU

@misc{pith2026250706224,
  author       = {Pith},
  title        = {Pith review of: EC-Flow: Enabling Versatile Robotic Manipulation from Action-Unlabeled Videos via Embodiment-Centric Flow},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JB6IJBNU}},
  note         = {Machine review of arXiv:2507.06224}
}
read the original abstract

Current language-guided robotic manipulation systems often require low-level action-labeled datasets for imitation learning. While object-centric flow prediction methods mitigate this issue, they remain limited to scenarios involving rigid objects with clear displacement and minimal occlusion. In this work, we present Embodiment-Centric Flow (EC-Flow), a framework that directly learns manipulation from action-unlabeled videos by predicting embodiment-centric flow. Our key insight is that incorporating the embodiment's inherent kinematics significantly enhances generalization to versatile manipulation scenarios, including deformable object handling, occlusions, and non-object-displacement tasks. To connect the EC-Flow with language instructions and object interactions, we further introduce a goal-alignment module by jointly optimizing movement consistency and goal-image prediction. Moreover, translating EC-Flow to executable robot actions only requires a standard robot URDF (Unified Robot Description Format) file to specify kinematic constraints across joints, which makes it easy to use in practice. We validate EC-Flow on both simulation (Meta-World) and real-world tasks, demonstrating its state-of-the-art performance in occluded object handling (62% improvement), deformable object manipulation (45% improvement), and non-object-displacement tasks (80% improvement) than prior state-of-the-art object-centric flow methods. For more information, see our project website at https://ec-flow1.github.io .

Figures

Figures reproduced from arXiv: 2507.06224 by the authors.

Figure 1
Figure 1. Limitations of object-centric flow: (1) Deformable ob [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Embodiment-centric flow prediction network architecture. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Process of allocating sampled points to specific joints. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Occlusion and hallucination problems of object-centric [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Left: Influence of choice of sampled points Np. Right: Influence of choice of predict horizon T. 4.2. Real-world Evaluation Real-world Setup Our real-world experiment setup con￾sists of a statically mounted Franka Research 3 robotic arm and a static third-person view I…
Figure 6
Figure 6. Figure 6: Visualization of the EC-Flow method for learning robotic manipulation from action-unlabeled video data. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The real-world setup of VERM. D.2. Real-world Baselines (1) BC: Standard behavior cloning baseline with dual￾stream encoder architecture. Processes RGB obser￾vations through ResNet-18 [12] and language instruc￾tions via CLIP [26], trained end-to-end on action-labeled d…
Figure 8
Figure 8. Figure 8: Visualization of the EC-Flow in Meta-World. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Visualization of the EC-Flow in the real-world. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UAOR: Uncertainty-aware Observation Reinjection for Vision-Language-Action Models

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Uncertainty-aware observation reinjection into FFN layers improves VLA manipulation success rates across LIBERO, SIMPLER, CALVIN, and real-robot tasks with no training.

Reference graph

Works this paper leans on

42 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    Track2act: Predicting point tracks from internet videos enables generalizable robot manipula- tion

    Homanga Bharadhwaj, Roozbeh Mottaghi, Abhinav Gupta, and Shubham Tulsiani. Track2act: Predicting point tracks from internet videos enables generalizable robot manipula- tion. In European Conference on Computer Vision, 2024. 2, 6, 7, 8, 13

    work page 2024

  2. [2]

    π0: A vision-language- action flow model for general robot control

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0: A vision-language- action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. 2

    arXiv

  3. [3]

    Zero-shot robotic manipulation with pre-trained image-editing diffusion models

    Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Rich Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pre-trained image-editing diffusion models. InThe Twelfth International Conference on Learning Representations, 2024. 1

    work page 2024

  4. [4]

    Rt-1: Robotics transformer for real-world control at scale

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakr- ishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. 2

    arXiv 2022

  5. [5]

    Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation

    Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024. 1, 2

    arXiv

  6. [6]

    Diffusion policy: Visuomotor policy learning via action dif- fusion

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action dif- fusion. The International Journal of Robotics Research, page 02783649241273668, 2023. 6

    work page 2023

  7. [7]

    Learning universal policies via text-guided video genera- tion

    Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video genera- tion. Advances in Neural Information Processing Systems , 36, 2024. 6

    work page 2024

  8. [8]

    Tenenbaum, Leslie Pack Kaelbling, Andy Zeng, and Jonathan Tompson

    Yilun Du, Sherry Yang, Pete Florence, Fei Xia, Ayzaan Wahid, brian ichter, Pierre Sermanet, Tianhe Yu, Pieter Abbeel, Joshua B. Tenenbaum, Leslie Pack Kaelbling, Andy Zeng, and Jonathan Tompson. Video language planning. In The Twelfth International Conference on Learning Represen- tations, 2024. 1, 2

    work page 2024

Show all 42 references
  1. [9]

    Graspnet-1billion: A large-scale benchmark for general ob- ject grasping

    Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu. Graspnet-1billion: A large-scale benchmark for general ob- ject grasping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11444– 11453, 2020. 12

    2020

  2. [10]

    FLIP: Flow-centric generative planning as general-purpose manipulation world model

    Chongkai Gao, Haozhuo Zhang, Zhixuan Xu, Cai Zhehao, and Lin Shao. FLIP: Flow-centric generative planning as general-purpose manipulation world model. In The Thir- teenth International Conference on Learning Representa- tions, 2025. 2

    2025

  3. [11]

    The” something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In ...

    2017

  4. [12]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. 3, 6, 13

    2016

  5. [13]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 3

    2020

  6. [14]

    Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos

    Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Co- tracker3: Simpler and better point tracking by pseudo- labelling real videos. arXiv preprint arXiv:2410.11831 ,

    arXiv

  7. [15]

    Openvla: An open- source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An op...

    2025

  8. [16]

    Tenenbaum

    Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences. In The Twelfth In- ternational Conference on Learning Representations , 2024. 2, 6

    2024

  9. [17]

    Bridgevla: Input-output alignment for efficient 3d manipu- lation learning with vision-language models

    Peiyan Li, Yixiang Chen, Hongtao Wu, Xiao Ma, Xiangnan Wu, Yan Huang, Liang Wang, Tao Kong, and Tieniu Tan. Bridgevla: Input-output alignment for efficient 3d manipu- lation learning with vision-language models. arXiv preprint arXiv:2506.07961, 2025. 2

    2025

  10. [18]

    Gr-mg: Leveraging partially- annotated data via multi-modal goal-conditioned policy

    Peiyan Li, Hongtao Wu, Yan Huang, Chilam Cheang, Liang Wang, and Tao Kong. Gr-mg: Leveraging partially- annotated data via multi-modal goal-conditioned policy. IEEE Robotics and Automation Letters, 2025. 1, 2

    2025

  11. [19]

    Dreamitate: Real-world visuomotor policy learn- ing via video generation

    Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sud- hakar, Achal Dave, Pavel Tokmakov, Shuran Song, and Carl V ondrick. Dreamitate: Real-world visuomotor policy learn- ing via video generation. In8th Annual Conference on Robot Learning, 2024. 1, 2

    2024

  12. [20]

    RDT-1b: a diffusion foundation model for bimanual manipu- lation

    Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, and Jun Zhu. RDT-1b: a diffusion foundation model for bimanual manipu- lation. In The Thirteenth International Conference on Learn- ing Representations, 2025. 2

    2025

  13. [21]

    Struc- tured world models from human videos

    Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Struc- tured world models from human videos. In Robotics: Sci- ence and Systems, 2023. 2 9

    2023

  14. [22]

    R3m: A universal visual rep- resentation for robot manipulation

    Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual rep- resentation for robot manipulation. In Proceedings of The 6th Conference on Robot Learning, pages 892–909. PMLR,

  15. [23]

    Generate subgoal images before act: Unlocking the chain-of-thought reasoning in diffusion model for robot manipulation with multimodal prompts

    Fei Ni, Jianye Hao, Shiguang Wu, Longxin Kou, Jiashun Liu, Yan Zheng, Bin Wang, and Yuzheng Zhuang. Generate subgoal images before act: Unlocking the chain-of-thought reasoning in diffusion model for robot manipulation with multimodal prompts. In Proceedings of the IEEE/CVF Co...

    2024

  16. [24]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Charles Xu, Jianlan Luo, Tobias Kreiman, You Liang Tan, Pannag Sanketi, Quan Vuong, Ted Xiao, Dorsa Sadigh, Chelsea Finn, and Sergey Levine. Octo: An open-source general...

    2024

  17. [25]

    Spatialvla: Exploring spatial represen- tations for visual-language-action model

    Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, et al. Spatialvla: Exploring spatial represen- tations for visual-language-action model. arXiv preprint arXiv:2501.15830, 2025. 2

    2025 arXiv

  18. [26]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In International Conference on Machine Learning...

    2021

  19. [27]

    SAM 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollar, and Christoph Feich...

    2025

  20. [28]

    Motion tracks: A unified representation for human-robot transfer in few-shot imitation learning

    Juntao Ren, Priya Sundaresan, Dorsa Sadigh, Sanjiban Choudhury, and Jeannette Bohg. Motion tracks: A unified representation for human-robot transfer in few-shot imitation learning. arXiv preprint arXiv:2501.06994, 2025. 2, 3

    2025

  21. [29]

    Grounding dino 1.5: Advance the” edge” of open-set object detection

    Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wen- long Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, et al. Grounding dino 1.5: Advance the” edge” of open-set object detection. arXiv preprint arXiv:2405.10300, 2024. 3

    2024 arXiv

  22. [30]

    Grounded sam: Assembling open-world models for diverse visual tasks

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 ,

    arXiv

  23. [31]

    Denois- ing diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois- ing diffusion implicit models. In International Conference on Learning Representations, 2021. 4

    2021

  24. [32]

    Zhao, Ken Goldberg, Ryan Hoque, Lawrence Yunliang Chen, Simeon Adebola, Gaurav S

    Quan Vuong, Sergey Levine, Homer Rich Walke, Karl Pertsch, Anikait Singh, Ria Doshi, Charles Xu, Jianlan Luo, Liam Tan, Dhruv Shah, Chelsea Finn, Max Du, Moo Jin Kim, Alexander Khazatsky, Jonathan Heewon Yang, Tony Z. Zhao, Ken Goldberg, Ryan Hoque, Lawrence Yunliang Chen, Sim...

    2023

  25. [33]

    Any-point trajectory modeling for policy learning

    Chuan Wen, Xingyu Lin, John So, Kai Chen, Qi Dou, Yang Gao, and Pieter Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023. 2

    2023 arXiv

  26. [34]

    Unleashing large-scale video generative pre-training for visual robot manipulation

    Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In International Conference on Learning Representations, 2024. 2

    2024

  27. [35]

    Flow as the 10 cross-domain manipulation interface

    Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, and Shuran Song. Flow as the 10 cross-domain manipulation interface. In 8th Annual Confer- ence on Robot Learning, 2024. 2

    2024

  28. [36]

    Learning interactive real-world simulators

    Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuur- mans, and Pieter Abbeel. Learning interactive real-world simulators. In The Twelfth International Conference on Learning Representations, 2024. 2

    2024

  29. [37]

    Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning

    Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning ,

  30. [38]

    Dexgrasp- net 2.0: Learning generative dexterous grasping in large- scale synthetic cluttered scenes

    XinQiang Yu, Jialiang Zhang, Haoran Liu, Danshi Li, Hao- ran Geng, He Wang, Yufei Ding, and Jiayi Chen. Dexgrasp- net 2.0: Learning generative dexterous grasping in large- scale synthetic cluttered scenes. In 2nd Workshop on Dex- terous Manipulation: Design, Perception and Con...

  31. [39]

    General flow as foundation affordance for scalable robot learning

    Chengbo Yuan, Chuan Wen, Tong Zhang, and Yang Gao. General flow as foundation affordance for scalable robot learning. In 8th Annual Conference on Robot Learning ,

  32. [40]

    Vlmpc: Vision-language model pre- dictive control for robotic manipulation

    Wentao Zhao, Jiaming Chen, Ziyu Meng, Donghui Mao, Ran Song, and Wei Zhang. Vlmpc: Vision-language model pre- dictive control for robotic manipulation. In Robotics: Sci- ence and Systems, 2024. 1, 2

    2024

  33. [41]

    VIOLA: Object-centric imitation learning for vision-based robot manipulation

    Yifeng Zhu, Abhishek Joshi, Peter Stone, and Yuke Zhu. VIOLA: Object-centric imitation learning for vision-based robot manipulation. In 6th Annual Conference on Robot Learning, 2022. 7

    2022

  34. [42]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023. 2 11 ...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.