Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Generative Visual Foresight Meets Task-Agnostic Pose Estimation in Robotic Table-Top Manipulation

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A robot can be trained from unlabeled video and random exploration alone to follow language instructions for tabletop tasks, reaching 83% average success on the LIBERO benchmark.

desk verdict A genuinely label-light pipeline with a plausibly load-bearing pose estimator; the numbers are strong but the weak link is the one that goes unmeasured. read the letter →

arxiv 2509.00361 v1 pith:2KEIHSYF submitted 2025-08-30 cs.RO

Chuye Zhang , Xiaoxiong Zhang , Wei Pan , Linfang Zheng , Wei Zhang This is my paper
classification cs.RO
keywords generativevisualforesighttask-agnosticposeestimationaction-label-freemanipulationRGB-Dvideopredictionrectifiedflowclosed-looprobotcontrolLIBERObenchmarkrandomexploration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a robot can learn a wide range of tabletop manipulation tasks without any action labels, expert demonstrations, or task-specific training, by splitting the job into two parts: imagining the future and reading its own body. Given one side-view RGB image and a text instruction, a generative video model predicts the next RGB-D frames of the scene, and a pose-estimation network trained only on random arm movements converts those imagined frames into 6-DoF end-effector commands. On the LIBERO benchmark the system reaches 83.00% average success, 11.56 points above the best action-label-based baseline; in the real world it reaches 86% success when the video model is pretrained on human manipulation videos. If true, this means the expensive part of robot learning—collecting and labeling action data for every new task—can be replaced by cheap random exploration plus unlabeled video.

What carries the argument

The load-bearing mechanism is the decoupling of visual planning from action execution. The foresight module is a rectified-flow velocity model built on a 3D U-Net, conditioned by a CLIP text encoder; it interpolates between noise and a clean future RGB-D sequence and predicts the displacement, giving strong video with only three sampling steps. Because depth is part of the generated output and is supervised by Video Depth Anything, the model can be pretrained on RGB-only human videos. The pose module is a cross-attention ViT: the depth encoder's CLS token queries the RGB encoder's patch tokens, and a three-layer MLP regresses the 8-dimensional end-effector pose. Trained on uniform random exp

What would settle it

Run GVF-TAPE on a LIBERO-Goal drawer task with the pose estimator frozen, and record per-frame pose error on frames where the gripper is occluded; if the error there is much larger than on random-exploration frames and the success rate stays near the reported 66.8%, the central claim that random-exploration training generalizes to task poses is falsified.

Watch

Extended reading notes

Core claim

GVF-TAPE learns the mapping f: (x0, c) → T from a side-view RGB image x0 and language instruction c to a pose trajectory, each T_i = (p_i, q_i, g_i) holding 3D position, orientation quaternion, and gripper state. A rectified-flow video model generates six future RGB-D frames; depth is inferred implicitly and supervised by an off-the-shelf monocular depth estimator, allowing pretraining on RGB-only human videos. A cross-attention ViT pose estimator, trained only on uniformly sampled random exploration data, regresses each frame's 8-dimensional pose independently. Executing the predicted trajectory, re-observing, and re-planning closes the loop. The paper reports 83.00% LIBERO average success

Load-bearing premise

The pose estimator must generalize from the random-exploration poses it saw in training to the poses the robot actually takes during task execution—including occluded grippers, held objects, and dynamic contacts—because any wrong per-frame pose produces a wrong control command.

Editorial extensions

If this is right

  • Pose supervision comes from random exploration, not expert demos, so data collection is fully automatic and scales across robots and environments.
  • Depth is generated, not sensed: the foresight model outputs RGB-D and uses a monocular depth estimator for supervision, so training can start from RGB-only videos, including human hand videos, and transfer to the robot.
  • Data efficiency: with only 20% of LIBERO demonstrations plus pretraining on LIBERO-90, GVF-TAPE matches or beats ATM, which uses 100% of the action-labeled data; pretraining adds 9.2% success.
  • Human-video pretraining raises real-world success from 56% to 86%, showing cross-embodiment transfer from unlabeled human manipulation videos.
  • Closed-loop replanning provides failure recovery: when the first attempt misses (e.g., grabbing a tissue), the next video-generation cycle sees the failed state and re-plans toward the goal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own failure analysis (Table 11: 5 of 11 LIBERO-Spatial failures are pose-estimation errors, only 3 are hallucination) suggests that the bottleneck is not video prediction but the pose estimator's generalization; adding a wrist camera or multi-view input would most directly attack the reported failures.
  • Because pose training poses are sampled uniformly in workspace bounds, performance should degrade on trajectories that spend significant time in occluded or out-of-view regions (as seen in LIBERO-Goal and LIVING-ROOM-SCENE-5); a curriculum that samples more poses near contact or occlusion is a testable fix that does not require changing the architecture.
  • If depth is inferred by Video Depth Anything, then systematic depth errors in cluttered or transparent scenes will propagate into pose and therefore control; measuring pose error on generated vs. real frames would quantify how much the closed loop masks this.
  • The ability to pretrain the foresight module on human videos opens a route to web-scale RGB video pretraining, since only a text instruction and RGB observation are needed—no robot embodiment or action labels.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper introduces GVF-TAPE, a closed-loop manipulation framework that decouples visual foresight from action execution. A rectified-flow 3D U-Net conditioned on a single side-view RGB image and a CLIP text embedding predicts six future RGB-D frames; a transformer pose estimator trained on uniformly sampled random exploration poses regresses 6-DoF end-effector pose plus gripper state from each predicted RGB-D frame, and a low-level controller executes these poses via inverse kinematics. The pipeline is evaluated in LIBERO (Spatial/Object/Goal and living-room scenes) and on a real ARX-5 with five plus two deformable-object tasks. Headline results: 83.00% LIBERO overall success with no action labels (Table 1), 79.4% on the living-room comparison (Table 2), and 56%/86% real-world success without/with human-video pretraining (Table 3). Ablations motivate rectified flow, monocular depth, and cross-embodiment pretraining.

Significance. If the results hold, GVF-TAPE is a meaningful step toward scalable manipulation learning because it removes expert action labels for the policy and replaces them with automatically collected random-exploration pose data plus video demonstrations. The paper's strongest assets are the breadth of experiments, the explicit failure analysis, and the reproducibility details in the appendix (model architecture, random exploration algorithm, inference times). The central claim, however, rests on an unquantified generalization assumption: the pose estimator must succeed on distribution-shifted generated frames containing occlusions, held objects, and contacts. The failure analysis in Table 11 and the real-world per-task variability in Table 3 suggest this assumption is the fragile link. The manuscript would be substantially strengthened by reporting pose-estimation error on task-distribution frames and by presenting real-world results with confidence intervals. With those additions the contribution would be credible.

major comments (4)
  1. [Section 3.4, Tables 11 and 3] The pose estimator is the only link between predicted frames and executable commands. It is trained on random exploration poses sampled uniformly within a fixed workspace (Algorithm 1), where the gripper is usually empty and scenes are uncluttered. During deployment, the input distribution shifts to generated frames containing held objects, occluded grippers, dynamic contacts, and arm configurations outside the training support. Since each frame is processed independently, a per-frame pose error directly produces an incorrect controller target and is not averaged away. The paper never reports pose accuracy (position, orientation, gripper) on task-distribution frames, despite Table 11 listing pose-estimation error as the leading failure cause (5 of 11 LIBERO-Spatial failures) and Table 3 showing the largest real-world failures precisely in tasks with deformable/occluded interactions (grab
  2. [Sections 3.3-3.4 and 7.7-7.8, Eq. (4), Table 4] Depth is not measured but synthesized. The real-world pose-estimation training set uses Video-Depth-Anything labels (Section 7.7), and the simulation pipeline also passes rendered data through a monocular depth estimator (Section 7.8). The video model outputs an RGB-D prediction whose depth channel is a generated quantity. The paper does not compare this estimated depth against sensor depth (the D435i is available), nor does it quantify how depth error propagates to pose error. Table 4 isolates the benefit of w/ depth versus w/o depth, but not the cost of using estimated depth instead of true depth. Since the pose estimator's cross-attention uses the depth CLS token as the query (Eq. 4), depth quality is load-bearing. Please report depth accuracy and/or run an ablation with ground-truth or sensor depth.
  3. [Table 1 and Fig. 4] Table 1 compares GVF-TAPE to baselines that use 20% action-labeled data (10 action trajectories plus 50 video demos), but it does not state how many video demonstrations or what pretraining GVF-TAPE uses in that table. Section 4.1 mentions 50 video demos for baselines, Section 4.2 says 20 demos for the living-room comparison, and Fig. 4 says 20% data corresponds to 10 demos. It is therefore unclear whether the Table 1 headline (83.00%, outperforms the next-best by 11.56%) is obtained with 10, 20, or 50 demonstrations per task, and whether LIBERO-90 pretraining is included. Please specify the exact data budget for each reported GVF-TAPE result, including a column in Table 1, so the comparison is auditable.
  4. [Section 4.3, Table 3] Real-world success rates are based on 10 trials per task with no confidence intervals, and the seven-condition protocol (5 close, 2 far, 2 distractors, 1 lighting) conflates condition and task. With n=10, differences such as 30% vs 70% cannot be distinguished from chance, so the claim that human-video pretraining consistently improves performance and the average 56% to 86% gain is not statistically supported. Please report binomial confidence intervals or per-condition results, and consider increasing trials or pooling across tasks appropriately.
minor comments (5)
  1. [Section 7.10.3] Occultation should be Occlusion. Also, the method name AVDC is written as A VDC in Table 2 and in several places; unify the spelling.
  2. [Algorithm 1] The while-condition logic (resample while current is close to desired) is unconventional and should be commented in the pseudocode. Specify the units and value of the threshold DT and how the initial samples are drawn within workspace W.
  3. [Eq. (3)] The pose estimator uses a SmoothL1 loss on an 8-dimensional output that includes a quaternion. Unit quaternions require special handling; please state how the quaternion is normalized or whether the loss is applied to raw outputs.
  4. [Section 7.8] A gripper aperture threshold is mentioned but its value and sensitivity are not reported. Since it directly affects grasp success and is a free parameter, provide the chosen value and, ideally, a short sensitivity study.
  5. [Table 5] The VLA baseline results in Table 5 are reported without standard deviations. If these numbers are taken from prior papers, cite the source; if re-evaluated, report variability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: all reported results are held-out empirical evaluations, and neither the pose estimator nor the video model is fitted to task outcomes.

full rationale

GVF-TAPE's central claims are measured success rates on LIBERO and real-world rollouts (Tables 1, 2, 3, 10, 11), not quantities derived from training targets. The video model minimizes Eq. (2) against ground-truth future RGB-D frames from demonstrations, while the pose estimator minimizes Eq. (3) against poses sampled by random exploration (Sec. 3.4, Algorithm 1); at test time the same pose model is applied to generated frames. No parameter is fitted to the reported success rates, and no equation defines the benchmark outcome in terms of the model's own inputs. The failure analysis (Table 11) and Limitations section candidly attribute failures to pose-estimation error, hallucination, and occlusion, which are generalization and robustness concerns rather than circular reductions. Author self-citations (e.g., Refs. [38,43,44]) appear only in pose-estimation background and are not load-bearing for the framework's derivation. The depth signal is synthesized by Video Depth Anything at both training and inference, a consistency choice that raises metric-depth validity questions but does not make the success prediction equivalent to an input. Thus no circular step is present; the paper is self-contained with respect to its empirical evaluation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim depends on the pose estimator generalizing beyond its random-exploration training distribution and on the reliability of the monocular depth estimator that supplies depth for both training and deployment.

free parameters (3)
  • Gripper aperture threshold = not reported
    Used to zero the gripper command when predicted aperture falls below threshold (Sec 7.8); tuned by hand and directly affects grasp success.
  • Number of future frames per prediction = 6
    The video model generates 6 frames per cycle (Table 9), setting the planning horizon; a hand-chosen design parameter.
  • Maximum replanning cycles = 15 (real-world)
    Evaluation protocol limits each rollout to 15 video generation cycles (Sec 7.7); this cutoff influences the success metric.
assumptions (5)
  • domain assumption Randomly sampled exploration poses are representative of the pose distribution needed for task execution
    The pose estimator is trained only on uniform random poses within a workspace (Sec 3.4) but must estimate poses during precise, occluded task interactions; this distribution shift is assumed benign.
  • domain assumption Video Depth Anything provides depth maps consistent enough for pose estimation
    Depth for both training and runtime is produced by the off-the-shelf monocular estimator [55]; its error is not validated on the robot's workspace.
  • standard math Rectified flow converges to a valid generative video model
    The velocity model is trained with the rectified flow objective (Eq. 2); correctness of the flow formulation is taken from [56].
  • domain assumption CLIP text embeddings accurately encode task semantics
    Task descriptions are encoded with a CLIP text encoder (Sec 3.3); semantic alignment is assumed.
  • domain assumption The low-level IK controller can execute any feasible pose trajectory
    Estimated poses are translated to actuator commands via inverse kinematics; joint-limit feasibility is assumed during execution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generative Visual Foresight Meets Task-Agnostic Pose Estimation in Robotic Table-Top Manipulation." pith.science (2026). https://pith.science/paper/2KEIHSYF

@misc{pith2026250900361,
  author       = {Pith},
  title        = {Pith review of: Generative Visual Foresight Meets Task-Agnostic Pose Estimation in Robotic Table-Top Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2KEIHSYF}},
  note         = {Machine review of arXiv:2509.00361}
}
read the original abstract

Robotic manipulation in unstructured environments requires systems that can generalize across diverse tasks while maintaining robust and reliable performance. We introduce {GVF-TAPE}, a closed-loop framework that combines generative visual foresight with task-agnostic pose estimation to enable scalable robotic manipulation. GVF-TAPE employs a generative video model to predict future RGB-D frames from a single side-view RGB image and a task description, offering visual plans that guide robot actions. A decoupled pose estimation model then extracts end-effector poses from the predicted frames, translating them into executable commands via low-level controllers. By iteratively integrating video foresight and pose estimation in a closed loop, GVF-TAPE achieves real-time, adaptive manipulation across a broad range of tasks. Extensive experiments in both simulation and real-world settings demonstrate that our approach reduces reliance on task-specific action data and generalizes effectively, providing a practical and scalable solution for intelligent robotic systems.

Figures

Figures reproduced from arXiv: 2509.00361 by the authors.

Figure 1
Figure 1. High-level illustration of GVF-TAPE. Given a single RGB observation and a task description, GVF-TAPE predicts future RGB-D frames via a generative foresight model. A decoupled pose estimator then extracts end-effector poses, enabling closed-loop manipulation without action labels. Abstract: Robotic manipulation in unstructured environments requires systems that can gen￾eralize across diverse tasks while maintaining … view at source ↗
Figure 2
Figure 2. Framework Overview. GVF-TAPE first generates a future RGB-D video conditioned on the current RGB observation and task description. A transformer-based pose estimation model then extracts the end￾effector pose from each predicted frame and sends it to a low-level controller for execution. After completing the predicted trajectory, the system receives a new observation and repeats the process in a closed-loop manner. … view at source ↗
Figure 3
Figure 3. Challenging scenarios in LIBERO. The left two panels show tasks from LIVING-ROOM-SCENE-5, where the robot’s end effector moves outside the cam￾era’s field of view, making pose estimation unreliable. The right two panels illustrate limited gripper visibility from a fixed side￾view camera, which affects accuracy in fine￾grained tasks from LIBERO-Goal. Our current setup uses a single fixed camera, which can limit visib… view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: Performance of our method with and without pretraining. Using only 20% of the video data, our method matches prior SOTA (ATM); pretraining on LIBERO-90 boosts performance by 9.2%, outperforming ATM by 5.43%. To evaluate GVF-TAPE’s data efficiency, we pretrain the video…
Figure 5
Figure 5. Figure 5: Pretraining and model choice critically affect video generation quality and efficiency. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: (a) Real-world setup. We use an ARX-5 robotic arm equipped with a fixed side-view Intel Re￾alSense D435i camera. The evaluation environment includes dynamic contacts, deformable objects, back￾ground clutter, and varying lighting conditions. (b) Effect of human video pr…
Figure 7
Figure 7. Figure 7: An overview of the LIBERO benchmark. 7.3 Inference Time All real-world evaluations were performed on an NVIDIA RTX 4080 GPU. To improve inference speed, we utilized mixed-precision computation with TensorFloat-32 (TF32) tensor operations. The 15 [PITH_FULL_IMAGE:figur…
Figure 8
Figure 8. Figure 8: Comparison of model architecture. Performance evaluation using AUC for ResNet50, RGB 3- channel ViT, RGBD 4-channel ViT, and Depth-RGB cross attention model as pose estimation network, trained separately on same amount of random exploration data. Each point on the curv…
Figure 9
Figure 9. Figure 9: Real-world task setups for deformable object manipulation. (a) Fold the cloth: [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Eval environment roll out of successfully grabbing a tissue through multiple replans. The first and second rows show generated RGB and depth frames, respectively; the third row shows the real world environment. The robot arm fail to grab out the tissue during the firs…
Figure 11
Figure 11. Figure 11: Evaluation rollout of the system with Video-Depth-Anything successfully opening the drawer. The first and second rows show generated RGB and depth frames, respectively; the third row shows the simulation environment [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Evaluation rollout of the system without Video-Depth-Anything failing to open the drawer due to biased spatial pose estimation. The first row shows generated RGB frames; the second row shows the simulation environment. 7.10.3 Failure Analysis We summarize several fact…
Figure 13
Figure 13. Figure 13: Hallucination in the video generation model leads to task failure. The figure above illustrates a scenario where the model generates a novel bowl, resulting in failure to complete the task. The first and second rows display the generated RGB and depth frames, respecti…
Figure 14
Figure 14. Figure 14: Occultation of the gripper leads to failure. The figure above demonstrates a scenario where the robotic arm moves out of the camera’s view, resulting in unreliable pose estimation. The first and second rows display the generated RGB and depth frames, respectively, whi…
Figure 15
Figure 15. Figure 15: Pose estimation errors lead to failure. The figure above illustrates a scenario where the robotic arm fails to grasp the bowl due to inaccurate pose estimation. The first and second rows display the generated RGB and depth frames, respectively, while the third row dep…
Figure 16
Figure 16. Figure 16: Evaluation rollout of real world task pick up the blue bowl and place it on the pink plate. [PITH_FULL_IMAGE:figures/full_fig_p022_16.png]
Figure 17
Figure 17. Figure 17: Evaluation rollout of real world task grab a tissue. [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Evaluation rollout of real world task place the sponge on the plate. [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]
Figure 19
Figure 19. Figure 19: Evaluation rollout of real world task put the blue bowl into the microwave and close it. [PITH_FULL_IMAGE:figures/full_fig_p023_19.png]
Figure 20
Figure 20. Figure 20: Evaluation rollout of real world task put the pepper in the basket. [PITH_FULL_IMAGE:figures/full_fig_p024_20.png]
Figure 21
Figure 21. Figure 21: Evaluation rollout of real world task fold the cloth. [PITH_FULL_IMAGE:figures/full_fig_p024_21.png]
Figure 22
Figure 22. Figure 22: Evaluation rollout of real world task put the rag in the trash bin. [PITH_FULL_IMAGE:figures/full_fig_p024_22.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From World Models to World Action Models: A Concise Tutorial for Robotics

    cs.RO 2026-07 unverdicted novelty 4.0 of 10

    World models are action-conditioned predictors of task-relevant futures; world action models couple those futures to robot actions via four paradigms: imagine-then-execute, feature-conditioned, joint, and auxiliary pr...

Reference graph

Works this paper leans on

69 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    Pilacinski, A

    A. Pilacinski, A. Vandenberghe, G. Andrietta, and G. Vannuscorps. Humans underestimate the movement range of their own hands. Communications Psychology, 2(1):104, 2024. ISSN 2731 - 9121. doi:10.1038/s44271-024-00153-x. URL https://doi.org/10.1038/ s44271-024-00153-x

    work page doi:10.1038/s44271-024-00153-x 2024

  2. [2]

    K. C. Dieter, B. Hu, D. C. Knill, R. Blake, and D. Tadin. Kinesthesis can make an invisible hand visible. Psychological Science, 25(1):66 – 75, 2014. doi:10.1177/0956797613497968. URL https://doi.org/10.1177/0956797613497968

    work page doi:10.1177/0956797613497968 2014

  3. [3]

    Faivre, R

    N. Faivre, R. Salomon, and O. Blanke. Visual consciousness and bodily self - consciousness. Curr Opin Neurol, 28(1):23–28, 02 2015. doi:10.1097/WCO.0000000000000160

    work page doi:10.1097/wco.0000000000000160 2015

  4. [4]

    Yokosaka, S

    T. Yokosaka, S. Kuroki, S. Nishida, and J. Watanabe. Apparent time interval of visual stimuli is compressed during fast hand movement. PLoS One, 10(4):e0124901, 04 2015. doi:10.1371/ journal.pone.0124901

    work page 2015

  5. [5]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. Openvla: An open-source vision-language-action model, 2024. URL https://arxiv.org/abs/2406.09246

    arXiv 2024

  6. [6]

    Ghosh, H

    Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y . Tan, L. Y . Chen, P. Sanketi, Q. Vuong, T. Xiao, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, 2024

    2024

  7. [7]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. Leal, K.-H. Lee, S. Levine, Y . Lu, U. Malla, D. Man- junath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsc...

    arXiv 2023

  8. [8]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y . Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y . Lu, H. Michalewski, I. Mordatch, K. Pe...

    arXiv 2023

Show all 69 references
  1. [9]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Haus- man, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky. π0: A vision...

    2024 arXiv

  2. [10]

    S. Nair, A. Rajeswaran, V . Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation, 2022. URL https://arxiv.org/abs/2203.12601

    2022 arXiv

  3. [11]

    Jiang, Y

    G. Jiang, Y . Sun, T. Huang, H. Li, Y . Liang, and H. Xu. Robots pre-train robots: Manipulation- centric robotic representation from large-scale robot dataset.arXiv preprint arXiv:2410.22325, 2024. 10

    2024 arXiv

  4. [12]

    S. Ye, J. Jang, B. Jeon, S. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y .-W. Chao, B. Y . Lin, L. Liden, K. Lee, J. Gao, L. Zettlemoyer, D. Fox, and M. Seo. Latent action pretraining from videos, 2024. URL https://arxiv.org/abs/2410.11758

    2024 arXiv

  5. [13]

    Black, M

    K. Black, M. Nakamoto, P. Atreya, H. Walke, C. Finn, A. Kumar, and S. Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models, 2023. URL https: //arxiv.org/abs/2310.10639

    2023 arXiv

  6. [14]

    Y . Du, M. Yang, B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. arXiv e-prints , pages arXiv– 2302, 2023

    2023

  7. [15]

    Q. Bu, J. Zeng, L. Chen, Y . Yang, G. Zhou, J. Yan, P. Luo, H. Cui, Y . Ma, and H. Li. Closed- loop visuomotor control with generative expectation for robotic manipulation, 2024. URL https://arxiv.org/abs/2409.09016

    2024 arXiv

  8. [16]

    C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y . Gao, and P. Abbeel. Any-point trajectory modeling for policy learning, 2024. URL https://arxiv.org/abs/2401.00025

    2024 arXiv

  9. [17]

    M. Xu, Z. Xu, Y . Xu, C. Chi, G. Wetzstein, M. Veloso, and S. Song. Flow as the cross-domain manipulation interface, 2024. URL https://arxiv.org/abs/2407.15208

    2024 arXiv

  10. [19]

    P.-C. Ko, J. Mao, Y . Du, S.-H. Sun, and J. B. Tenenbaum. Learning to Act from Actionless Videos through Dense Correspondences. arXiv:2310.08576, 2023

    2023 arXiv

  11. [20]

    Luo and Y

    Y . Luo and Y . Du. Grounding video models to actions through goal conditioned exploration,

  12. [21]

    Liang, R

    J. Liang, R. Liu, E. Ozguroglu, S. Sudhakar, A. Dave, P. Tokmakov, S. Song, and C. V ondrick. Dreamitate: Real-world visuomotor policy learning via video generation, 2024. URLhttps: //arxiv.org/abs/2406.16862

    2024 arXiv

  13. [22]

    Shridhar, Y

    M. Shridhar, Y . L. Lo, and S. James. Generative image as action models, 2024. URLhttps: //arxiv.org/abs/2407.07875

    2024 arXiv

  14. [23]

    Y . Tian, S. Yang, J. Zeng, P. Wang, D. Lin, H. Dong, and J. Pang. Predictive inverse dynamics models are scalable learners for robotic manipulation, 2024. URL https://arxiv.org/ abs/2412.15109

    2024 arXiv

  15. [24]

    Cheang, G

    C.-L. Cheang, G. Chen, Y . Jing, T. Kong, H. Li, Y . Li, Y . Liu, H. Wu, J. Xu, Y . Yang, H. Zhang, and M. Zhu. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation, 2024. URL https://arxiv.org/abs/2410.06158

    2024 arXiv

  16. [25]

    H. Wu, Y . Jing, C. Cheang, G. Chen, J. Xu, X. Li, M. Liu, H. Li, and T. Kong. Unleashing large-scale video generative pre-training for visual robot manipulation, 2023. URL https: //arxiv.org/abs/2312.13139

    2023 arXiv

  17. [26]

    Merrill, Y

    N. Merrill, Y . Guo, X. Zuo, X. Huang, S. Leutenegger, X. Peng, L. Ren, and G. Huang. Sym- metry and uncertainty-aware object slam for 6dof object pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14901–14910, ...

    2022

  18. [27]

    Rad and V

    M. Rad and V . Lepetit. BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects Without Using Depth. In IEEE International Conference on Computer Vision (ICCV), Oct 2017. 11

    2017

  19. [28]

    W. Chen, X. Jia, H. J. Chang, J. Duan, and A. Leonardis. G2L-Net: Global to Local Network for Real-Time 6D Pose Estimation With Embedding Vector Features. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  20. [29]

    Y . He, W. Sun, H. Huang, J. Liu, H. Fan, and J. Sun. Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  21. [30]

    Tremblay, T

    J. Tremblay, T. To, B. Sundaralingam, Y . Xiang, D. Fox, and S. Birchfield. Deep object pose estimation for semantic robotic grasping of household objects, 2018. URL https: //arxiv.org/abs/1809.10790

    2018 arXiv

  22. [31]

    Y . Su, M. Saleh, T. Fetzer, J. Rambach, N. Navab, B. Busam, D. Stricker, and F. Tombari. Zebrapose: Coarse to fine surface encoding for 6dof object pose estimation, 2022. URL https://arxiv.org/abs/2203.09418

    2022 arXiv

  23. [32]

    Zakharov, I

    S. Zakharov, I. Shugurov, and S. Ilic. DPOD: 6D Pose Object Detector and Refiner. In The IEEE International Conference on Computer Vision (ICCV), October 2019

    2019

  24. [33]

    J. Sun, Z. Wang, S. Zhang, X. He, H. Zhao, G. Zhang, and X. Zhou. Onepose: One-shot object pose estimation without cad models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6825–6834, 2022

    2022

  25. [34]

    H. Chen, P. Wang, F. Wang, W. Tian, L. Xiong, and H. Li. Epro-pnp: Generalized end- to-end probabilistic perspective-n-points for monocular object pose estimation, 2022. URL https://arxiv.org/abs/2203.13254

    2022 arXiv

  26. [35]

    R. L. Haugaard and A. G. Buch. Surfemb: Dense and continuous correspondence distributions for object pose estimation with learnt surface embeddings.CoRR, abs/2111.13489, 2021. URL https://arxiv.org/abs/2111.13489

    2021 arXiv

  27. [36]

    Tekin, S

    B. Tekin, S. Sinha, and P. Fua. Real-time seamless single shot 6d object pose prediction. pages 292–301, 06 2018. doi:10.1109/CVPR.2018.00038

    2018

  28. [37]

    J. Zhou, K. Chen, L. Xu, Q. Dou, and J. Qin. Deep fusion transformer network with weighted vector-wise keypoints voting for robust 6d object pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 13967–13977, Oc- tober 2023

    2023

  29. [38]

    Linfang, L

    Z. Linfang, L. Ales, T. Tze Ho, Elden, H. Nora, C. Hua, Z. Wei, and C. Hyung Jin. Tp-ae: Temporally primed 6d object pose tracking with auto-encoders. In 2022 IEEE International Conference on Robotics and Automation (ICRA), 2022

    2022

  30. [39]

    M. Tian, M. H. Ang, and G. H. Lee. Shape prior deformation for categorical 6d object pose and size estimation. In European Conference on Computer Vision (ECCV) , pages 530–546. Springer, 2020

    2020

  31. [40]

    T. Lee, B. Lee, I. Shin, J. Choe, U. Shin, I. S. Kweon, and K. Yoon. UDA-COPE: unsupervised domain adaptation for category-level object pose estimation. CoRR, abs/2111.12580, 2021. URL https://arxiv.org/abs/2111.12580

    2021 arXiv

  32. [41]

    D. Chen, J. Li, Z. Wang, and K. Xu. Learning canonical shape space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  33. [42]

    M. Z. Irshad, S. Zakharov, R. Ambrus, T. Kollar, Z. Kira, and A. Gaidon. Shapo: Implicit representations for multi object shape appearance and pose optimization. 2022. URLhttps: //arxiv.org/abs/2207.13691. 12

    2022 arXiv

  34. [43]

    Zheng, T

    L. Zheng, T. H. E. Tse, C. Wang, Y . Sun, H. Chen, A. Leonardis, and W. Zhang. Georef: Geometric alignment across shape variation for category-level object pose refinement, 2024. URL https://arxiv.org/abs/2404.11139

    2024 arXiv

  35. [44]

    Zheng, C

    L. Zheng, C. Wang, Y . Sun, E. Dasgupta, H. Chen, A. Leonardis, W. Zhang, and H. J. Chang. Hs-pose: Hybrid scope feature extraction for category-level object pose estimation. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 17163– 17173, 20...

    2023

  36. [45]

    Huang, C

    W. Huang, C. Wang, Y . Li, R. Zhang, and L. Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation, 2024. URL https://arxiv. org/abs/2409.01652

    2024 arXiv

  37. [46]

    Y . Zuo, W. Qiu, L. Xie, F. Zhong, Y . Wang, and A. L. Yuille. Craves: Controlling robotic arm with a vision-based, economic system. CVPR, 2019

    2019

  38. [47]

    T. E. Lee, J. Tremblay, T. To, J. Cheng, T. Mosier, O. Kroemer, D. Fox, and S. Birchfield. Camera-to-robot pose estimation from a single image. CoRR, abs/1911.09231, 2019. URL http://arxiv.org/abs/1911.09231

    1911 arXiv

  39. [48]

    J. Lu, Z. Liang, T. Xie, F. Ritcher, S. Lin, S. Liu, and M. C. Yip. Ctrnet-x: Camera-to- robot pose estimation in real-world conditions using a single camera, 2024. URL https: //arxiv.org/abs/2409.10441

    2024 arXiv

  40. [49]

    Simoni, G

    A. Simoni, G. Borghi, L. Garattoni, G. Francesca, and R. Vezzani. D-spdh: Improving 3d robot pose estimation in sim2real scenario via depth data. IEEE Access, 12:166660–166673,

  41. [50]

    Y . Tian, J. Zhang, G. Huang, B. Wang, P. Wang, J. Pang, and H. Dong. Robokeygen: Robot pose and joint angles estimation via diffusion-based 3d keypoint generation, 2024. URL https://arxiv.org/abs/2403.18259

    2024 arXiv

  42. [51]

    doi:10.1109/ACCESS.2024.3492812

    2024

  43. [52]

    B. Wen, W. Yang, J. Kautz, and S. Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects, 2024. URL https://arxiv.org/abs/2312.08344

    2024 arXiv

  44. [53]

    Labb´e, J

    Y . Labb´e, J. Carpentier, M. Aubry, and J. Sivic. Single-view robot pose and joint angle estima- tion via render & compare, 2021. URL https://arxiv.org/abs/2104.09359

    2021 arXiv

  45. [54]

    Cheng, Y

    H. Cheng, Y . Wang, and M. Q.-H. Meng. Real-time robot end-effector pose estimation with deep network. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10921–10926, 2020. doi:10.1109/IROS45743.2020.9341760

    2020

  46. [55]

    T. Wu, J. Zhang, S. Liang, Z. Han, and H. Dong. Foundation feature-driven online end-effector pose estimation: A marker-free and learning-free approach, 2025. URL https://arxiv. org/abs/2503.14051

    2025 arXiv

  47. [56]

    X. Liu, C. Gong, and Q. Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow, 2022. URL https://arxiv.org/abs/2209.03003

    2022 arXiv

  48. [57]

    S. Chen, H. Guo, S. Zhu, F. Zhang, Z. Huang, J. Feng, and B. Kang. Video depth anything: Consistent depth estimation for super-long videos. arXiv:2501.12375, 2025

    2025 arXiv

  49. [58]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv.org/abs/2103.00020. 13

    2021 arXiv

  50. [59]

    Zheng, X

    Z. Zheng, X. Peng, T. Yang, C. Shen, S. Li, H. Liu, Y . Zhou, T. Li, and Y . You. Open-sora: Democratizing efficient video production for all, March 2024. URL https://github. com/hpcaitech/Open-Sora

    2024

  51. [60]

    B. Liu, Y . Zhu, C. Gao, Y . Feng, Q. Liu, Y . Zhu, and P. Stone. Libero: Benchmarking knowl- edge transfer for lifelong robot learning, 2023. URL https://arxiv.org/abs/2306. 03310

    2023

  52. [61]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR, abs/2010.11929, 2020. URL https:/...

    2010 arXiv

  53. [62]

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023

    2023

  54. [63]

    Baker, I

    B. Baker, I. Akkaya, P. Zhokhov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos,

  55. [64]

    J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  56. [65]

    D. Qu, H. Song, Q. Chen, Y . Yao, X. Ye, Y . Ding, Z. Wang, J. Gu, B. Zhao, D. Wang, et al. Spatialvla: Exploring spatial representations for visual-language-action model. arXiv preprint arXiv:2501.15830, 2025

    2025 arXiv

  57. [66]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  58. [67]

    Zheng, Y

    R. Zheng, Y . Liang, S. Huang, J. Gao, H. Daum ´e III, A. Kolobov, F. Huang, and J. Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. arXiv preprint arXiv:2412.10345, 2024. 14 7 Appendix 7.1 Performance comparison with ...

    2024 arXiv

  59. [69]

    S. Xu, Y . Wang, C. Xia, D. Zhu, T. Huang, and C. Xu. Vla-cache: Towards efficient vision- language-action model via adaptive token caching in robotic manipulation. arXiv preprint arXiv:2502.02175, 2025

    2025

  60. [2022]

    URL https://arxiv.org/abs/2206.11795

    arXiv

  61. [2024]

    URL https://arxiv.org/abs/2411.07223

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.