Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

EgoZero: Robot Learning from Smart Glasses

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EGOZERO claims that a robot can acquire closed-loop manipulation skills from in-the-wild egocentric human demonstrations captured only with smart glasses—no robot data, no teleoperation—and demonstrates 70% zero-shot success over seven…

desk verdict A real proof-of-concept for zero-robot-data manipulation from egocentric human video, but the 70% number rests on an unquantified DIFT transfer step that deserves an oracle-keypoint control. read the letter →

arxiv 2505.20290 v2 pith:5IYMMIQD submitted 2025-05-26 cs.RO cs.AI

Vincent Liu , Ademi Adeniji , Haotian Zhan , Siddhant Haldar , Raunaq Bhirangi , Pieter Abbeel , Lerrel Pinto This is my paper
classification cs.ROcs.AI
keywords egocentricimitationlearninghumandemonstrationspoint-basedrepresentationszero-shottransfermanipulationpoliciesProjectAriasmartglassesclosed-looppolicy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EGOZERO claims that a robot can learn closed-loop manipulation skills from nothing but in-the-wild human demonstrations captured with a pair of smart glasses, with no robot-collected data at all. The system turns human hand motions and object positions into a common 3D point-based state-action language, trains a Transformer policy by behavior cloning on that language, and then executes it on a different robot hand and a different camera from the one used during data collection. The headline result is an average 70% zero-shot success rate across seven tasks, from about 100 demonstrations (20 minutes) per task. The claim matters because if it holds, everyday human activity becomes a scalable source of robot training data that requires no specialized teleoperation hardware.

What carries the argument

The load-bearing object is the unified egocentric point-state space $\tilde{S} \times \tilde{A}$: object points and hand/end-effector action points expressed in the first demonstration frame. Action extraction composes Aria's 6DoF hand pose with HaMeR's local hand mesh so that fingertip points survive the camera change. Object localization solves a robust triangulation problem (Eq. 2) over the Aria trajectory, with epipolar filtering, RANSAC, Huber reprojection loss, and a soft depth penalty that counters point-tracker lag. On top of this representation, a closed-loop Transformer policy with action chunking and strong 3D augmentations maps point states to point actions.

What would settle it

Record demonstrations for a task like 'sort fruit' while nudging the target object partway through the reach (violating the stationary-object assumption), or replace DIFT correspondences with manually annotated ground-truth keypoints at inference; if success stays near 70% in the first case or fails to rise in the second, the paper's bottleneck attribution is wrong.

Watch

Extended reading notes

Core claim

The central discovery is that the human-robot morphology gap and the camera gap can both be closed by representing every observation and action as a set of 3D points in the human's egocentric frame, and by extracting those points from data sources that do not need depth sensors or multi-camera calibration. Object points are obtained by labeling keypoints once, transferring them to the start frame with pretrained correspondence models, and triangulating them across the Aria camera trajectory using SLAM pose estimates. Action points are obtained by combining Aria's 6DoF hand pose with a learned hand mesh model, giving thumb and index fingertip positions plus a gripper closure signal. The resulting demonstrations live in a morphology-agnostic state-action space, so a closed-loop behavior-cloned Transformer, trained with strong 3D augmentations, transfers zero-shot to a Franka Panda gripper whose hand and camera are entirely different from the human's.

Load-bearing premise

The load-bearing premise is that the pretrained correspondence stack (Grounding DINO, DIFT, CoTracker3) transfers expert-labeled keypoints from Aria's fisheye training frames to iPhone pinhole inference frames, and that each object stays stationary during the human demonstration; the paper itself reports DIFT as the largest source of inference error.

Editorial extensions

If this is right

  • With only 100 human demonstrations per task, the learned policies transfer to unseen object poses, new object instances, and a different inference camera (iPhone instead of Aria).
  • Closed-loop control is necessary: open-loop affordance baselines fail on tasks requiring nonlinear motions such as dragging bread onto a plate or pushing an eraser across a board.
  • 3D augmentations are necessary for in-the-wild transfer; without them, all 15 evaluations per task fail.
  • Monocular metric depth models are not a viable substitute for triangulation under Aria's fisheye distortion; depth-ablated policies fail completely.
  • The point-based representation is camera-agnostic, so the same policy can be deployed with a different camera without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the 70% result holds, the same point-space recipe should extend to bimanual and dexterous hands, because the action space is defined per fingertip rather than per gripper.
  • Editorial inference: the stationary-object assumption suggests an immediate stress test: record demonstrations while the target object moves slightly and predict that success drops; if it does not, the triangulation is more robust than the paper assumes.
  • Editorial inference: since DIFT is identified as the largest source of inference error, swapping in a stronger correspondence model or fine-tuning DIFT on fisheye-phone pairs is the most direct route to higher success, because the rest of the pipeline already operates in a shared 3D space.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes EgoZero, a pipeline for learning robot manipulation policies entirely from egocentric human demonstrations recorded with Project Aria smart glasses, with no robot-collected training data. Human demonstrations are converted into a morphology-agnostic state-action space: object states are 3D point sets obtained by triangulating tracked 2D keypoints over the camera trajectory, and actions are 3D thumb/index keypoint positions plus a binary gripper signal derived from Aria hand pose and HaMeR. A closed-loop transformer policy is trained by behavior cloning on these point states and actions, with 3D augmentation. At inference, object keypoints are detected on an iPhone frame with Grounding DINO and DIFT and unprojected with iPhone depth; a one-time Aruco calibration maps predicted points to the Franka Panda. Across seven manipulation tasks (100 demonstrations each, 15 trials per task), the method achieves 74/105 (70%) zero-shot success, while image-based and affordance baselines and two ablations score much lower, and the paper reports generalization to new object instances, viewpoints, and spatial configurations.

Significance. The result is significant if it holds: it provides a concrete, reproducible counterexample to the assumption that robot data or teleoperation is required for closed-loop manipulation policies, and it shows that point-based states and actions can absorb the Aria-to-iPhone camera gap. The paper's strengths include real-robot evaluation on seven tasks, zero robot data in training, honest enumeration of limitations, and release of code and videos. However, the evidential base is thinner than the abstract suggests: no confidence intervals or seeds, no quantitative evaluation of the DIFT correspondence step, and ablations that are all-or-nothing make it difficult to determine whether the reported 70% success is robust or dominated by the pretrained correspondence stack.

major comments (4)
  1. [Sec. 3.1, Sec. 4.5, Algorithm 1] The inference-time object state is produced by Grounding DINO + DIFT mapping expert-labeled 2D keypoints from Aria training frames to an iPhone frame, followed by depth unprojection (Algorithm 1). The paper states in Sec. 4.5 that DIFT is the largest source of inference error, but it does not report a single quantitative accuracy number for this cross-camera, cross-instance correspondence step, nor does it include an ablation that replaces DIFT with oracle keypoints (e.g., manually labeled or template-matched keypoints in the iPhone frame). Because the policy conditions on this point state and has no visual feedback, a corrupted keypoint set cannot be corrected during rollout; the 70% average could therefore reflect DIFT succeeding on easy trials and failing silently on hard ones. An oracle-keypoint control and per-task correspondence success rates are needed to separate correspondence error from policy error.
  2. [Table 1, Sec. 4.1] Each task is evaluated on only 15 trials, and no confidence intervals, multiple training seeds, or per-trial breakdowns are reported. With 15 Bernoulli trials, the 95% confidence interval for 9/15 is approximately [0.38, 0.79], and the overall 74/105 rate has a CI of roughly [0.61, 0.78]; several apparent differences between EGOZERO and the affordance baseline (e.g., 11/15 vs 7/15 on sort fruit) are within sampling noise. Please report CIs (at least Wilson intervals), the number of seeds, and ideally per-task videos or failure categories so the reader can judge trial difficulty.
  3. [Sec. 4.3] All ablations and the vision baseline score 0/15 across all tasks, making the comparisons all-or-nothing. This pattern is unusual and limits interpretability: a reader cannot tell whether removing 3D augmentation or triangulated depth leads to complete policy collapse, or whether the evaluation protocol simply lacks grading of partial progress. Reporting graded metrics (e.g., proportion of trajectory completed, distance to goal, contact with target object) and a small number of non-binary evaluation criteria would substantiate the claim that each component is necessary.
  4. [Sec. 3.2, Algorithm 1] The policy is described as closed-loop, but the object-point state ~s is computed once at initialization and never updated during rollout; feedback is limited to robot proprioception and action history. This is a meaningful scope limitation: if the object moves, is occluded, or the initial keypoint localization is wrong, the policy cannot correct. The term "closed-loop" should be qualified, and the paper should state clearly whether any visual re-detection occurs during the episode.
minor comments (5)
  1. [Sec. 3.1] The value of lambda in Eq. (2) is never given; please report it and any sensitivity analysis.
  2. [Sec. 3.1] The phrase "We linearize It as a 1408x1408 RGB image" should be reworded to "rectified/undistorted" for clarity.
  3. [Sec. 4.4] The camera-generalization claim is based on a single iPhone camera model; please state whether results are expected to hold across other pinhole cameras and whether iPhone depth accuracy was validated.
  4. [Sec. 3.2] The demonstration-filtering rule (1 median absolute deviation from closest fingertip) is described without reporting how many demonstrations were discarded per task; please include this number.
  5. [Abstract and Sec. 1] The phrase "zero-shot in-the-wild transfer" is used although inference occurs in a robot workspace with a fixed camera and calibration; consider rephrasing to "zero-shot transfer from in-the-wild human demonstrations" to avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 70% zero-shot success claim is an empirically measured robot evaluation, not a quantity derived from fitted constants or self-cited premises.

full rationale

The central claim is supported by 15 physical Franka Panda rollouts per task (Table 1), reported as raw success counts that are not computed from any fitted parameter. The object state (Eq. 2) is obtained by RANSAC triangulation with a hand-tuned depth penalty lambda; the action labels (Eq. 1) compose Aria MPS hand pose with HaMeR keypoints; and the policy is trained with standard behavior cloning (Eq. 3). None of these equations defines the reported success rate in terms of its own inputs. The paper's self-citations ([16] Point Policy, [27] P3-PO, [62] Baku) supply the point-representation and transformer design, but the empirical transfer result is evaluated anew on unseen object poses, camera (iPhone vs Aria fisheye), and environments, so the citations are not load-bearing derivations. Section 4.5 explicitly identifies DIFT correspondence error as the largest inference error source; this is an acknowledged robustness limitation and a reason to treat the 70% figure cautiously, but it is not a circularity because DIFT's outputs are not fitted to the evaluation outcomes. The triangulation and hand-model limitations similarly bound performance without making the prediction equivalent to the training data by construction. Overall, the derivation chain is self-contained: inputs (human videos, Aria MPS, pretrained models) are external and the output (robot success rate) is measured independently.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim depends on a long chain of pretrained components and modeling choices. The free parameters are mostly hand-set hyperparameters (lambda, thresholds, augmentation ranges); none are fitted to the success numbers, but they are tuned to make the pipeline work. The axioms include the reliability of Aria MPS and HaMeR, the transferability of DIFT/CoTracker3, and the static-scene triangulation assumptions. No new physical entities are introduced.

free parameters (6)
  • Depth penalty weight lambda = unspecified
    In Eq. (2)/(7), biases triangulation toward closer solutions to counter CoTracker3 lag. Hand-tuned, no value reported.
  • Policy loss variance sigma = 0.1
    Fixed variance in the negative log-likelihood objective (Eq. 3), from prior work [62,27] but a free choice affecting training.
  • 3D augmentation ranges = rotations U(-pi/6, pi/6), translations U(-0.5, 0.5) m
    Hand-set augmentation magnitudes shown necessary for transfer (Sec. 4.3).
  • Stationary point removal threshold = 1 cm
    Throws out consecutive points with Euclidean distance under 1cm to disambiguate proprioceptive position from grasp closure.
  • Grasp detection distance threshold = unspecified
    Distance between thumb and index used to binarize gripper closure; not numerically reported.
  • Demonstration filtering threshold = 1 median absolute deviation
    Discards demonstrations whose object points are more than 1 MAD from the closest fingertip point, to remove DIFT failures.
assumptions (5)
  • domain assumption Aria MPS provides accurate 6DoF hand pose and camera intrinsics/extrinsics.
    Sec. 3.1 relies on MPS outputs as ground truth for the entire state and action extraction.
  • domain assumption HaMeR hand keypoints and Aria hand pose can be composed into accurate robot-executable actions.
    Sec. 3.1 Eq. (1) composes local HaMeR deformation with Aria egocentric pose; Sec. 4.5 admits 1-2cm action-label error.
  • domain assumption Pretrained correspondence models (CoTracker3, DIFT, Grounding DINO) transfer across domains and time.
    Sec. 3.1 uses them for tracking and labeling; Sec. 4.5 identifies DIFT as the largest inference error source.
  • domain assumption Objects are stationary and the environment is non-stochastic during the demonstration.
    Explicitly stated in Sec. 3.1 as required for triangulation over the camera trajectory.
  • domain assumption Point-based state/action representations are sufficient for closed-loop manipulation control.
    Inherited from [16,27]; the paper does not question this and trains the entire policy on point sets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EgoZero: Robot Learning from Smart Glasses." pith.science (2026). https://pith.science/paper/5IYMMIQD

@misc{pith2026250520290,
  author       = {Pith},
  title        = {Pith review of: EgoZero: Robot Learning from Smart Glasses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5IYMMIQD}},
  note         = {Machine review of arXiv:2505.20290}
}
abstract

Despite recent progress in general purpose robotics, robot policies still lag far behind basic human capabilities in the real world. Humans interact constantly with the physical world, yet this rich data resource remains largely untapped in robot learning. We propose EgoZero, a minimal system that learns robust manipulation policies from human demonstrations captured with Project Aria smart glasses, $\textbf{and zero robot data}$. EgoZero enables: (1) extraction of complete, robot-executable actions from in-the-wild, egocentric, human demonstrations, (2) compression of human visual observations into morphology-agnostic state representations, and (3) closed-loop policy learning that generalizes morphologically, spatially, and semantically. We deploy EgoZero policies on a gripper Franka Panda robot and demonstrate zero-shot transfer with 70% success rate over 7 manipulation tasks and only 20 minutes of data collection per task. Our results suggest that in-the-wild human data can serve as a scalable foundation for real-world robot learning - paving the way toward a future of abundant, diverse, and naturalistic training data for robots. Code and videos are available at https://egozero-robot.github.io.

Figures

Figures reproduced from arXiv: 2505.20290 by the authors.

Figure 1
Figure 1. EGOZERO trains policies in a unified state-action space defined as egocentric 3D points. Unlike previous methods which leverage multi-camera calibration and depth sensors, EGOZERO localizes object points via triangulation over the camera trajectory, and computes action points via Aria MPS hand pose and a hand estimation model. These points supervise a closed-loop Transformer policy, which is rolled out on unprojecte… view at source ↗
Figure 2
Figure 2. Our 7 tasks. Top: open oven door, put bread on plate, sweep board with broom, erase [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Distribution of bread keypoints for “Put bread in plate” task. The columns are projections [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Object semantic generalization. Human demonstrations are done with only black ovens [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Open oven door [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Put bread on plate [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Sweep board with broom. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Erase board [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Sort fruit in bowl [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Fold towel. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Insert book in shelf. B Triangulating Object Keypoints We estimate 3D coordinates q ∗ ∈ R 3 of an object point in the world frame at t = 0 from 2D observations {(Ti , ui)} N i=1, where ui ∈ R 2 is the UV coordinate tracked in frame i, and Ti ∈ SE(3) is the camera-to-w…
Figure 12
Figure 12. Figure 12: Monocular depth estimation [64] calibrated to Aruco tags in the scene. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World

    cs.RO 2026-04 unverdicted novelty 6.5 of 10

    EgoVerse releases 1,362 hours of standardized egocentric human data across 1,965 tasks and shows via multi-lab experiments that robot policy performance scales with human data volume when the data aligns with robot ob...

  2. EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration

    cs.RO 2026-07 conditional novelty 6.0 of 10

    EgoRecovery co-trains robot recovery policies from egocentric human recovery videos by sharing a gated corrective-intent bottleneck, raising average closed-loop recovery success from 52.5% (robot-only) to 85% on four ...

  3. Object-centric 3D Motion Field for Robot Learning from Human Videos

    cs.RO 2025-06 conditional novelty 6.0 of 10

    A policy trained only on human RGBD videos, with a denoised object-centric 3D motion field as action representation, achieves about 55% average success on five real manipulation tasks where prior flow-based methods st...

  4. Scene Text Detection and Recognition "in light of" Challenging Environmental Conditions using Aria Glasses Egocentric Vision Cameras

    cs.CV 2025-07 conditional novelty 5.0 of 10

    On a small custom dataset captured with Aria glasses, distance and resolution drive OCR errors more than lighting, and 2x image upscaling cuts CER from 0.65 to 0.48 for the EAST+CRNN pipeline.

Reference graph

Works this paper leans on

71 extracted references · 26 canonical work pages · cited by 4 Pith papers

  1. [1]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. URLhttps://arxiv.org/abs/2303.08774

    arXiv 2024

  2. [2]

    Language models are few-shot learners.CoRR, abs/2005.14165, 2020

    OpenAI. Language models are few-shot learners.CoRR, abs/2005.14165, 2020. URLhttps: //arxiv.org/abs/2005.14165

    arXiv 2005

  3. [3]

    Ramesh, M

    A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. V oss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation, 2021. URLhttps://arxiv.org/abs/2102.12092

    arXiv 2021

  4. [4]

    Rombach, A

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image syn- thesis with latent diffusion models, 2022. URLhttps://arxiv.org/abs/2112.10752

    arXiv 2022

  5. [5]

    Imagen 3, 2024

    Imagen-Team-Google. Imagen 3, 2024. URLhttps://arxiv.org/abs/2408.07009

    arXiv 2024

  6. [6]

    Brooks, B

    T. Brooks, B. Peebles, C. Holmes, W. DePue, Y . Guo, L. Jing, D. Schnurr, J. Tay- lor, T. Luhman, E. Luhman, C. Ng, R. Wang, and A. Ramesh. Video genera- tion models as world simulators. 2024. URLhttps://openai.com/research/ video-generation-models-as-world-simulators

    work page 2024

  7. [7]

    Blattmann, T

    A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y . Levi, Z. En- glish, V . V oleti, A. Letts, V . Jampani, and R. Rombach. Stable video diffusion: Scaling la- tent video diffusion models to large datasets, 2023. URLhttps://arxiv.org/abs/2311. 15127

    work page 2023

  8. [8]

    Evans, J

    Z. Evans, J. D. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons. Stable audio open, 2024. URLhttps://arxiv.org/abs/2407.14358

    arXiv 2024

Show all 71 references
  1. [9]

    C. Wang, S. Chen, Y . Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y . Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei. Neural codec language models are zero-shot text to speech synthesizers,

  2. [10]

    Radford, J

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URLhttps://arxiv.org/abs/2212. 04356

    2022

  3. [11]

    Silver, T

    D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.Science, 362(6419): ...

    2018 doi

  4. [12]

    Dota 2 with large scale deep reinforcement learning, 2019

    OpenAI. Dota 2 with large scale deep reinforcement learning, 2019. URLhttps://arxiv. org/abs/1912.06680

    2019 arXiv

  5. [13]

    Collaboration

    E. Collaboration. Open x-embodiment: Robotic learning datasets and rt-x models, 2024. URL https://arxiv.org/abs/2310.08864

    2024 arXiv

  6. [14]

    C. Wang, H. Shi, W. Wang, R. Zhang, L. Fei-Fei, and C. K. Liu. Dexcap: Scalable and portable mocap data collection system for dexterous manipulation, 2024. URLhttps://arxiv.org/ abs/2403.07788. 11

    2024 arXiv

  7. [15]

    Kareer, D

    S. Kareer, D. Patel, R. Punamiya, P. Mathur, S. Cheng, C. Wang, J. Hoffman, and D. Xu. Egomimic: Scaling imitation learning via egocentric video, 2024. URLhttps://arxiv. org/abs/2410.24221

    2024 arXiv

  8. [16]

    Haldar and L

    S. Haldar and L. Pinto. Point policy: Unifying observations and actions with key points for robot manipulation, 2025. URLhttps://arxiv.org/abs/2502.20391

    2025 arXiv

  9. [17]

    Guzey, Y

    I. Guzey, Y . Dai, G. Savva, R. Bhirangi, and L. Pinto. Bridging the human to robot dexterity gap through object-oriented rewards, 2024. URLhttps://arxiv.org/abs/2410.23289

    2024 arXiv

  10. [18]

    S. Bahl, A. Gupta, and D. Pathak. Human-to-robot imitation in the wild, 2022. URLhttps: //arxiv.org/abs/2207.09450

    2022 arXiv

  11. [19]

    J. Shi, Z. Zhao, T. Wang, I. Pedroza, A. Luo, J. Wang, J. Ma, and D. Jayaraman. Zeromimic: Distilling robotic manipulation skills from web videos, 2025. URLhttps://arxiv.org/ abs/2503.23877

    2025 arXiv

  12. [20]

    Lepert, J

    M. Lepert, J. Fang, and J. Bohg. Phantom: Training robots without robots using only human videos, 2025. URLhttps://arxiv.org/abs/2503.00779

    2025 arXiv

  13. [21]

    Khazatsky

    A. Khazatsky. Droid: A large-scale in-the-wild robot manipulation dataset, 2024. URLhttps: //arxiv.org/abs/2403.12945

    2024 arXiv

  14. [22]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale.arXiv preprint arXiv:2212.06817, 2022

    2022 arXiv

  15. [23]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

    2023 arXiv

  16. [24]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Haus- man, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky.π 0: A vision...

    2024 arXiv

  17. [25]

    Intelligence, K

    P. Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y . Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A....

    2025 arXiv

  18. [26]

    Engel, K

    J. Engel, K. Somasundaram, M. Goesele, A. Sun, A. Gamino, A. Turner, A. Talattof, A. Yuan, B. Souti, B. Meredith, C. Peng, C. Sweeney, C. Wilson, D. Barnes, D. DeTone, D. Caruso, D. Valleroy, D. Ginjupalli, D. Frost, E. Miller, E. Mueggler, E. Oleinik, F. Zhang, G. Soma- sunda...

    2023 arXiv

  19. [27]

    M. Levy, S. Haldar, L. Pinto, and A. Shirivastava. P3-po: Prescriptive point priors for visuo- spatial generalization of robot policies, 2024. URLhttps://arxiv.org/abs/2412.06784. 12

    2024 arXiv

  20. [28]

    Mandlekar, D

    A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y . Zhu, and R. Mart´ın-Mart´ın. What matters in learning from offline human demonstrations for robot manipulation. In5th Annual Conference on Robot Learning, 2021. URLhttps: //openrevie...

    2021

  21. [29]

    Mandlekar, J

    A. Mandlekar, J. Booher, M. Spero, A. Tung, A. Gupta, Y . Zhu, A. Garg, S. Savarese, and L. Fei-Fei. Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity. In2019 IEEE/RSJ International Conference on In...

    2019

  22. [30]

    E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. BC-Z: zero-shot task generalization with robotic imitation learning.CoRR, abs/2202.02005, 2022. URLhttps://arxiv.org/abs/2202.02005

    2022 arXiv

  23. [31]

    B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration.Robotics and autonomous systems, 57(5):469–483, 2009

    2009

  24. [32]

    Hussein, M

    A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods.ACM Computing Surveys (CSUR), 50(2):1–35, 2017

    2017

  25. [33]

    T. Z. Zhao, V . Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware.arXiv preprint arXiv:2304.13705, 2023

    2023 arXiv

  26. [34]

    T. Z. Zhao, J. Tompson, D. Driess, P. Florence, K. Ghasemipour, C. Finn, and A. Wahid. Aloha unleashed: A simple recipe for robot dexterity, 2024. URLhttps://arxiv.org/abs/2410. 13126

    2024

  27. [35]

    P. Wu, Y . Shentu, Z. Yi, X. Lin, and P. Abbeel. Gello: A general, low-cost, and intuitive tele- operation framework for robot manipulators, 2024. URLhttps://arxiv.org/abs/2309. 13037

    2024

  28. [36]

    Kang and K

    S.-R. Kang and K. Ikeuchi. Toward automatic robot instruction from perception—mapping human grasps to manipulator grasps. InProceedings of IEEE International Conference on Robotics and Automation, pages 1932–1937. IEEE, 1994

    1932

  29. [37]

    Ikeuchi, K

    K. Ikeuchi, K. Minamizawa, K. Harada, A. Yamaguchi, and S. Kagami. Semantic con- straints to represent common sense required in household actions for multimodal learning- from-observation robot.The International Journal of Robotics Research, 43(4):399–414, 2024. doi:10.1177/02...

    2024 doi

  30. [38]

    [Virtual reality platform]

    Meta quest.https://www.meta.com/quest/, 2024. [Virtual reality platform]

    2024

  31. [39]

    [Virtual reality platform]

    Apple vision pro.https://www.apple.com/apple-vision-pro/, 2024. [Virtual reality platform]

    2024

  32. [40]

    [Virtual re- ality platform]

    Steamvr.https://store.steampowered.com/app/250820/SteamVR/, 2024. [Virtual re- ality platform]

    2024

  33. [41]

    Mannam, K

    P. Mannam, K. Shaw, D. Bauer, J. Oh, D. Pathak, and N. Pollard. Designing anthropo- morphic soft hands through interaction. In2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), pages 1–8, 2023. doi:10.1109/Humanoids57100.2023. 10375195

    2023

  34. [42]

    [Motion capture sys- tem]

    Movella xsens.https://www.movella.com/products/xsens, 2024. [Motion capture sys- tem]

    2024

  35. [43]

    [Motion capture gloves]

    Manusmetagloves.https://www.manus-meta.com, 2024. [Motion capture gloves]

    2024

  36. [44]

    [Motion capture solution]

    Rokoko.https://www.rokoko.com, 2024. [Motion capture solution]. 13

    2024

  37. [45]

    D. Shan, J. Geng, M. Shu, and D. Fouhey. Understanding human hands in contact at internet scale. InCVPR, 2020

    2020

  38. [46]

    Grauman, A

    K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19...

    2022

  39. [47]

    Damen, H

    D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pages 720–736, 2018

    2018

  40. [48]

    Goyal, S

    R. Goyal, S. Ebrahimi Kahou, V . Michalski, J. Materzynska, S. Westphal, H. Kim, V . Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic. The ”something something” video database for learning and evaluating visual common sense. InP...

    2017

  41. [49]

    Y .-W. Chao, W. Yang, Y . Xiang, P. Molchanov, A. Handa, J. Tremblay, Y . S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield, et al. Dexycb: A benchmark for capturing hand grasp- ing of objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2021

  42. [50]

    Qin, Y .-H

    Y . Qin, Y .-H. Wu, S. Liu, H. Jiang, R. Yang, Y . Fu, and X. Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. InEuropean Conference on Computer Vision, pages 570–587. Springer, 2022

    2022

  43. [51]

    Papagiannis, N

    G. Papagiannis, N. D. Palo, P. Vitiello, and E. Johns. R+x: Retrieval and execution from everyday human videos, 2024. URLhttps://arxiv.org/abs/2407.12957

    2024 arXiv

  44. [52]

    S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos as a versatile representation for robotics. 2023

    2023

  45. [53]

    H. G. Singh, A. Loquercio, C. Sferrazza, J. Wu, H. Qi, P. Abbeel, and J. Malik. Hand-object interaction pretraining from videos, 2024. URLhttps://arxiv.org/abs/2409.08273

    2024 arXiv

  46. [54]

    Pavlakos, D

    G. Pavlakos, D. Shan, I. Radosavovic, A. Kanazawa, D. Fouhey, and J. Malik. Reconstructing hands in 3d with transformers.arXiv preprint arXiv:2312.05251, 2023

    2023 arXiv

  47. [55]

    Zhang, Q

    X. Zhang, Q. Li, H. Mo, W. Zhang, and W. Zheng. End-to-end hand mesh recovery from a monocular rgb image. In2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2354–2364, 2019. doi:10.1109/ICCV .2019.00244

    2019

  48. [56]

    S. Baek, K. I. Kim, and T. Kim. Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering.CoRR, abs/1904.04196, 2019. URLhttp://arxiv.org/ abs/1904.04196

    1904 arXiv

  49. [57]

    Boukhayma, R

    A. Boukhayma, R. A. de Bem, and P. H. S. Torr. 3d hand shape and pose from images in the wild.CoRR, abs/1902.03451, 2019. URLhttp://arxiv.org/abs/1902.03451

    1902 arXiv

  50. [58]

    Romero, D

    J. Romero, D. Tzionas, and M. J. Black. Embodied hands: Modeling and capturing hands and bodies together.CoRR, abs/2201.02610, 2022. URLhttps://arxiv.org/abs/2201. 02610

    2022 arXiv

  51. [59]

    Karaev, I

    N. Karaev, I. Makarov, J. Wang, N. Neverova, A. Vedaldi, and C. Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos, 2024. URLhttps: //arxiv.org/abs/2410.11831

    2024 arXiv

  52. [60]

    S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2024. URLhttps://arxiv.org/abs/2303.05499. 14

    2024 arXiv

  53. [61]

    L. Tang, M. Jia, Q. Wang, C. P. Phoo, and B. Hariharan. Emergent correspondence from image diffusion, 2023. URLhttps://arxiv.org/abs/2306.03881

    2023 arXiv

  54. [62]

    Haldar, Z

    S. Haldar, Z. Peng, and L. Pinto. Baku: An efficient transformer for multi-task policy learning,

  55. [63]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. Openvla: An open-source vision-language-action model, 2024. URL https...

    2024 arXiv

  56. [64]

    Bochkovskii, A

    A. Bochkovskii, A. Delaunoy, H. Germain, M. Santos, Y . Zhou, S. R. Richter, and V . Koltun. Depth pro: Sharp monocular metric depth in less than a second. InInternational Conference on Learning Representations, 2025. URLhttps://arxiv.org/abs/2410.02073

    2025 arXiv

  57. [65]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V . Alwala, N. Carion, C.-Y . Wu, R. Girshick, P. Doll ´ar, and C. Feichtenhofer. Sam 2: Segment anything in images and videos, 2024. URLhttps: //arxiv...

    2024 arXiv

  58. [68]

    Given two framesiandj, we compute the fundamental matrix: Fij =K −T [tij]×RijK −1,(4) whereR ij =R jR⊤ i ,t ij =t j −R ijti, and[·] × is the skew-symmetric matrix

    Epipolar Filtering.To discard geometrically inconsistent views, we apply pairwise epipolar constraints. Given two framesiandj, we compute the fundamental matrix: Fij =K −T [tij]×RijK −1,(4) whereR ij =R jR⊤ i ,t ij =t j −R ijti, and[·] × is the skew-symmetric matrix. A frameii...

  59. [69]

    Robust RANSAC Triangulation.Using the filtered inlier views, we perform RANSAC over subsets of sizekto find the best triangulated candidateq ∗ minimizing reprojection error: qRANSAC = arg min q X i∈I ⊮ ui − P(T−1 i q) 2 < τ .(6)

  60. [70]

    This formulation encour- ages geometrically consistent triangulation while avoiding ambiguous far-away solutions in cases of degenerate motion or lag in Cotracker3 predictions

    Least Squares with Depth Bias.We refineq RANSAC via nonlinear least squares with a Huber loss and a soft depth penalty: q∗ = arg min q∈Ω X i∈I ui − P(T−1 i q) ρ +λq z,(7) where|| · ||ρ is the Huber loss,q z is the depth (z-coordinate in world frame),λis the depth bias coeffici...

  61. [71]

    Unified Object Representations.We repeat Steps 1-3 for each point that we label on the object, and concatenate each triangulated object point to obtain the object representation for the entire trajectory˜s. C Policy Inference Algorithm 1EGOZEROPolicy Inference 1:Obtain object ...

  62. [2023]

    URLhttps://arxiv.org/abs/2301.02111

    arXiv

  63. [2024]

    URLhttps://arxiv.org/abs/2406.07539

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.