Pith. sign in

REVIEW 4 major objections 4 minor 70 references

VLAff: Vision-Language-Affordance Model for Unified Actionable Affordances

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single vision-language model that jointly predicts where to touch, how to grasp, and how to move can transfer human-video skills to robots without robot action data.

desk verdict Large new dataset and a sensible unified affordance model, but the headline results are measured against self-generated labels and underpowered robot trials. read the letter →

arxiv 2608.05215 v1 pith:IMOLGPBX submitted 2026-08-05 cs.RO cs.CV

Jihoon Oh , Kento Kawaharazuka , Kei Okada This is my paper · ORCID
classification cs.ROcs.CV
keywords affordancelearningvision-language-actionmodelszero-shotroboticmanipulationegocentricvideorobotfromhumangraspposeestimationtrajectorygenerationEgodataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VLAff argues that a robot does not need robot-specific action data to learn manipulation: the where, how, and how-to-move of a task can be extracted from ordinary egocentric human video as object-centric affordances, and a single vision-language model can learn to emit all three at once. The paper builds EgoAffordance, a dataset of over 200K episodes with 5.6M visual affordance heatmaps and 11.6M grasp and trajectory affordances generated automatically from hand-object interactions in egocentric video. VLAff then predicts these affordances from an image and a natural-language instruction, and converts them into executable robot actions using 3D scene geometry. The result is the best reported visual affordance prediction and a 68 percent real-world zero-shot manipulation success rate, 16 points above prior work. If correct, this shows that human videos can substitute for expensive robot demonstration data across many manipulation tasks.

What carries the argument

The load-bearing object is the unified affordance triple $f_{\text{VLAff}}(I,L)=(A_v,A_g,A_t)$ produced by a vision-language model extended with three token types: a <SEG> token whose embedding feeds a segmentation decoder for the visual heatmap, a <GRASP> token decoded into 96-dimensional MANO hand parameters, and discretized trajectory tokens generated autoregressively through spatial binning of a 6D pose space. Two inference strategies convert these outputs into physically plausible 3D actions: in-context trajectory guidance, in which a high-level VLM supplies a coarse 3D direction vector used to sample the first trajectory token, and sampling-based selection, in which K candidate trajectories are scored against voxelized scene occupancy for collision avoidance. All grasp and trajectory predictions are expressed relative to the 3D interaction point derived from the heatmap, which is what makes the affordances object-centric rather than tied to a particular hand or robot.

What would settle it

Independently acquire 3D ground truth for a sample of EgoAffordance episodes using a calibrated multi-camera or RGB-D motion-capture setup, correct the contact points and trajectories accordingly, retrain VLAff, and also evaluate on a held-out domain not represented in the training videos. If performance drops to near the baselines, the reported gains reflect the label-generation pipeline and training-test overlap rather than transferable skill.

Watch

Extended reading notes

Core claim

The paper's central claim is that jointly predicting three complementary actionable affordances—a visual affordance heatmap $A_v\in\mathbb{R}^{H\times W}$ indicating where to interact, a grasp affordance $A_g\in\mathbb{R}^{96}$ (MANO hand parameters in a 6D rotation representation) encoding how the hand holds, and a trajectory $A_t\in\mathbb{R}^{T\times 6}$ describing how the hand moves—within a large vision-language model produces better affordance grounding and policies that transfer directly to robots. The authors state that this joint training is why VLAff outperforms methods that learn affordance modalities in isolation, and that anchoring the grasp and trajectory at the heatmap's peak interaction point makes the representation object-centric and embodiment-agnostic. Empirically, VLAff reports the best visual affordance metrics in its comparison (IoU 0.121, NSS 1.542) and an average real-world zero-shot manipulation success rate of 68.0 percent, 16 points above the VidBot baseline.

Load-bearing premise

The automatically generated contact points (fingertip–object mask intersections) and 3D trajectories (from monocular depth, structure-from-motion, and inpainting) are accurate enough to be treated as ground truth labels, and the visual-affordance test set drawn from the same videos does not hide systematic errors in those labels.

Editorial extensions

If this is right

  • Human egocentric video can replace a large fraction of robot-specific action-state data used in imitation learning and vision-language-action models, lowering the cost of building generalist manipulation policies.
  • Joint training of the three affordance modalities improves each one: visual affordance prediction beats isolated baselines across all reported metrics.
  • The 68 percent real-world zero-shot success rate indicates that affordances alone, without any robot-specific training, are enough to complete a meaningful fraction of everyday kitchen tasks on two different robot platforms.
  • Affordance-guided reinforcement learning converges faster when all three modalities are used; removing the visual heatmap makes the policy fail entirely, showing the heatmap is the dominant cue.
  • The object-centric formulation transfers across embodiments, since grasp and trajectory are expressed relative to the interaction point rather than to a particular robot kinematic frame.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the pseudo-label pipeline is systematically biased (for instance, contact points biased by hand morphology or trajectories biased by egocentric depth errors), the visual affordance and zero-shot gains could partly reflect internal consistency rather than true physical correctness; an independent 3D ground-truth check would settle this.
  • The real-world superiority over VidBot might be driven less by joint affordance learning than by the added in-context trajectory guidance from an external high-level VLM, which injects a privileged semantic prior during inference; ablating the guidance would isolate this.
  • The same extraction pipeline could be applied to third-person or mixed-view human videos, removing the egocentric bias in depth and occlusion and potentially scaling the dataset further.
  • The object-centric trajectory anchoring should make VLAff usable with any fixed-base or mobile manipulator that has a parallel gripper; testing on a wider range of embodiments would clarify how embodiment-agnostic the representation really is.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes VLAff, a vision-language-action affordance model that jointly predicts visual affordance heatmaps, MANO grasp parameters, and trajectory tokens from an RGB image and a language instruction. To train it, the authors build EgoAffordance, a large-scale dataset of 204K episodes with 5.6M visual affordances and 11.6M grasp/trajectory affordances, extracted from egocentric human videos through hand-object detection, hand mesh reconstruction, monocular depth, SfM, and inpainting (Section III). VLAff extends a pretrained VLM (Qwen2.5-VL) with <SEG>, <GRASP>, and trajectory tokens, and is trained with soft Dice, Smooth L1, and cross-entropy losses (Section IV). Experiments (Section V) claim state-of-the-art visual affordance prediction on a test set of 500 scenes per egocentric dataset, zero-shot manipulation in simulation (83.0% vs VidBot's 85.0%) and real-world (68.0% vs VidBot's 52.0%), and success of affordance-guided RL on an Open Fridge task.

Significance. If the empirical claims hold, the paper makes a substantial contribution: it demonstrates that automatically extracted, embodiment-agnostic actionable affordances from human videos can support a unified VLM-based model that transfers zero-shot to real robots, and it releases a large dataset of visual, grasp, and trajectory affordances. The architectural combination of a VLM with specialized decoders and the object-centric formulation are original and plausible. The paper also provides an extensive experimental suite, including real-robot trials and an RL-guidance ablation. The main value is in showing a concrete pathway from internet-scale human video to robot manipulation without robot-specific action labels.

major comments (4)
  1. [V.A.c and Table II] The visual affordance test set is evaluated on 500 randomly sampled scenes from each egocentric video dataset, and the ground-truth labels are generated by the same automatic pipeline (Section III.B) used to produce the training data, with no stated episode-level train/test split and no human validation. Under these conditions, the reported IoU, NSS, SIM, and KLD values measure agreement with the pseudo-labeler, so the 'state-of-the-art' visual affordance claim in the abstract and Section V.B.1 is not established. The authors should provide a disjoint split (e.g., held-out episodes or objects) and, ideally, a human-annotated or otherwise independent validation set for the contact-point and trajectory labels.
  2. [V.B.2 and Table III] The real-robot comparison is underpowered: 10 trials per task for 5 tasks gives 50 attempts per method, so the 68% versus 52% gap (34/50 vs 26/50) is within binomial sampling uncertainty (the standard error of a 68% rate over 50 trials is about 6.6 percentage points, and the difference is about 1.6 standard errors with no correction for multiple tasks). Additionally, baseline grasps come from GraspNet while VLAff uses hand retargeting, so the comparison conflates grasp-prediction quality with affordance-prediction quality. Please report per-task confidence intervals or significance tests, and use a shared grasp prediction strategy (or add an ablation that isolates the grasp source) to support the central claim that joint affordance learning beats isolated baselines.
  3. [V.A.d and V.B] Section V.A.d promises an evaluation of 'trajectory generation effectiveness,' but no such quantitative evaluation is reported anywhere in Section V.B. The only trajectory-related results are indirect, through zero-shot manipulation success, and the model's trajectory tokens are never directly measured (e.g., waypoint error, percentage of physically plausible trajectories, or comparison against other trajectory predictors). Since trajectory prediction is one of the three core affordance outputs and is explicitly named in the abstract, this missing evaluation is load-bearing for the 'unified actionable affordances' claim. Add a dedicated trajectory evaluation or clearly state that trajectory quality is only assessed through the downstream robot tasks.
  4. [V.A.b and V.A.c] The description of the visual affordance test set does not specify whether the 500 randomly sampled scenes come from the same episodes that were used to build EgoAffordance (Section V.A.b) and whether the additional HANDAL and SceneFun3D data are included in the test sampling. If scenes are sampled at the frame level without an episode-level split, the test set may overlap with the training data, further weakening the generalization claim. Please specify the exact split procedure (episode-level disjoint sets, object categories, or source datasets) and report results separately per data source.
minor comments (4)
  1. [V.A.c] The methods listed as 'segmentation-based' include LISA, which is a large-language-model-based reasoning segmentation model rather than a classic segmentation method; the normalization of its output to [0,1] should be described in more detail for reproducibility.
  2. [IV.C, Eq. (6)] The trajectory sampling strategy in Eq. (6) samples the continuous coordinate p1 but does not specify how this continuous value is mapped to the nearest trajectory token bin used by the autoregressive model; please clarify the quantization mapping.
  3. [V.B.2, Figure 5] Figure 5 shows learning curves for the Open Fridge RL task without error bars or standard deviations; it is unclear whether each configuration was run with multiple seeds, which is important for the claim that removing the visual heatmap leads to complete failure.
  4. [Table I] The 'Frame' column in Table I mixes raw frames for some datasets and interaction-sampled frames for EgoAffordance; this makes the comparison misleading and should be stated in the table caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: losses and predictions do not reduce to fitted inputs; the visual-affordance benchmark self-referentiality is a validity concern, not a circular step.

full rationale

The derivation chain is self-contained. EgoAffordance labels are produced by an external pipeline (hand-object detector, segmentation, 3D hand reconstruction, monocular depth, SfM, inpainting), and VLAff is trained with standard supervised losses: soft Dice for heatmaps, smooth L1 for MANO grasp parameters, and cross-entropy for quantized trajectory tokens. No loss term or evaluation metric is defined in terms of the model's own predictions, and no parameter is fitted to the test set. The visual-affordance evaluation does sample 500 scenes from the same egocentric video datasets that contribute to EgoAffordance and uses the same automatic label-generation pipeline, so Table II may partly measure agreement with the pseudo-labeler rather than independent ground truth; this is a benchmark-validity risk, especially without a stated episode-level train/test split or human validation, but it is not circularity in the derivation. The model is not defined in terms of the test labels, and the claimed zero-shot manipulation results use external real-robot and simulation evaluations. References to LISA, DINOv2, and other prior methods are genuinely external prior work; no load-bearing claim rests on a self-citation. Therefore no circular step can be exhibited from the paper's equations or construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a chain of unverified automatic annotation steps rather than on a small set of mathematical axioms. The largest burden is the assumption that pseudo-labels generated from human video, including contact points, MANO grasps, and SfM trajectories, are accurate enough to train and evaluate a transferable affordance model. No new physical entities are introduced.

free parameters (4)
  • Loss weights lambda_v, lambda_g, lambda_t = not reported
    Equation 5; chosen by hand to balance visual, grasp, and trajectory losses. No ablation or reported values, and they directly shape the learned trade-offs.
  • Soft Dice loss hyperparameters p and s = p=1.5, s=1.0
    Equation 2; manual loss-shaping choices not tuned or justified.
  • Trajectory quantization bins and vocabulary size N = not reported
    Section IV; spatial binning of 6D poses into discrete tokens. The bin resolution is a free design choice that affects trajectory fidelity.
  • Direction guidance sampling scale s and variance sigma = not reported
    Equation 6; the initial trajectory token is sampled around p_contact + s times the VLM-provided direction. These hand-picked values control how strongly the external GPT guess biases inference.
assumptions (5)
  • domain assumption Contact regions defined by fingertip-object mask intersection identify the true interaction point.
    Section III.B.b; this pseudo-label is used both as a training target and as evaluation ground truth, with no human validation.
  • domain assumption Monocular depth plus COLMAP and DROID-SLAM on inpainted egocentric video produces metric 3D trajectories.
    Section III.B.c; no validation against ground-truth 3D trajectories or LiDAR is provided.
  • domain assumption Projecting the peak heatmap point to 3D and re-anchoring grasp and trajectory makes affordances embodiment-agnostic.
    Section III.A; transferability across embodiments is asserted, with only limited real-robot success as evidence.
  • domain assumption The automatic pipeline's labels are a valid benchmark for comparing methods.
    Section V.B.1; test scenes are drawn from the same egocentric datasets used for training and labeled by the same extraction pipeline, which can favor models trained on the same label distribution.
  • domain assumption External foundation models (Qwen2.5-VL, DINOv2, SAM2, GPT, MANO/WiLoR) behave reliably in this pipeline.
    Sections IV and V; outputs of several pretrained models are chained without independent verification at each stage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VLAff: Vision-Language-Affordance Model for Unified Actionable Affordances." pith.science (2026). https://pith.science/paper/IMOLGPBX

@misc{pith2026260805215,
  author       = {Pith},
  title        = {Pith review of: VLAff: Vision-Language-Affordance Model for Unified Actionable Affordances},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IMOLGPBX}},
  note         = {Machine review of arXiv:2608.05215}
}
read the original abstract

Learning manipulation skills from human videos is promising for scalable robot learning. However, the embodiment mismatch between humans and robots makes this challenging. One promising solution is to learn object-centric actionable affordances that are embodiment-agnostic. In this work, we propose a framework that leverages egocentric human videos with state-of-the-art 3D Structure-from-Motion and hand mesh reconstruction to extract actionable affordances such as visual, grasp, and trajectory affordances that explicitly encode where to interact, how to grasp, and how to move. We construct EgoAffordance, a large-scale dataset comprising 204K episodes with 5.6M visual affordances and 11.6M grasp and trajectory affordances. Building on this, we introduce VLAff, a large vision-language model-based unified foundation model that learns cross-modal correlations across all actionable affordances. Given a visual observation and instruction, VLAff generates visual affordance heatmaps, grasp poses, and trajectories, which are then converted into directly executable actions by utilizing 3D scene information. Through extensive experiments, we demonstrate that VLAff not only achieves state-of-the-art performance on visual affordance prediction, but can also be effectively applied to real robot applications such as zero-shot manipulation and affordance-guided robot learning.

Figures

Figures reproduced from arXiv: 2608.05215 by the authors.

Figure 1
Figure 1. We present VLAff, a Vision-Language-Affordance model that learns actionable affordances such as visual, grasp, trajectory from large-scale egocentric human videos to enable robot manipulation across diverse tasks. TABLE I COMPARISON OF AFFORDANCE-RELATED DATASETS. OUR EGOAFFORDANCE DATASET UNIQUELY PROVIDES COMPREHENSIVE ACTIONABLE AFFORDANCES INCLUDING VISUAL AFFORDANCE MASKS, HAND POSE ANNOTATIONS, AND CAMERA TRAJ… view at source ↗
Figure 2
Figure 2. VLAff model architecture showing the integration of vision encoder, VLM, and specialized decoders for visual, grasp, and trajectory affordance [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visual affordance prediction results in the wild. We show visual affordance predictions from VLAff and baseline methods on diverse manipulation [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Robot manipulation experiments. We visualize the generated [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Affordance-guided manipulation learning results on real-world [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 42 canonical work pages

  1. [1]

    Affordance detection of tool parts from geometric features,

    A. Myerset al., “Affordance detection of tool parts from geometric features,” inIEEE International Conference on Robotics and Automa- tion (ICRA), 2015, pp. 1374–1381

    work page 2015

  2. [2]

    Learning affordance grounding from exocentric im- ages,

    H. Luoet al., “Learning affordance grounding from exocentric im- ages,” inCVPR, 2022, pp. 2252–2261

    work page 2022

  3. [3]

    Object-based affordances detection with convo- lutional neural networks and dense conditional random fields,

    A. Nguyenet al., “Object-based affordances detection with convo- lutional neural networks and dense conditional random fields,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 5908–5915

    work page 2017

  4. [4]

    Learning to act properly: Predicting and explain- ing affordances from images,

    C.-Y . Chuanget al., “Learning to act properly: Predicting and explain- ing affordances from images,” inCVPR, 2018, pp. 975–983

    work page 2018

  5. [5]

    Glover++: Unleashing the potential of affordance learn- ing from human behaviors for robotic manipulation,

    T. Maet al., “Glover++: Unleashing the potential of affordance learn- ing from human behaviors for robotic manipulation,”arXiv preprint arXiv:2505.11865, 2025

    arXiv 2025

  6. [6]

    H2o: Two hands manipulating objects for first person interaction recognition,

    T. Kwonet al., “H2o: Two hands manipulating objects for first person interaction recognition,” inICCV, 2021, pp. 10 138–10 148

    work page 2021

  7. [7]

    Hoi4d: A 4d egocentric dataset for category-level human-object interaction,

    Y . Liuet al., “Hoi4d: A 4d egocentric dataset for category-level human-object interaction,” inCVPR, 2022, pp. 21 013–21 022

    work page 2022

  8. [8]

    Scaling egocentric vision: The epic-kitchens dataset,

    D. Damenet al., “Scaling egocentric vision: The epic-kitchens dataset,” inECCV, 2018, pp. 720–736

    work page 2018

Show all 70 references
  1. [9]

    Hd-epic: A highly-detailed egocentric video dataset,

    H. Doughtyet al., “Hd-epic: A highly-detailed egocentric video dataset,”arXiv preprint arXiv:2502.04144, 2025

    2025 arXiv

  2. [10]

    Ego4d: Around the world in 3,000 hours of egocentric video,

    K. Graumanet al., “Ego4d: Around the world in 3,000 hours of egocentric video,” inCVPR, 2022, pp. 18 995–19 012

    2022

  3. [11]

    Ego-exo4d: Understanding skilled human ac- tivity from first- and third-person perspectives,

    K. Graumanet al., “Ego-exo4d: Understanding skilled human ac- tivity from first- and third-person perspectives,”arXiv preprint arXiv:2311.18259, 2023

    2023 arXiv

  4. [12]

    Llama: Open and efficient foundation language models,

    H. Touvronet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  5. [13]

    Gpt-4 technical report,

    OpenAIet al., “Gpt-4 technical report,” 2024. [Online]. Available: https://arxiv.org/abs/2303.08774

    2024 arXiv

  6. [14]

    Efficient training of artificial neural networks for autonomous navigation,

    D. A. Pomerleau, “Efficient training of artificial neural networks for autonomous navigation,”Neural Computation, vol. 3, no. 1, pp. 88–97, 1991

    1991

  7. [15]

    Vip: Towards universal visual reward and representa- tion via value-implicit pre-training,

    Y . J. Maet al., “Vip: Towards universal visual reward and representa- tion via value-implicit pre-training,”arXiv preprint arXiv:2210.00030, 2023

    2023 arXiv

  8. [16]

    Masked visual pre-training for motor control,

    T. Xiaoet al., “Masked visual pre-training for motor control,”arXiv preprint arXiv:2203.06173, 2022

    2022 arXiv

  9. [17]

    R3m: A universal visual representation for robot manipulation,

    S. Nairet al., “R3m: A universal visual representation for robot manipulation,”arXiv preprint arXiv:2203.12601, 2022

    2022 arXiv

  10. [18]

    J. J. Gibson,The Ecological Approach to Visual Perception. Houghton Mifflin, 1979

    1979

  11. [19]

    Affordances from human videos as a versatile repre- sentation for robotics,

    S. Bahlet al., “Affordances from human videos as a versatile repre- sentation for robotics,” inCVPR, 2023, pp. 13 778–13 790

    2023

  12. [20]

    Glover: Generalizable open-vocabulary affordance rea- soning for task-oriented grasping,

    T. Maet al., “Glover: Generalizable open-vocabulary affordance rea- soning for task-oriented grasping,”arXiv preprint arXiv:2411.12286, 2024

    2024 arXiv

  13. [21]

    Dexycb: A benchmark for capturing hand grasping of objects,

    Y .-W. Chaoet al., “Dexycb: A benchmark for capturing hand grasping of objects,” inCVPR, 2021, pp. 9044–9053

    2021

  14. [22]

    Videodex: Learning dexterity from internet videos,

    K. Shawet al., “Videodex: Learning dexterity from internet videos,” inConference on Robot Learning, 2023, pp. 654–665

    2023

  15. [23]

    Where are we in the search for an artificial visual cortex for embodied intelligence?

    A. Majumdaret al., “Where are we in the search for an artificial visual cortex for embodied intelligence?”arXiv preprint arXiv:2303.18240, 2023

    2023 arXiv

  16. [24]

    Liv: Language-image representations and rewards for robotic control,

    Y . J. Maet al., “Liv: Language-image representations and rewards for robotic control,”arXiv preprint arXiv:2306.00958, 2023

    2023 arXiv

  17. [25]

    Hrp: Human affordances for robotic pre- training,

    M. K. Sriramaet al., “Hrp: Human affordances for robotic pre- training,” inProceedings of Robotics: Science and Systems (RSS), Delft, Netherlands, 2024

    2024

  18. [26]

    Vidbot: Learning generalizable 3d actions from in- the-wild 2d human videos for zero-shot robotic manipulation,

    Z. Hanet al., “Vidbot: Learning generalizable 3d actions from in- the-wild 2d human videos for zero-shot robotic manipulation,”arXiv preprint arXiv:2503.07135, 2025

    2025 arXiv

  19. [27]

    Weakly supervised affordance detection,

    J. Sawatzkyet al., “Weakly supervised affordance detection,” inCVPR, 2017, pp. 2795–2804

    2017

  20. [28]

    Locate: Localize and transfer object parts for weakly supervised affordance grounding,

    G. Liet al., “Locate: Localize and transfer object parts for weakly supervised affordance grounding,” inCVPR, 2023, pp. 10 922–10 931

    2023

  21. [29]

    Affordancellm: Grounding affordance from vision language models,

    S. Qianet al., “Affordancellm: Grounding affordance from vision language models,” inCVPR, 2024, pp. 7587–7597

    2024

  22. [30]

    Uad: Unsupervised affordance distillation for gener- alization in robotic manipulation,

    Y . Tanget al., “Uad: Unsupervised affordance distillation for gener- alization in robotic manipulation,”arXiv preprint arXiv:2506.09284, 2025

    2025 arXiv

  23. [31]

    Arctic: A dataset for dexterous bimanual hand-object manipulation,

    Z. Fanet al., “Arctic: A dataset for dexterous bimanual hand-object manipulation,” inCVPR, 2023, pp. 927–938

    2023

  24. [32]

    Contactpose: A dataset of grasps with object contact and hand pose,

    S. Brahmbhattet al., “Contactpose: A dataset of grasps with object contact and hand pose,” inECCV, 2020

    2020

  25. [33]

    Oakink: A large-scale knowledge repository for understanding hand-object interaction,

    L. Yanget al., “Oakink: A large-scale knowledge repository for understanding hand-object interaction,” inCVPR, 2022

    2022

  26. [34]

    Taco: Benchmarking generalizable bimanual tool-action- object understanding,

    Y . Liet al., “Taco: Benchmarking generalizable bimanual tool-action- object understanding,”arXiv preprint arXiv:2401.08399, 2024

    2024 arXiv

  27. [35]

    Rt-1: Robotics transformer for real-world control at scale,

    A. Brohanet al., “Rt-1: Robotics transformer for real-world control at scale,”arXiv preprint arXiv:2212.06817, 2022

    2022 arXiv

  28. [36]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control,

    A. Brohanet al., “Rt-2: Vision-language-action models transfer web knowledge to robotic control,”arXiv preprint arXiv:2307.15818, 2023

    2023 arXiv

  29. [37]

    Openvla: An open-source vision-language-action model,

    M. J. Kimet al., “Openvla: An open-source vision-language-action model,”arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  30. [38]

    Octo: An open-source generalist robot policy,

    O. M. Teamet al., “Octo: An open-source generalist robot policy,” arXiv preprint arXiv:2405.12213, 2024

    2024 arXiv

  31. [39]

    Open x-embodiment: Robotic learning datasets and rt-x models,

    A. Padalkaret al., “Open x-embodiment: Robotic learning datasets and rt-x models,”arXiv preprint arXiv:2310.08864, 2023

    2023 arXiv

  32. [40]

    Droid: A large-scale in-the-wild robot manipu- lation dataset,

    A. Khazatskyet al., “Droid: A large-scale in-the-wild robot manipu- lation dataset,” inRobotics: Science and Systems, 2024

    2024

  33. [41]

    Embodied hands: Modeling and capturing hands and bodies together,

    J. Romeroet al., “Embodied hands: Modeling and capturing hands and bodies together,” inACM TOG, vol. 36, no. 6, 2017, pp. 1–17

    2017

  34. [42]

    On the continuity of rotation representations in neural networks,

    Y . Zhouet al., “On the continuity of rotation representations in neural networks,” inCVPR, 2019, pp. 5745–5753

    2019

  35. [43]

    Understanding human hands in contact at internet scale,

    D. Shanet al., “Understanding human hands in contact at internet scale,” inCVPR, 2020, pp. 9869–9878

    2020

  36. [44]

    Sam 2: Segment anything in images and videos,

    N. Raviet al., “Sam 2: Segment anything in images and videos,”arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  37. [45]

    Vitpose: Simple vision transformer baselines for human pose estimation,

    Y . Xuet al., “Vitpose: Simple vision transformer baselines for human pose estimation,” inNeurIPS, 2022, pp. 38 571–38 584

    2022

  38. [46]

    Cotracker3: Simpler and better point tracking by pseudo-labelling real videos,

    N. Karaevet al., “Cotracker3: Simpler and better point tracking by pseudo-labelling real videos,”arXiv preprint arXiv:2410.11831, 2024

    2024 arXiv

  39. [47]

    Wilor: End-to-end 3d hand localization and reconstruction in-the-wild,

    R. A. Li and D. Shan, “Wilor: End-to-end 3d hand localization and reconstruction in-the-wild,”arXiv preprint arXiv:2407.10034, 2024

    2024 arXiv

  40. [48]

    Egohos: Dataset and method for hand and object segmentation in egocentric videos,

    B. Zhanget al., “Egohos: Dataset and method for hand and object segmentation in egocentric videos,” inCVPR, 2022, pp. 21 381– 21 391

    2022

  41. [49]

    Propainter: Improving propagation and transformer for video inpainting,

    S. Zhouet al., “Propainter: Improving propagation and transformer for video inpainting,”arXiv preprint arXiv:2309.03897, 2023

    2023 arXiv

  42. [50]

    Moge-2: Accurate monocular geometry with metric scale and sharp details,

    R. Wanget al., “Moge-2: Accurate monocular geometry with metric scale and sharp details,”arXiv preprint arXiv:2507.02546, 2025

    2025 arXiv

  43. [51]

    Structure-from-motion revisited,

    J. L. Sch ¨onberger and J.-M. Frahm, “Structure-from-motion revisited,” inIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4104–4113

    2016

  44. [52]

    Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras,

    Z. Teed and J. Deng, “Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras,” inNeurIPS, 2021, pp. 16 558–16 569

    2021

  45. [53]

    Tapip3d: Tracking any point in persistent 3d geometry,

    A. W. Harleyet al., “Tapip3d: Tracking any point in persistent 3d geometry,”arXiv preprint arXiv:2312.03904, 2024

    2024 arXiv

  46. [54]

    Lisa: Reasoning segmentation via large language model,

    X. Laiet al., “Lisa: Reasoning segmentation via large language model,” inCVPR, 2024, pp. 9579–9589

    2024

  47. [55]

    Dinov2: Learning robust visual features without supervision,

    M. Oquabet al., “Dinov2: Learning robust visual features without supervision,”arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  48. [56]

    Qwen2.5-vl technical report,

    S. Baiet al., “Qwen2.5-vl technical report,”arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  49. [57]

    Handal: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions,

    A. Guoet al., “Handal: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023, pp. 11 428–11 435

    2023

  50. [58]

    Scenefun3d: Fine-grained functionality and affor- dance understanding in 3d scenes,

    A. Delitzaset al., “Scenefun3d: Fine-grained functionality and affor- dance understanding in 3d scenes,” inCVPR, 2024

    2024

  51. [59]

    Understanding 3d object interaction from a single image,

    S. Qian and D. F. Fouhey, “Understanding 3d object interaction from a single image,” inICCV, 2023, pp. 21 753–21 763

    2023

  52. [60]

    Ram: Retrieval-based affordance transfer for generalizable zero-shot robotic manipulation,

    Y . Kuanget al., “Ram: Retrieval-based affordance transfer for generalizable zero-shot robotic manipulation,”arXiv preprint arXiv:2407.04689, 2024

    2024 arXiv

  53. [61]

    Generalflow: Generalizable manipulation policy with flow matching,

    Z. Yuanet al., “Generalflow: Generalizable manipulation policy with flow matching,”arXiv preprint arXiv:2410.10649, 2024

    2024

  54. [62]

    Graspnet-1billion: A large-scale benchmark for general object grasping,

    H.-S. Fanget al., “Graspnet-1billion: A large-scale benchmark for general object grasping,” inCVPR, 2020

    2020

  55. [63]

    R+x: Retrieval and execution from everyday human videos,

    G. Papagianniset al., “R+x: Retrieval and execution from everyday human videos,” inIEEE International Conference on Robotics and Automation (ICRA), 2025

    2025

  56. [64]

    Isaac gym: High performance gpu- based physics simulation for robot learning,

    V . Makoviychuket al., “Isaac gym: High performance gpu- based physics simulation for robot learning,”arXiv preprint arXiv:2108.10470, 2021

    2021 arXiv

  57. [65]

    Partmanip: Learning cross-category generalizable part manipulation policy from point cloud observations,

    H. Genget al., “Partmanip: Learning cross-category generalizable part manipulation policy from point cloud observations,” inCVPR, 2023

    2023

  58. [66]

    Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning,

    A. Guptaet al., “Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning,” inConference on Robot Learning (CoRL), 2019

    2019

  59. [67]

    Maniskill2: A unified benchmark for generalizable manipulation skills,

    J. Guet al., “Maniskill2: A unified benchmark for generalizable manipulation skills,” inICLR, 2023

    2023

  60. [68]

    Ag2manip: Learning novel manipulation skills with agent-agnostic visual and action representations,

    H. Genget al., “Ag2manip: Learning novel manipulation skills with agent-agnostic visual and action representations,” inIEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), 2024

    2024

  61. [69]

    Pointllm: Empowering large language models to understand point clouds,

    R. Xuet al., “Pointllm: Empowering large language models to understand point clouds,” inECCV, 2024

    2024

  62. [70]

    Generating 6dof object manipulation trajectories from action description in egocentric vision,

    T. Yoshidaet al., “Generating 6dof object manipulation trajectories from action description in egocentric vision,” inCVPR, 2025

    2025

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.