Pith. sign in

REVIEW 3 major objections 7 minor 2 cited by

HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos

T0 review · 3 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HOT3D introduces a public egocentric multi-view dataset with motion-capture ground truth for hands and objects, and shows that multi-view methods beat single-view baselines on hand tracking, object pose, and 3D lifting.

desk verdict A genuinely novel, large-scale egocentric multi-view hand-object dataset with mocap ground truth; the experiments are suggestive but the annotation accuracy is under-quantified. read the letter →

arxiv 2411.19167 v2 pith:VC4XHWIO submitted 2024-11-28 cs.CV cs.AIcs.RO

Prithviraj Banerjee , Sindi Shkodrani , Pierre Moulon , Shreyas Hampali , Shangchen Han , Fan Zhang , Linguang Zhang , Jade Fountain
show 6 more authors
Edward Miller Selen Basol Richard Newcombe Robert Wang Jakob Julian Engel Tomas Hodan
This is my paper · ORCID
classification cs.CVcs.AIcs.RO
keywords egocentricvisionhandtracking6DoFobjectposemulti-viewdatasethand-objectinteraction3Dliftingmotioncapture
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HOT3D is a large-scale public dataset for egocentric 3D hand and object tracking, built from hardware-synchronized multi-view video recorded with two real head-mounted devices. The paper's central claim is that this combination of real headsets, multiple simultaneous views, and motion-capture ground truth is new at this scale, and that it matters because multi-view methods for hand tracking, 6DoF object pose estimation, and 3D lifting of unknown in-hand objects clearly outperform single-view baselines. If true, the dataset supplies what AR/VR and contextual-AI systems need: a way to train and benchmark perception that exploits the multiple cameras already on headsets, without relying on power-hungry depth sensors.

What carries the argument

The load-bearing object is the dataset's capture protocol: two head-mounted devices record several cameras that are triggered by a hardware timecode to produce synchronized views of the same hand-object scene, while a rig of infrared optical-marker cameras tracks small reflective markers glued to the hands and objects. Marker trajectories are fit to scanned 3D models of each hand and object to produce per-frame ground-truth poses. The claim-carrying mechanism is the controlled comparison: the paper deliberately runs single-view and multi-view versions of the same method on the same frames, so the only changed variable is the number of synchronized views. The multi-view baselines are simple, involving two-view training with view masking for the hand tracker, generalized PnP over correspondences from all views for pose estimation, and row-wise nearest-neighbor matching of self-supervised features across a stereo pair for 3D lifting.

What would settle it

Take a held-out subset of the released training frames, re-run the marker-to-model fitting with an independent procedure, and compare the result with the published ground truth; if the disagreement approaches the 5 cm or 5 degree thresholds used in the recall metrics on a substantial fraction of frames, the supervision quality and the resulting multi-view gains would need to be re-derived. A second check is to train the same single-view baselines on HOT3D training splits and submit to the official test server: if a single-view model matches the reported multi-view recall, the claimed multi-view advantage collapses.

Watch

Extended reading notes

Core claim

The discovery is the dataset itself and the empirical pattern it exposes. HOT3D offers over 833 minutes of egocentric multi-view video, more than 3.7 million images from 19 subjects interacting with 33 rigid objects, with per-frame ground-truth poses of both hands and objects obtained by an optical-marker motion-capture system, plus 3D object meshes, hand models in two formats, and, for one device, SLAM point clouds and eye gaze. The paper runs three controlled comparisons: a hand tracker trained and tested in single-view versus two-view mode, a pose estimator extended from a single RGB image to multiple synchronized views, and a stereo-match method for finding the 3D location of a handheld object. In all three, the multi-view variant outperforms its single-view counterpart, with lower mean keypoint error, 8 to 12 percentage points higher object-pose recall, and a jump in 5 cm 3D-lifting recall from 14.3 percent to 64.4 percent when ground-truth masks are used.

Load-bearing premise

The load-bearing premise is that the optical-marker motion-capture system yields ground-truth poses accurate enough to train and evaluate on; the paper filters out about a quarter of frames as missing or low quality, but reports no quantitative error of the mocap pipeline.

Editorial extensions

If this is right

  • HOT3D gives the community a public test bed where multi-view egocentric methods can be trained and compared on real-device data, not just synthetic or exocentric captures.
  • The two-view consumer-headset configuration being sufficient for large gains means hardware that already ships to consumers can support improved tracking.
  • The curated clips and the validity mask let researchers use the unannotated frames for self-supervised pretraining while benchmarking on the 1.16 million validated frames.
  • The multi-view baseline results define a realistic first benchmark for the three tasks, so subsequent methods can be measured against numbers that already show the single-view ceiling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same multi-view advantage likely transfers to other egocentric tasks the paper does not evaluate, such as hand-object contact estimation or action recognition, because extra views reduce occlusion and provide metric depth cues.
  • Because the glasses-prototype recordings include eye gaze, one untested extension is gaze-conditioned object lifting, using the gaze ray as an additional correspondence prior to sharpen localization in the coarse 10 cm regime.
  • The roughly 23 percent of frames released without valid annotations could be exploited by self-supervised multi-view consistency losses; measuring how much of the reported gap closes with unlabeled frames would isolate the value of the labeled ground truth.
  • A direct comparison of the marker-based ground truth against an independent RGB-D optimization on the same scenes would put a number on the annotation quality, which the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper introduces HOT3D, a public dataset for egocentric 3D hand and object tracking, containing 833 minutes / 1.5M multi-view frames (3.7M+ images) from Project Aria and Quest 3, with 19 subjects and 33 rigid objects. Ground-truth 6DoF poses of hands (in UmeTrack and MANO formats) and objects are obtained from an optical-marker OptiTrack mocap system; object models come from an in-house scanner with PBR materials. The paper presents three experimental comparisons of multi-view vs single-view baselines: UmeTrack-based 3D hand tracking (Table 2), a multi-view extension of FoundPose for 6DoF object pose estimation (Table 3), and a DINOv2 stereo-matching method for 3D lifting of unknown in-hand objects (Table 5), all reporting that multi-view input substantially improves accuracy over single-view. The dataset also provides validity masks, training/test splits with hidden test annotations, curated clips, and object onboarding sequences.

Significance. If the claims hold, HOT3D fills a clear gap: it is the first large-scale dataset combining multi-view, hardware-time-synchronized egocentric images from real headsets with marker-based mocap ground truth for hands and objects, and it enables benchmarking of multi-view methods on realistic AR/VR hardware. The paper is transparent in several respects: it releases all 1.5M frames, provides validity masks from visual inspection, defines train/test subject splits, uses a public test server for test annotations, and includes object meshes with PBR materials. The proposed baselines, especially the multi-view FoundPose extension and StereoMatch, provide useful starting points. However, the core premise of 'high-quality ground-truth annotations' is not quantitatively validated, and the significance of the multi-view gains is not statistically substantiated; these issues must be addressed before the dataset's value can be fully assessed.

major comments (3)
  1. [Sec. 3 and Appendix C] The paper repeatedly characterizes the ground-truth annotations as 'high-quality' (Abstract, Sec. 1, Table 1 caption), but no quantitative accuracy measure of the marker-based mocap pipeline is reported anywhere. The only quality control is manual visual inspection (Sec. 3), which flags gross misalignments but cannot measure sub-centimeter pose errors. Appendix C describes 3 mm optical markers, ~19 markers per hand and ~10 per object, semi-automatic registration, and model fitting, yet gives no reprojection errors, marker-trajectory residuals, or comparison with an independent ground truth. Because the same poses are used to train and evaluate all baselines (Tables 2-5) and to define the in-hand masks and 3D locations (Sec. 4.3-4.4), any systematic GT error is inherited by every reported number. I request a quantitative validation, e.g., per-hand and per-object model-fit residuals, marker reconstruction errors, temporal consistency statistics, and a spot check against manually annotated keypoints or depth-based ICP on a subset of frames, along with the distribution of the visual-inspection rejections.
  2. [Tables 2-5, Sec. 4] All experimental results are reported as point estimates without error bars, confidence intervals, or per-sequence/per-subject variation. The abstract and conclusion state that multi-view methods 'significantly outperform' single-view methods; this claim is not statistically supported. The frames in a clip are temporally highly correlated, so the effective sample size is much smaller than the number of frames, and a difference of 8-12 percentage points (Table 3) may not be significant if only a few subjects or sequences drive the effect. Please provide variance estimates (e.g., bootstrap confidence intervals over clips or subjects) for the key comparisons in Tables 2, 3, and 5, and describe how the authors accessed the hidden test annotations (public evaluation server or direct labels), since this affects the reproducibility of the reported numbers.
  3. [Sec. 4.3, Tables 4-5] The evaluation protocol for the in-hand object tasks relies on threshold definitions stated without justification: an object is 'in-hand' if the minimum distance between object and hand mesh vertices is below 1 cm and the object moves faster than 1 cm/s (Sec. 4.3). The reported mIoU (Table 4) and recall rates (Table 5) are conditional on these thresholds, and the multi-view advantage in Table 5 is computed on this filtered subset. No sensitivity analysis is provided, so it is unclear whether the magnitude of the multi-view gains, or even their sign, would persist under reasonable alternative thresholds (e.g., 0.5 cm or 2 cm, or velocity thresholds of 0.5/2 cm/s). I request a sensitivity analysis for the main results in Table 5, or explicit evidence that the conclusions are robust to these choices.
minor comments (7)
  1. [Title] The title contains 'T racking' with an erroneous space; it should be 'Tracking'.
  2. [Sec. 3] The sentence 'manually flagging frames were rendering of hand and object models in the ground-truth poses is not closely aligned with the observed image' is ungrammatical; it should read 'manually flagging frames where the rendering of hand and object models in the ground-truth poses is not closely aligned with the observed image'.
  3. [Sec. 4.1] The '41% improvement' is a relative reduction in MKPE (13.4/9.5 = 15.4/10.9 = 1.41); please state this explicitly to avoid reader confusion with absolute differences.
  4. [Sec. 4.4] The 'robust mean of the 3D point set' in StereoMatch is undefined; specify the estimator (e.g., trimmed mean, median, or RANSAC-based consensus) and its parameters.
  5. [Sec. 4.2-4.4] The code for the proposed multi-view extensions (FoundPose-mv and StereoMatch) is not linked; providing executables or detailed hyperparameters would improve reproducibility of the baseline results.
  6. [Table 5] The HandProxy row shows a dash in the 'Views' column; clarify whether this baseline is evaluated in a single-view or multi-view manner.
  7. [Abstract and Sec. 3] The phrasing 'over 3.7M+ images' in the abstract and '3.7M+ images' in Sec. 3 is inconsistent; unify the notation.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: dataset paper with independent held-out evaluations; self-citations are not load-bearing.

full rationale

HOT3D is an empirical dataset paper rather than a derivation. The load-bearing claim that multi-view methods outperform single-view counterparts is supported by evaluations on held-out test clips (Sec. 3: 'Ground-truth annotations for the test split are accessible only by dedicated evaluation servers'), with metrics computed against motion-capture-derived poses that are not produced by the methods being evaluated. The multi-view baselines are not equivalent to their inputs by construction: the FoundPose extension solves generalized PnP from correspondences across all views (Sec. 4.2), and StereoMatch triangulates DINOv2 stereo correspondences (Sec. 4.4); neither reduces to a fitted parameter. The UmeTrack hand-tracking comparison uses the same trained model in single- and two-view modes evaluated on UmeTrack and HOT3D test frames (Sec. 4.1, Table 2). Author-overlap citations exist (UmeTrack [26], optical-marker hand fitting [25], FoundPose [50]), but they are used as published baselines or annotation components, not as assumed conclusions or uniqueness theorems. The main caveat is data quality: the motion-capture pipeline is described without a quantitative accuracy report (Appendix C; Sec. 3 notes 1.16M of 1.5M frames passed manual visual inspection). That is a correctness/quality risk that would affect all numbers, but it is not circularity because the ground truth is independent of the predictor outputs and the single-view vs multi-view comparisons inherit the same ground truth.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on data-collection infrastructure rather than on axioms in a mathematical sense. The most consequential assumptions are the accuracy of the optical-marker mocap ground truth, the reliability of the manual validity mask, and the correctness of Aria's SLAM and calibration; none of these is independently quantified in the paper.

free parameters (3)
  • In-hand object definition thresholds = 1 cm mesh-to-mesh distance and 1 cm/s velocity
    Sec. 4.3 defines which objects are 'in hand' for the segmentation and 3D-lifting ground truth; changing these thresholds would change the evaluated frames and reported mIoU or recall.
  • StereoMatch correspondence hyperparameters = 420x420 crop resolution, up to 500 correspondences
    Sec. 4.4 uses these values for the DINOv2 stereo-matching baseline; they are chosen by hand and are not swept, and they affect 3D-lifting recall.
  • Correctness thresholds for pose and location recall = 5/10/20 cm and 5/10/20 degrees
    Tables 3 and 5 define 'correct' poses and locations with these user-chosen tolerances; the qualitative claim that multi-view outperforms single-view holds at all thresholds, but the magnitude depends on them.
assumptions (4)
  • domain assumption Optical marker mocap yields ground-truth poses accurate enough for training and evaluation, and 3 mm markers do not alter natural hand-object interaction.
    Appendix C and Sec. 3; no quantitative accuracy or error analysis of the OptiTrack-based ground truth is reported.
  • domain assumption Manual visual inspection correctly separates valid from low-quality annotated frames, and the 1.16M-frame validity mask does not bias results.
    Sec. 3 states 1.16M of 1.5M frames passed visual inspection; the flagging criterion is manual and not measured.
  • domain assumption Aria MPS SLAM and the 7DoF alignment to the OptiTrack frame yield metric, gravity-aligned, millimeter-accurate camera poses.
    Appendix A.2; multi-view pose estimation and triangulation assume these camera-to-world transforms are correct.
  • domain assumption DINOv2 features are repeatable enough across synchronized egocentric views for template retrieval and stereo matching.
    Secs. 4.2 and 4.4 use DINOv2 as the backbone of two baselines; no separate feature-repeatability analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos." pith.science (2026). https://pith.science/paper/VC4XHWIO

@misc{pith2026241119167,
  author       = {Pith},
  title        = {Pith review of: HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VC4XHWIO}},
  note         = {Machine review of arXiv:2411.19167}
}
read the original abstract

We introduce HOT3D, a publicly available dataset for egocentric hand and object tracking in 3D. The dataset offers over 833 minutes (3.7M+ images) of recordings that feature 19 subjects interacting with 33 diverse rigid objects. In addition to simple pick-up, observe, and put-down actions, the subjects perform actions typical for a kitchen, office, and living room environment. The recordings include multiple synchronized data streams containing egocentric multi-view RGB/monochrome images, eye gaze signal, scene point clouds, and 3D poses of cameras, hands, and objects. The dataset is recorded with two headsets from Meta: Project Aria, which is a research prototype of AI glasses, and Quest 3, a virtual-reality headset that has shipped millions of units. Ground-truth poses were obtained by a motion-capture system using small optical markers attached to hands and objects. Hand annotations are provided in the UmeTrack and MANO formats, and objects are represented by 3D meshes with PBR materials obtained by an in-house scanner. In our experiments, we demonstrate the effectiveness of multi-view egocentric data for three popular tasks: 3D hand tracking, model-based 6DoF object pose estimation, and 3D lifting of unknown in-hand objects. The evaluated multi-view methods, whose benchmarking is uniquely enabled by HOT3D, significantly outperform their single-view counterparts.

Figures

Figures reproduced from arXiv: 2411.19167 by the authors.

Figure 1
Figure 1. HOT3D overview. The dataset includes multi-view egocentric image streams from Aria [13] and Quest 3 [41] annotated with high-quality ground-truth 3D poses and models of hands and objects. Three multi-view frames from Aria are shown on the left, with contours of 3D models of hands and objects in the ground-truth poses in white and green, respectively. Aria also provides 3D point clouds from SLAM and eye gaze informat… view at source ↗
Figure 2
Figure 2. Sample images from Aria (top) and Quest 3 (bottom). Aria recordings include one RGB and two monochrome image streams, while Quest 3 recordings include two monochrome streams (only images from one of the multi-view streams are shown). Contours of 3D models of hands and objects in the ground-truth poses are shown in white and green respectively. In addition to simple pick-up/observe/put-down actions, the subjects perf… view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Distances traveled by HOT3D objects. In total, subjects moved the 33 objects over 13 km. While objects like the keyboard and waffles were mostly resting, the white mug is a true explorer. dard). Annotations in some frames may be missing or be of a lower quality. Out of…
Figure 5
Figure 5. Figure 5: Motion-capture lab. The HOT3D dataset was collected using a motion-capture rig equipped with a few dozens of infrared exocentric OptiTrack cameras and light diffuser panels for illumination variability. from Aria and 1849 from Quest 3). Each clip has 150 frames (5 seco…
Figure 7
Figure 7. Figure 7: Example 6DoF pose estimation results by FoundPose [50]. Each row shows synchronized views of the same object from three Aria cameras. Our multi-view extension of FoundPose estimates the object pose from 2D-3D correspondences established between all available views and …
Figure 8
Figure 8. Figure 8: Example results of 2D segmentation of in-hand objects. Masks predicted by EgoHOS [75] (1st and 3rd row) are compared with masks predicted by our MRCNN-DA (2nd and 4th row). Predicted masks are shown in blue, and the contour of ground-truth masks in green. masks of an i…
Figure 9
Figure 9. Figure 9: Example results of 3D lifting of in-hand objects. Ground￾truth 3D object locations are shown in green, predictions from Stere￾oMatch in red, MonoDepth in blue, and HandProxy in orange. The locations are projected to the three Aria views. In each example, the RGB image …
Figure 10
Figure 10. Figure 10: Project Aria research glasses [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 12
Figure 12. Figure 12: Aria MPS output. Shown is output for three recordings in a living room, office and kitchen scenario respectively (left to right). Top: RGB view and gaze (green dot). Middle: Point cloud and estimated egocentric camera trajectory for the full recording. Bottom: 3D view…
Figure 13
Figure 13. Figure 13: Meta Quest 3 headset for virtual and mixed reality [PITH_FULL_IMAGE:figures/full_fig_p013_13.png]
Figure 14
Figure 14. Figure 14: Sample images from Quest 3. Shown are synchronized im￾ages from the two front Quest 3 cameras used for the HOT3D collection [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]
Figure 15
Figure 15. Figure 15: Object orientation statistics. Top: 3D object models in their canonical poses. Bottom: Distribution of azimuth and elevation angles under which the objects are observed across the dataset. The vertical axis is the azimuth angle [0◦ , 360◦ ] (angle along the green axis…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CoMind: Understanding Collaborative Human Activity from Multiple Minds and Views

    cs.CV 2026-07 accept novelty 6.5 of 10

    CoMind releases 41 h of synchronized multi-view cooking collaboration with social-cue annotations and three ToM-oriented benchmarks on which current VLMs score poorly until fine-tuned.

  2. Open-AoE: An Open Egocentric Manipulation Dataset and Toolchain for Embodied Learning

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Open-AoE releases 2,000 hours of smartphone egocentric manipulation video with MANO hand poses, camera trajectories, atomic action labels, and training tools for VLA and world-model pipelines.

Reference graph

Works this paper leans on

76 extracted references · 70 canonical work pages · cited by 2 Pith papers

  1. [1]

    Crandall, and Chen Y u

    Sven Bambach, Stefan Lee, David J. Crandall, and Chen Y u. Lending a hand: Detecting hands and recognizing activities in complex egocentric interactions. InICCV, 2015. 3

    work page 2015

  2. [2]

    The precision of the human hand: variability in pinch strength and manual dexterity.Symmetry, 2022

    Ameline Bardo, Katie Town, Tracy L Kivell, Georgina Donati, Haiko Ballieux, Cosmin Stamate, Trudi Edginton, and Gillian S Forrester. The precision of the human hand: variability in pinch strength and manual dexterity.Symmetry, 2022. 1

    work page 2022

  3. [3]

    BEHA VE: Dataset and method for tracking human object interactions

    Bharat Lal Bhatnagar, Xianghui Xie, Ilya Petrov, Cristian Smin- chisescu, Christian Theobalt, and Gerard Pons-Moll. BEHA VE: Dataset and method for tracking human object interactions. In CVPR, 2022. 3

    work page 2022

  4. [4]

    Learning OpenCV: Computer vision with the OpenCV library.O’REILLY, 2008

    Gary Bradski. Learning OpenCV: Computer vision with the OpenCV library.O’REILLY, 2008. 8

    work page 2008

  5. [5]

    ContactPose: A dataset of grasps with object contact and hand pose

    Samarth Brahmbhatt, Catherine Ham, Charles C Kemp, and James Hays. ContactPose: A dataset of grasps with object contact and hand pose. InECCV, 2020. 3

    work page 2020

  6. [6]

    Bullock, Thomas Feix, and Aaron M

    Ian M. Bullock, Thomas Feix, and Aaron M. Dollar. The Y ale human grasping dataset: Grasp, object, and task data in household and machine shop environments.IJRR, 2015. 3

    work page 2015

  7. [7]

    The YCV object and model set: Towards common benchmarks for manipulation research

    Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar. The YCV object and model set: Towards common benchmarks for manipulation research. In ICAR, 2015. 3

    work page 2015

  8. [8]

    Narang, Karl V an Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox

    Y u-Wei Chao, Wei Y ang, Y u Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Y ashraj S. Narang, Karl V an Wyk, Umar Iqbal, Stan Birchfield, Jan Kautz, and Dieter Fox. DexYCB: A benchmark for capturing hand grasping of objects. InCVPR,

    work page

Show all 76 references
  1. [9]

    Dense hand-object(HO) GraspNet with full grasping taxonomy and dynamics

    Woojin Cho, Jihyun Lee, Minjae Yi, Minje Kim, Taeyun Woo, Donghwan Kim, Taewook Ha, Hyokeun Lee, Je-Hwan Ryu, Woon- tack Woo, and Tae-Kyun Kim. Dense hand-object(HO) GraspNet with full grasping taxonomy and dynamics. InECCV, 2024. 3

    2024

  2. [10]

    Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100.IJCV, 2022

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for EPIC-KITCHENS-100.IJCV, 2022. 3

    2022

  3. [11]

    BlenderProc: Reducing the reality gap with photorealistic rendering

    Maximilian Denninger, Martin Sundermeyer, Dominik Winkel- bauer, Dmitry Olefir, Tom´aˇs Hodaˇn, Y oussef Zidan, Mohamad Elbadrawy, Markus Knauer, Harinandan Katam, and Ahsan Lodhi. BlenderProc: Reducing the reality gap with photorealistic rendering. RSS W orkshops, 2020. 3

    2020

  4. [12]

    Blenderproc

    Maximilian Denninger, Martin Sundermeyer, Dominik Winkel- bauer, Y oussef Zidan, Dmitry Olefir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam. Blenderproc. arXiv preprint arXiv:1911.01911, 2019. 3

    1911 arXiv

  5. [13]

    Project Aria: A new tool for egocentric multi-modal AI research, 2023

    Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, Andrew Turner, Arjang Talattof, Arnie Y uan, Bilal Souti, Brighid Meredith, Cheng Peng, Chris Sweeney, Cole Wilson, Dan Barnes, Daniel DeTone, David Caruso, Derek V alleroy, Dinesh Ginjupalli, Dunc...

    2023

  6. [14]

    The P ASCAL visual object classes (VOC) challenge.IJCV, 2010

    Mark Everingham, Luc V an Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The P ASCAL visual object classes (VOC) challenge.IJCV, 2010. 3

    2010

  7. [15]

    ARCTIC: A dataset for dexterous bimanual hand-object manipulation

    Zicong Fan, Omid Taheri, Dimitrios Tzionas, Muhammed Kocabas, Manuel Kaufmann, Michael J Black, and Otmar Hilliges. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. InCVPR, 2023. 3, 4

    2023

  8. [16]

    Alireza Fathi, Xiaofeng Ren, and James M. Rehg. Learning to recognize objects in egocentric activities. InCVPR, 2011. 3

    2011

  9. [17]

    Category-level 6D object pose estimation in the wild: A semi-supervised learning approach and a new dataset

    Y ang Fu and Xiaolong Wang. Category-level 6D object pose estimation in the wild: A semi-supervised learning approach and a new dataset. NeurIPS, 2022. 3

    2022

  10. [18]

    First-person hand action benchmark with RGB-D videos and 3D hand pose annotations

    Guillermo Garcia-Hernando, Shanxin Y uan, Seungryul Baek, and Tae-Kyun Kim. First-person hand action benchmark with RGB-D videos and 3D hand pose annotations. InCVPR, 2018. 3

    2018

  11. [19]

    Are we ready for autonomous driving? The KITTI vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? The KITTI vision benchmark suite. In CVPR, 2012. 3

    2012

  12. [20]

    Zero-shot category-level object pose estimation

    Walter Goodwin, Sagar V aze, Ioannis Havoutis, and Ingmar Posner. Zero-shot category-level object pose estimation. InECCV, 2022. 8

    2022

  13. [21]

    HANDAL: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions

    Andrew Guo, Bowen Wen, Jianhe Y uan, Jonathan Tremblay, Stephen Tyree, Jeffrey Smith, and Stan Birchfield. HANDAL: A dataset of real-world manipulable object categories with pose annotations, affordances, and reconstructions. InIROS, 2023. 3

    2023

  14. [22]

    In-hand 3D object scanning from an RGB sequence

    Shreyas Hampali, Tomas Hodan, Luan Tran, Lingni Ma, Cem Keskin, and Vincent Lepetit. In-hand 3D object scanning from an RGB sequence. InCVPR, 2023. 5

    2023

  15. [23]

    HOnnotate: A method for 3D annotation of hand and object poses

    Shreyas Hampali, Mahdi Rad, Markus Oberweger, and Vincent Lepetit. HOnnotate: A method for 3D annotation of hand and object poses. InCVPR, pages 3196–3206, 2020. 3

    2020

  16. [24]

    Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3D pose estimation

    Shreyas Hampali, Sayan Deb Sarkar, Mahdi Rad, and Vincent Lepetit. Keypoint transformer: Solving joint identification in challenging hands and object interactions for accurate 3D pose estimation. InCVPR, 2022. 3

    2022

  17. [25]

    Twigg, and Kenrick Kin

    Shangchen Han, Beibei Liu, Robert Wang, Y uting Y e, Christo- pher D. Twigg, and Kenrick Kin. Online optical marker-based hand tracking with deep labels.ACM Trans. Graph., 37(4):166:1– 166:10, July 2018. 13

    2018

  18. [26]

    UmeTrack: Unified multi-view end-to-end hand tracking for VR

    Shangchen Han, Po-chen Wu, Y ubo Zhang, Beibei Liu, Linguang Zhang, Zheng Wang, Weiguang Si, Peizhao Zhang, Y ujun Cai, Tomas Hodan, et al. UmeTrack: Unified multi-view end-to-end hand tracking for VR. InSIGGRAPH Asia 2022, 2022. 3, 4, 5, 6, 13

    2022

  19. [27]

    Learning joint reconstruction of hands and manipulated objects

    Y ana Hasson, G¨ul V arol, Dimitrios Tzionas, Igor Kalevatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated objects. InCVPR, 2019. 3

    2019

  20. [28]

    Mask R-CNN

    Kaiming He, Georgia Gkioxari, Piotr Doll´ar, and Ross Girshick. Mask R-CNN. ICCV, 2017. 7

    2017

  21. [29]

    Towards 3D hand tracking using a deformable model

    Tony Heap and David Hogg. Towards 3D hand tracking using a deformable model. InProceedings of the Second International Conference on Automatic F ace and Gesture Recognition, 1996. 3

    1996

  22. [30]

    Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes

    Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab. Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. InACCV, 2013. 3

    2013

  23. [31]

    T -LESS: An RGB-D dataset for 6D pose estimation of texture-less objects.WACV, 2017

    Tom´aˇs Hodaˇn, Pavel Haluza,ˇStˇep´an Obdrˇz´alek, Jiˇr´ı Matas, Mano- lis Lourakis, and Xenophon Zabulis. T -LESS: An RGB-D dataset for 6D pose estimation of texture-less objects.WACV, 2017. 3

    2017

  24. [32]

    BOP: Benchmark for 6D object pose estimation.ECCV, 2018

    Tom´aˇs Hodaˇn, Frank Michel, Eric Brachmann, Wadim Kehl, Anders Glent Buch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, Caner Sahin, Fabian Manhardt, Federico Tombari, Tae-Kyun Kim, Jiˇr´ı Matas, and Carsten Rother. BOP: Benchmark for 6D object pos...

    2018

  25. [33]

    BOP Challenge 2020 on 6D object localization

    Tom´aˇs Hodaˇn, Martin Sundermeyer, Bertram Drost, Y ann Labb´e, Eric Brachmann, Frank Michel, Carsten Rother, and Jiˇr´ı Matas. BOP Challenge 2020 on 6D object localization. InECCV, 2020. 3, 4

    2020

  26. [34]

    BOP challenge 2023 on detection, segmentation and pose estimation of seen and unseen rigid objects

    Tom´aˇs Hodaˇn, Martin Sundermeyer, Y ann Labb´e, V an Nguyen Nguyen, Gu Wang, Eric Brachmann, Bertram Drost, Vincent Lepetit, Carsten Rother, and Jiˇr´ı Matas. BOP challenge 2023 on detection, segmentation and pose estimation of seen and unseen rigid objects. CVPRW, 2024. 3, 5

    2023

  27. [35]

    Photorealistic image synthesis for object instance detection

    Tom´aˇs Hodaˇn, Vibhav Vineet, Ran Gal, Emanuel Shalev, Jon Hanzelka, Treb Connell, Pedro Urbina, Sudipta Sinha, and Brian Guenter. Photorealistic image synthesis for object instance detection. ICIP, 2019. 4

    2019

  28. [36]

    House- Cat6D – a large-scale multi-modal category level 6D object perception dataset with household objects in realistic scenarios

    HyunJun Jung, Guangyao Zhai, Shun-Cheng Wu, Patrick Ruhkamp, Hannah Schieber, Giulia Rizzoli, Pengyuan Wang, Hongcheng Zhao, Lorenzo Garattoni, Sven Meier, et al. House- Cat6D – a large-scale multi-modal category level 6D object perception dataset with household objects in rea...

    2022 arXiv

  29. [37]

    Efficient intersection of three quadrics and applications in computer vision

    Zuzana Kukelova, Jan Heller, and Andrew Fitzgibbon. Efficient intersection of three quadrics and applications in computer vision. In CVPR, 2016. 7

    2016

  30. [38]

    H2O: A benchmark for egocentric hand-object interaction recognition.IEEE Transactions on Multimedia, 2020

    Taein Kwon, Bugra Tekin, Jan Stuhmer, Federica Bogo, and Marc Pollefeys. H2O: A benchmark for egocentric hand-object interaction recognition.IEEE Transactions on Multimedia, 2020. 3

    2020

  31. [39]

    HOI4D: A 4D egocentric dataset for category-level human-object interaction

    Y unze Liu, Y un Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. HOI4D: A 4D egocentric dataset for category-level human-object interaction. In CVPR, 2022. 3

    2022

  32. [40]

    The PBR guide

    Wes McDermott. The PBR guide. Allegorithmic, 2018. 2, 4

    2018

  33. [41]

    Quest 3.https://www.meta.com/quest/quest- 3/, 2023

    Meta. Quest 3.https://www.meta.com/quest/quest- 3/, 2023. 1, 2, 4, 13

    2023

  34. [42]

    NeRF: Representing scenes as neural radiance fields for view synthesis.ECCV, 2020

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis.ECCV, 2020. 5

    2020

  35. [43]

    Interhand2.6M: A dataset and baseline for 3D interacting hand pose estimation from a single RGB image

    Gyeongsik Moon, Shoou-I Y u, He Wen, Takaaki Shiratori, and Kyoung Mu Lee. Interhand2.6M: A dataset and baseline for 3D interacting hand pose estimation from a single RGB image. In ECCV, 2020. 3

    2020

  36. [44]

    Real-time hand tracking under occlusion from an egocentric RGB-D sensor

    Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. Real-time hand tracking under occlusion from an egocentric RGB-D sensor. In ICCV, 2017. 3

    2017

  37. [45]

    Visual learning and recognition of 3-D objects from appearance.IJCV, 1995

    Hiroshi Murase and Shree K Nayar. Visual learning and recognition of 3-D objects from appearance.IJCV, 1995. 3

    1995

  38. [46]

    KinectFusion: Real-time dense surface mapping and tracking.ISMAR, 2011

    Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. KinectFusion: Real-time dense surface mapping and tracking.ISMAR, 2011. 3

    2011

  39. [47]

    AssemblyHands: Towards egocentric activ- ity understanding via 3D hand pose estimation

    Takehiko Ohkawa, Kun He, Fadime Sener, Tomas Hodan, Luan Tran, and Cem Keskin. AssemblyHands: Towards egocentric activ- ity understanding via 3D hand pose estimation. InCVPR, 2023. 3

    2023

  40. [48]

    HANDS18: Methods, techniques and applications for hand observation

    Iason Oikonomidis, Guillermo Garcia-Hernando, Angela Y ao, Antonis Argyros, Vincent Lepetit, and Tae-Kyun Kim. HANDS18: Methods, techniques and applications for hand observation. In ECCVW, 2018. 1

    2018

  41. [49]

    DINOv2: Learning robust visual features without supervision.TMLR, 2024

    Maxime Oquab, Timoth´ee Darcet, Th´eo Moutakanni, Huy V o, Marc Szafraniec, V asil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision.TMLR, 2024. 2, 5, 8

    2024

  42. [50]

    FoundPose: Unseen ob- ject pose estimation with foundation features.ECCV, 2024

    Evin Pınar ¨Ornek, Y ann Labb´e, Bugra Tekin, Lingni Ma, Cem Ke- skin, Christian Forster, and Tomas Hodan. FoundPose: Unseen ob- ject pose estimation with foundation features.ECCV, 2024. 2, 5, 6

    2024

  43. [51]

    Spatial cognition from egocentric video: Out of sight, not out of mind.arXiv preprint arXiv:2404.05072, 2024

    Chiara Plizzari, Shubham Goel, Toby Perrett, Jacob Chalk, Angjoo Kanazawa, and Dima Damen. Spatial cognition from egocentric video: Out of sight, not out of mind.arXiv preprint arXiv:2404.05072, 2024. 5, 7, 8

    2024 arXiv

  44. [52]

    Enigma-51: Towards a fine-grained understanding of human-object interactions in industrial scenarios.IEEE Winter Conference on Application of Computer V ision (WACV), 2024

    Francesco Ragusa, Rosario Leonardi, Michele Mazzamuto, Claudia Bonanno, Rosario Scavo, Antonino Furnari, and Giovanni Maria Farinella. Enigma-51: Towards a fine-grained understanding of human-object interactions in industrial scenarios.IEEE Winter Conference on Application of ...

    2024

  45. [53]

    EgoBlur: Responsible innovation in Aria, 2023

    Nikhil Raina, Guruprasad Somasundaram, Kang Zheng, Sagar Miglani, Steve Saarinen, Jeff Meissner, Mark Schwesinger, Luis Pesqueira, Ishita Prasad, Edward Miller, Prince Gupta, Mingfei Y an, Richard Newcombe, Carl Ren, and Omkar M Parkhi. EgoBlur: Responsible innovation in Aria,...

    2023

  46. [54]

    Visual tracking of high DoF articulated structures: an application to human hand tracking

    James M Rehg and Takeo Kanade. Visual tracking of high DoF articulated structures: an application to human hand tracking. In ECCV, 1994. 3

    1994

  47. [55]

    Machine perception of three-dimensional solids

    Lawrence G Roberts. Machine perception of three-dimensional solids. PhD thesis, Massachusetts Institute of Technology, 1963. 3

    1963

  48. [56]

    Embodied hands: Modeling and capturing hands and bodies together.TOG,

    Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together.TOG,

  49. [57]

    Imagenet large scale visual recognition challenge.IJCV, 2015

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge.IJCV, 2015. 3

    2015

  50. [58]

    A taxonomy and evaluation of dense two-frame stereo correspondence algorithms

    Daniel Scharstein and Richard Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV, 2002. 3

    2002

  51. [59]

    Structure-from- motion revisited

    Johannes L Schonberger and Jan-Michael Frahm. Structure-from- motion revisited. InCVPR, 2016. 5

    2016

  52. [60]

    Understanding human hands in contact at internet scale

    Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at internet scale. InCVPR,

  53. [61]

    Real-time joint tracking of a hand manipulating an object from RGB-D input

    Srinath Sridhar, Franziska Mueller, Michael Zollh¨ofer, Dan Casas, Antti Oulasvirta, and Christian Theobalt. Real-time joint tracking of a hand manipulating an object from RGB-D input. InECCV,

  54. [62]

    OnePose: One-shot object pose estimation without cad models

    Jiaming Sun, Zihao Wang, Siyu Zhang, Xingyi He, Hongcheng Zhao, Guofeng Zhang, and Xiaowei Zhou. OnePose: One-shot object pose estimation without cad models. InCVPR, 2022. 3, 5

    2022

  55. [63]

    Cascaded hand pose regression

    Xiao Sun, Yichen Wei, Shuang Liang, Xiaoou Tang, and Jian Sun. Cascaded hand pose regression. InCVPR, 2015. 3

    2015

  56. [64]

    BOP challenge 2022 on detection, segmentation and pose estimation of specific rigid objects.CVPRW, 2023

    Martin Sundermeyer, Tomas Hodan, Y ann Labb´e, Gu Wang, Eric Brachmann, Bertram Drost, Carsten Rother, and Jiri Matas. BOP challenge 2022 on detection, segmentation and pose estimation of specific rigid objects.CVPRW, 2023. 3

    2022

  57. [65]

    Latent regression forest: Structured estimation of 3D articulated hand posture

    Danhang Tang, Hyung Jin Chang, Alykhan Tejani, and Tae-Kyun Kim. Latent regression forest: Structured estimation of 3D articulated hand posture. InCVPR, 2014. 3

    2014

  58. [66]

    Real-time continuous pose recovery of human hands using convolutional networks.T oG, 2014

    Jonathan Tompson, Murphy Stein, Y ann Lecun, and Ken Perlin. Real-time continuous pose recovery of human hands using convolutional networks.T oG, 2014. 3

    2014

  59. [67]

    Capturing hands in action using discriminative salient points and physics simulation.IJCV, 2016

    Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall. Capturing hands in action using discriminative salient points and physics simulation.IJCV, 2016. 3

    2016

  60. [68]

    Hololens 2 research mode as a tool for computer vision research.arXiv, 2020

    Dorin Ungureanu, Federica Bogo, Silvano Galliani, Pooja Sama, Xin Duan, Casey Meekhof, Jan St¨uhmer, Thomas J Cashman, Bugra Tekin, Johannes L Sch¨onberger, et al. Hololens 2 research mode as a tool for computer vision research.arXiv, 2020. 2

    2020

  61. [69]

    Ho-cap: A capture system and dataset for 3d recon- struction and pose tracking of hand-object interaction, 2024

    Jikai Wang, Qifan Zhang, Y u-Wei Chao, Bowen Wen, Xiaohu Guo, and Y u Xiang. Ho-cap: A capture system and dataset for 3d recon- struction and pose tracking of hand-object interaction, 2024. 3, 4

    2024

  62. [70]

    Phocal: A multi-modal dataset for category-level object pose estimation with photometrically challenging objects

    Pengyuan Wang, HyunJun Jung, Yitong Li, Siyuan Shen, Rahul Parthasarathy Srikanth, Lorenzo Garattoni, Sven Meier, Nas- sir Navab, and Benjamin Busam. Phocal: A multi-modal dataset for category-level object pose estimation with photometrically challenging objects. InCVPR, 2022. 3

    2022

  63. [71]

    PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes.RSS, 2018

    Y u Xiang, Tanner Schmidt, V enkatraman Narayanan, and Dieter Fox. PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes.RSS, 2018. 3

    2018

  64. [72]

    Depth anything v2

    Lihe Y ang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv:2406.09414, 2024. 7, 8

    2024 arXiv

  65. [73]

    The 2017 HANDS in the million challenge on 3D hand pose estimation

    Shanxin Y uan, Qi Y e, Guillermo Garcia-Hernando, and Tae-Kyun Kim. The 2017 HANDS in the million challenge on 3D hand pose estimation. arXiv preprint arXiv:1707.02237, 2017. 3

    2017 arXiv

  66. [74]

    3D hand pose tracking and es- timation using stereo matching.arXiv preprint arXiv:1610.07214,

    Jiawei Zhang, Jianbo Jiao, Mingliang Chen, Liangqiong Qu, Xiaobin Xu, and Qingxiong Y ang. 3D hand pose tracking and es- timation using stereo matching.arXiv preprint arXiv:1610.07214,

    arXiv

  67. [75]

    Fine-grained egocentric hand-object segmentation: Dataset, model, and applications

    Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand-object segmentation: Dataset, model, and applications. InECCV, 2022. 3, 7, 14

    2022

  68. [76]

    FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images

    Christian Zimmermann, Duygu Ceylan, Jimei Y ang, Bryan Russell, Max Argus, and Thomas Brox. FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images. In ICCV, 2019. 3 Appendix In this appendix, we provide details about the Aria glasses (Sec. A) ...

    2019

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.