{"id":"f12912c0-ed93-4169-94e9-7f9d90e84f9e","arxiv_id":"2412.04244","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"GigaHands provides 14,000 bimanual hand clips, 84,000 text annotations, and 183 million frames from 51 cameras, outperforming smaller datasets in text-to-motion and captioning tasks.","lead":"This paper releases GigaHands, a dataset of 14,000 bimanual hand-activity clips captured from 51 cameras, with 84,000 text descriptions and automatically estimated 3D hand and object poses. It is the largest annotated bimanual hand dataset to date and is shown to improve text-driven hand motion generation and captioning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Automatic 3D pose estimation is used without ground-truth validation; if hand/object poses are biased, the claimed scale-driven improvements are unsupported.","rationale":"The paper makes a strong contribution: a large, multi-view, bimanual hand activity dataset with text annotations and a procedural pipeline to elicit diverse actions. The downstream experiments on text-to-motion and motion captioning are reasonable, and the scaling curves are suggestive. However, the central claim of accuracy and the conclusion that scale alone drives improvements rest on the correctness of the automatic annotation pipeline. The paper does not provide any independent validation of the 3D hand/object poses. The supplementary's \"valid rate\" and \"mask coverage\" are proxies that can be high even with systematic bias (e.g., a smoothed average pose would pass kinematic checks). Notably, the object mask coverage of 78.5% indicates substantial silhouette mismatch, and the paper's own admissions about HaMeR depth ambiguity and the need for RANSAC outlier removal suggest fragility. A robust dataset paper for 3D reconstruction should validate a subset against marker-based or depth ground truth; without this, users cannot trust the annotations for robotics or AR. I therefore agree with the reader's conditional verdict. Secondary issues—the overcounting of \"unique hand poses\" (0.37e9 vs 3.7M actual 3D poses in Table 1) and inflated verb counts from LLM augmentation—should also be corrected, but the accuracy validation is the load-bearing concern.","tokens_in":35902,"tokens_out":8962,"duration_ms":88743,"concrete_test":"Select a random subset of about 10 motion clips (≈5k frames) and capture the same performances with a marker-based system (e.g., VR gloves or OptiTrack) or with a calibrated high-resolution depth sensor array. Compute mean per-joint position error (MPJPE) between GigaHands MANO meshes and the ground-truth marker positions after Procrustes alignment. Also compute object ADD(S) error against manually verified 6D poses for a subset of rigid objects. Compare these errors against the reported inter-joint spacing (typically 1–2 cm) and against existing datasets (e.g., OakInk2 reports MPJPE around 8–12 mm). If MPJPE exceeds ~15 mm or object pose error exceeds ~2 cm/10°, the annotation accuracy claim fails and the dataset cannot be considered \"accurate\" as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper relies on a fully automatic 3D hand/object estimation pipeline (Sections 4.4–4.5) without quantitative validation against any ground truth. The supplementary reports only self-consistency metrics: a 97.9% \"valid rate\" for triangulated keypoints (defined by heuristics like kinematic plausibility, not accuracy) and an average 78.5% mask coverage for object pose (Supp. Table 11). The latter means that on average about 21% of the object silhouette is not explained by the fitted mesh, a sizable error. For hands, the paper admits HaMeR \"lacks accurate depth\" (Section 4.4) and that triangulation requires accurate 2D keypoints and calibration, but no reprojection error or comparison to a marker-based MoCap subset is provided. If poses are systematically biased in fast motions or occlusions—exactly where markerless RGB estimation struggles—the downstream motion synthesis and captioning models are trained on biased poses. The reported improvements are then measured on the same biased test set, so they could reflect the noise, not true scale benefits. No independent accuracy benchmark is offered.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GigaHands, a large multi-view RGB dataset of bimanual hand activities captured from 56 subjects and 417 objects, with 34 hours of footage, 14k motion clips, 183M frames, and 84k text annotations. The authors describe a markerless capture pipeline with procedural instruction elicitation, automatic 3D hand and object pose estimation, and text augmentation, and they report experiments on text-driven motion synthesis, motion captioning, and dynamic radiance field reconstruction. The central claims are that GigaHands is the largest bimanual hand dataset, that its automatic annotation pipeline produces accurate 3D hand/object poses, and that its scale and diversity improve downstream tasks.","tokens_in":36138,"tokens_out":3395,"duration_ms":35842,"significance":"If validated, GigaHands would be a substantial community resource: the scale in hours, clips, and camera views is larger than existing bimanual hand datasets, and the pairing of dense 3D annotations with rich text is a useful combination. The paper also ships a fully automatic annotation pipeline and reproducible experiments across three backbones, which are strengths. The main significance hinges on two facts that are currently not established: that the automatically estimated 3D hand and object poses are accurate enough to serve as training targets, and that the reported cross-dataset improvements are meaningful despite per-dataset evaluation embeddings. Because these points are load-bearing for the paper's claims, the significance is conditional on additional validation.","major_comments":[{"comment":"The hand motion estimation pipeline is validated only by a self-consistency 'valid rate' (97.9% for the chosen YOLOv9+HaMeR configuration), defined by missing keypoints, kinematic plausibility, and temporal consistency. This does not measure accuracy: a system can be kinematically plausible and temporally smooth while systematically biased in depth, especially since the paper itself states that HaMeR 'lacks accurate depth' (Section 4.4). The central claim that GigaHands provides accurate 3D hand poses requires a quantitative comparison against ground truth, for example a small marker-based MoCap subset or manually annotated keypoint frames, reported as 3D PCK or MPJPE. Without such validation, models trained on the annotations may inherit depth and pose biases, and the downstream improvements reported in Section 5.1 cannot be attributed to accurate supervision.","section":"Section 4.4, Supp. Section 7, Supp. Table 10"},{"comment":"Object pose accuracy is evaluated only by mask coverage, which averages 78.5% over sequences and is only 45.2% after coarse estimation. Mask coverage conflates segmentation errors, mesh shape errors, and pose errors, and it does not quantify translation or rotation accuracy. Given that the experimental sections include text-driven synthesis and motion forecasting that use object poses (e.g., Supp. Section 6), the claim that object motion estimates are 'precise' (Section 4.5) is not supported. The authors should report a direct object pose error (e.g., ADD or rotation/translation error) on a subset with known object poses, such as objects with fiducial markers or a small manually annotated test set.","section":"Section 4.5, Supp. Section 8, Supp. Table 11"},{"comment":"The evaluation of text-to-motion synthesis uses feature extractors trained independently for each dataset (Supp. Section 3), so the FID, R-Precision, and Multimodality scores are computed in different embedding spaces. While the 'upper bound' rows partially control for the difficulty of each test set, they do not make the scores directly comparable across datasets; for example, the GigaHands upper-bound FID is 0.002 while the TACO upper-bound FID is 0.045. The conclusion that 'models trained on GigaHands outperform others on all metrics except MM Dist.' (Section 5.1) is therefore not a valid cross-dataset comparison. The authors should either evaluate all models on a shared feature space, normalize metrics by their per-dataset upper bounds, or report cross-dataset evaluation on a common test set as the primary evidence.","section":"Section 5.1, Table 2, Supp. Section 3"},{"comment":"The verb count comparison in Figure 2 and the claim that GigaHands has more verbs than any other hand dataset are based on the 84k augmented text annotations, which are produced by LLM rephrasing of the original 14k descriptions. Because the augmentation prompt explicitly asks for varied verbs and descriptions, the 1,467-verb count partly reflects LLM paraphrasing rather than the diversity of the captured activities. The paper should report the verb count computed from the original 14k human/instructional annotations separately, and clarify whether the comparison in Figure 2 uses original or augmented text. This matters because verb diversity is presented as evidence of dataset diversity.","section":"Section 4.3, Section 1, Supp. Section 9.5"}],"minor_comments":[{"comment":"There is a typo: 'We demonstrate that multi-view RGB us sufficient' should read 'multi-view RGB is sufficient'.","section":"Section 2.2"},{"comment":"The phrase 'the right figure compares the verb sets' should be 'the right figure compares the verb counts' for accuracy, since the UpSet plot shows counts and overlaps, not sets themselves.","section":"Figure 2 caption"},{"comment":"The abstract states 'totaling 14k motion clips derived from 183 million frames', but Section 1 and Table 1 report 13.9k motion clips; please unify the rounding consistently.","section":"Abstract"},{"comment":"The sentence 'Feature extractors are trained independently for each dataset and subset' is important and should be stated in the main text before Table 2, not only in the supplement, because it affects the interpretation of all reported numbers.","section":"Section 5.1"},{"comment":"The phrase 'This expanded our 14k motion clips into 84k motion-text pairs' should clarify that this includes the original annotation plus five rephrasings; the current wording implies a six-fold expansion without explicitly stating the inclusion of the original description.","section":"Section 4.3"},{"comment":"Table 10 compares detection and keypoint methods on 'valid rate' but does not report whether the 60 motion clips used for this comparison are representative across the 25 scenes; a sentence describing the selection of these clips would improve reproducibility.","section":"Supp. Section 7, Table 10"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important data gap and the scale of the resource is genuinely impressive. The main issue is that the accuracy of the automatic 3D annotations, which are the core product of the dataset, is not validated against any ground truth; the reported heuristics and silhouette-based metrics are not sufficient for a dataset whose stated contribution is accurate dense 3D pose. The evaluation-comparability issue in Section 5.1 is also significant because it weakens the empirical claim that scale alone drives the improvements. I would support publication after the authors add a small but credible ground-truth validation subset for hands and objects, and either redo the cross-dataset comparison in a shared evaluation space or explicitly frame the results as within-dataset improvements only."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline is simple: GigaHands is the largest bimanual hand dataset by an order of magnitude—183M frames, 51 views, 14k clips, 84k text annotations, 56 subjects, 417 objects. The scale claims check out and the procedural instruct-to-annotate pipeline is a real contribution: it gets atomic verb-level instructions at capture time and uses LLM rephrasing only to augment text, not to invent new actions. The diversity analysis (t-SNE, verb sets) is reasonable, and the downstream experiments—motion synthesis, captioning, radiance fields—show plausible scaling benefits. The cross-dataset captioning and synthesis results are a plus.\n\nThe soft spot is the same one the supplementary tried to address and didn't: the 3D hand and object poses are produced by a fully automatic pipeline with no ground-truth validation. The 'valid rate' (97.9%) is a heuristic about kinematic plausibility, not accuracy. Object pose is evaluated by mask coverage: 45.2% coarse, 91.1% after first-frame refinement, dropping to 78.5% over full sequences—meaning roughly a fifth of the object silhouette is unexplained even in the best automatic estimate. For a dataset whose value depends on accurate 3D annotation, that is a load-bearing concern. The paper acknowledges HaMeR 'lacks accurate depth' and then triangulates; that may be fine, but nobody checks the final MANO fits against marker-based capture or even manual annotation on a sample. Without that, the claim that scale alone drives the observed improvements is not fully supported—the models train and evaluate on the same biased poses. This is fixable: a small MoCap subset or human-verified sample with per-joint error would settle it.\n\nThe verb-count story is slightly inflated: 1467 verbs include five paraphrases per clip, so the 'unique' verb set is partly augmentation. Not a fatal issue but should be reported separately.\n\nThe paper is a strong engineering contribution to a subfield that needs exactly this kind of resource. It deserves a serious referee, but it needs a revision that adds real validation and a public data release with a stable identifier.","headline":"GigaHands is genuinely massive and the pipeline is clever, but the lack of ground-truth validation of the automatic 3D poses is a load-bearing soft spot that a revision must fix.","tokens_in":36680,"tokens_out":2937,"would_cite":true,"duration_ms":26062,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GigaHands is the largest bimanual hand activity dataset, with 183 million annotated frames and 0.37 billion hand poses.","keywords":["bimanual hand activities","hand-object interaction","text-to-motion synthesis","motion captioning","3D hand pose estimation","multi-view capture","MANO hand model","large-scale dataset"],"falsifier":"Run marker-based motion capture on a few activities inside the same capture rig, then compare GigaHands' MANO hand poses and object 6D poses against that reference; large per-joint or per-frame errors on fast or occluded segments would undercut the claim that scale alone drives the reported improvements.","tokens_in":35732,"feed_emoji":"🖐️","tokens_out":5741,"duration_ms":53620,"temperature":0.7,"pith_summary":"The paper claims that existing datasets are too small, too narrow, and too sparsely annotated to support large models of bimanual hand activity, and introduces GigaHands to fill that gap. GigaHands contains 34 hours of activity from 56 subjects and 417 objects, organized into 14k motion clips paired with 84k text descriptions, with 3.7 million 3D bimanual hand poses and 51 camera views per clip. The paper argues that this scale and diversity improve text-driven hand motion synthesis and hand motion captioning, and it demonstrates both with quantitative comparisons against TACO and OakInk2. A sympathetic reader would care because the dataset is positioned as the missing resource for training models that can understand and generate natural two-handed manipulation.","feed_headline":"GigaHands delivers 183M annotated frames of two-handed activity","feed_subtitle":"Fourteen thousand motion clips paired with 84k text captions push text-to-motion and captioning models further.","key_machinery":"The load-bearing mechanism is the Instruct-to-Annotate acquisition pipeline. It starts from a pooled set of verbs mined from existing action datasets, uses an LLM to group verbs and objects into scenes and write temporally ordered instruction scripts, films subjects following those scripts with 51 synchronized RGB cameras, then has annotators split sequences into clips and uses an LLM to rephrase each clip into five text variations. The 3D annotations come from an automatic hybrid pipeline: hand bounding boxes and initial meshes from detection and keypoint models, multi-view triangulation with filtering, then MANO fitting; objects are segmented with foundation models and tracked by differentiable rendering against multi-view masks. This pipeline converts raw RGB video into paired motion and text data without markers, which is what allows the dataset to reach its scale.","core_discovery":"GigaHands is presented as the largest and most diverse bimanual hand activity dataset to date, with over 183 million unique frames each showing two hands, amounting to 0.37 billion unique hand poses. All frames come with 3D hand shape and pose in MANO form, 3D object shape and pose, segmentation masks, 2D/3D keypoints, camera poses, and dense text annotations. The authors claim that their markerless multi-camera setup plus a procedural instruct-to-annotate protocol produces this annotation at scale with minimal manual effort, and that models trained on GigaHands outperform models trained on existing datasets on most metrics for text-to-motion generation and motion captioning. On the dataset's own test set, a T2M-GPT model trained on GigaHands reaches higher R-Precision, lower FID, and better diversity than models trained on TACO or OakInk2, and the captioning model produces more diverse captions while matching retrieval accuracy.","pith_inferences":["If the automatic estimation pipeline carries systematic bias in fast or heavily occluded frames, then enlarging the dataset further may not fully replace ground-truth validation; a marker-based validation subset could test that directly.","The verb pool of 1,467 verbs, 580 of them unique to GigaHands, suggests that procedural instruction elicitation could be reused to expand the dataset into new scenarios such as larger-space activities or tool use without redesigning the pipeline.","The object meshes, 6D poses, and contact regions in GigaHands may support robotic manipulation learning, although the paper only sketches hand-object experiments and does not claim robot results."],"forward_implications":["Training on progressively larger subsets of GigaHands (10%, 20%, 50%, 80%, 100%) consistently improves FID, multimodal distance, and retrieval accuracy, so the paper's reported gains are driven by scale rather than by a single model choice.","A motion captioning model trained only on GigaHands can generate captions for unseen motions from TACO and OakInk2, suggesting that the dataset transfers to other 3D hand motion distributions.","The 51-view capture makes dynamic radiance field reconstruction possible, with synthesized test views reaching 29.50 PSNR on sampled clips.","The abundance of text annotations (84k for 14k clips) improves synthesis quality, and the paper's ablation shows that more annotations consistently boost generation metrics."],"supporting_citations":[{"why":"Supplies the TACO comparison baseline with bimanual tool-action-object clips and triplet labels used throughout the synthesis and captioning experiments.","marker":"[58]"},{"why":"Supplies the OakInk2 comparison baseline with bimanual manipulation and text descriptions that GigaHands is measured against.","marker":"[116]"},{"why":"Provides Ego-Exo4D, the current largest annotated 3D hand pose dataset, as the scale and continuity comparison for GigaHands.","marker":"[33]"},{"why":"Defines the MANO parametric hand model used for all hand meshes and derived 3D keypoints in the dataset.","marker":"[86]"},{"why":"Supplies the monocular hand mesh estimates that initialize the 2D keypoints in the automatic hand reconstruction pipeline.","marker":"[77]"},{"why":"Provides the SAM2 video segmentation model used to obtain object masks across views for object tracking.","marker":"[84]"},{"why":"Supplies the rotation-initialization strategy for object pose estimation from appearance features in the multi-view tracking pipeline.","marker":"[75]"},{"why":"Defines the R-Precision, FID, diversity, and multimodality metrics used to evaluate text-driven motion synthesis.","marker":"[34]"},{"why":"Provides the TM2T backbone used for motion captioning experiments and the reciprocal motion-text modeling approach.","marker":"[35]"}],"fun_headline_variants":["GigaHands: 183M frames of two-handed action","Largest bimanual hand dataset: 183M frames, 84k captions","GigaHands: 34 hours of dual-hand activity, fully annotated","Two-handed AI gets GigaHands: 183M annotated frames"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the automated 3D hand and object estimation pipeline producing accurate annotations despite having no quantitative validation against ground-truth motion capture, so systematic errors in fast or occluded frames would be baked into every model trained on the dataset.","fun_headline_variants_meta":{"raw":{"variants":["GigaHands: 183M frames of two-handed action","Largest bimanual hand dataset: 183M frames, 84k captions","GigaHands: 34 hours of dual-hand activity, fully annotated","Two-handed AI gets GigaHands: 183M annotated frames"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000484,"raw_usage":{"total_tokens":2368,"prompt_tokens":902,"completion_tokens":1466,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":1383}},"tokens_in":518,"tokens_out":1466,"duration_ms":10242,"temperature":1.0,"reasoning_tokens":1383,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:35:10.032706+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run marker-based motion capture on a few activities inside the same capture rig, then compare GigaHands' MANO hand poses and object 6D poses against that reference; large per-joint or per-frame errors on fast or occluded segments would undercut the claim that scale alone drives the reported improvements.","supporting_citations":[{"cited_title":"Image quality assessment through fsim, ssim, mse and psnr—a compar- ative study","cited_arxiv_id":null,"evidence_quote":"Provides Ego-Exo4D, the current largest annotated 3D hand pose dataset, as the scale and continuity comparison for GigaHands."},{"cited_title":"Mixture models for diverse machine translation: Tricks of the trade","cited_arxiv_id":null,"evidence_quote":"Defines the R-Precision, FID, diversity, and multimodality metrics used to evaluate text-driven motion synthesis."},{"cited_title":"Human motion diffusion model","cited_arxiv_id":null,"evidence_quote":"Provides the TM2T backbone used for motion captioning experiments and the reciprocal motion-text modeling approach."}],"review_version":1}