REVIEW 28 cited by
DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Imitation learning from human hand motion data presents a promising avenue for imbuing robots with human-like dexterity in real-world manipulation tasks. Despite this potential, substantial challenges persist, particularly with the portability of existing hand motion capture (mocap) systems and the complexity of translating mocap data into effective robotic policies. To tackle these issues, we introduce DexCap, a portable hand motion capture system, alongside DexIL, a novel imitation algorithm for training dexterous robot skills directly from human hand mocap data. DexCap offers precise, occlusion-resistant tracking of wrist and finger motions based on SLAM and electromagnetic field together with 3D observations of the environment. Utilizing this rich dataset, DexIL employs inverse kinematics and point cloud-based imitation learning to seamlessly replicate human actions with robot hands. Beyond direct learning from human motion, DexCap also offers an optional human-in-the-loop correction mechanism during policy rollouts to refine and further improve task performance. Through extensive evaluation across six challenging dexterous manipulation tasks, our approach not only demonstrates superior performance but also showcases the system's capability to effectively learn from in-the-wild mocap data, paving the way for future data collection methods in the pursuit of human-level robot dexterity. More details can be found at https://dex-cap.github.io
Forward citations
Cited by 28 Pith papers
-
Handroid: Bridging Dexterous Hand and Humanoid
A single 27-DoF body doubles as an anthropomorphic dexterous hand and a 0.33 m desktop humanoid, with a unified control stack for manipulation, locomotion, and embodiment switching.
-
Feel the Force: Contact-Driven Learning from Humans
FeelTheForce trains a robot policy on human tactile demonstrations, predicting desired contact forces and using a PD controller to track them on the robot gripper, achieving 77% success across five force-sensitive tasks.
-
DexDirect: Direct Kinesthetic Arm Guidance for Efficient Dexterous Demonstration Collection
A hybrid kinesthetic-arm-plus-webcam-hand teleoperation interface achieved 17x/3x higher demonstration throughput than vision baselines and trained a 90%-success pick-and-place policy in a ten-person study.
-
HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone
Robot-free HiFi-UMI demonstrations can replace teleoperated real-robot data in post-training: three policy backbones matched in-domain teleoperation within 3.1 percentage points, including 85% success on a precision i...
-
Scale Up Strategically: Learning Compositional Generalization via Bias-Aware Evaluation and Data Collection for Robotic Manipulation
Robot instruction-following policies consistently over-rely on color and under-ground verbs and size, and reallocating training demonstrations to under-grounded factors improves compositional generalization with fewer demos.
-
Towards Human-level Dexterous Teleoperation
A single-stage RL co-tracking controller trained on consecutive human-derived hand–object subgoals achieves ~75% real-robot success on long-horizon dexterous teleoperation where baselines fail.
-
TactiDex: A Real-World Tactile-Guided Benchmark for Human-Like Dexterous Manipulation
A tactile-rich HOI dataset plus a tri-component force reward improves contact fidelity and success of human-to-robot dexterous transfer over kinematic imitation alone.
-
Cross-Embodiment Robot Manipulation via a Unified Hand Action Space
UHAS maps hand actions to deformations of a shared unit sphere and recovers joint commands via cascade IK, enabling multi-hand RL, zero-shot transfer, and modest real-world cube reorientation on LEAP and Allegro.
-
Play2Perfect: What Matters in Dexterous Play Pretraining for Precise Assembly?
Task-agnostic RL play pretraining on diverse objects yields a reusable dexterous prior that makes sparse-reward assembly learning ~33× more sample-efficient and enables zero-shot sim-to-real transfer on tight insertio...
-
Act, Sense, Act: Learning Active Perception from Large-Scale Egocentric Human Data
CoMe-VLA combines cognitive subtask labels and dual-track memory with human egocentric pretraining, reaching 83% mean success on five active-perception manipulation tasks.
-
Dexplore: Scalable Neural Control for Dexterous Manipulation from Reference-Scoped Exploration
Dexplore learns dexterous robotic hand control from human MoCap demonstrations by treating them as soft, adaptively shrinking spatial references, then distills the policy into a vision-based controller.
-
H-RDT: Human Manipulation Enhanced Bimanual Robotic Manipulation
Pre-training a diffusion-transformer robot policy on 338K human hand-manipulation episodes, then fine-tuning with modular adapters, improves bimanual manipulation success across simulation and real robots.
-
RwoR: Generating Robot Demonstrations from Human Hand Collection for Policy Learning without Robot
A generative model and wrist camera turn human hand videos into robot gripper demonstrations that train manipulation policies at success rates close to those trained on real gripper data.
-
DexVLG: Dexterous Vision-Language-Grasp Model at Scale
DexVLG is a vision-language model trained on 170 million simulated dexterous grasps that generates hand poses aligned with language instructions about which part of an object to grasp.
-
Knowledge-Driven Imitation Learning: Enabling Generalization Across Diverse Conditions
A semantic keypoint graph matched to novel objects lets imitation-learned manipulation policies generalize with a quarter of the demonstrations.
-
Vision in Action: Learning Active Perception from Human Demonstrations
ViA trains bimanual manipulation policies from human demonstrations that include active head-camera movement, using a 6-DoF robot neck and a VR interface with point-cloud rendering, reporting large gains on three occl...
-
Object-centric 3D Motion Field for Robot Learning from Human Videos
A policy trained only on human RGBD videos, with a denoised object-centric 3D motion field as action representation, achieves about 55% average success on five real manipulation tasks where prior flow-based methods st...
-
DexMachina: Functional Retargeting for Bimanual Dexterous Manipulation
DexMachina uses decaying virtual object controllers as a curriculum to train bimanual dexterous policies that track demonstrated object states, and reports large gains over baselines on a new six-hand benchmark.
-
Learning Generalizable Robot Policy with Human Demonstration Video as a Prompt
A two-stage pipeline trains a robot policy that accepts a human demonstration video as a prompt and generalizes beyond its robot training tasks, with success rates of up to 79 percent on known task variations and unde...
-
EgoZero: Robot Learning from Smart Glasses
Robot policies trained only on egocentric human videos from smart glasses transfer zero-shot to a Franka gripper, with 70% success across 7 manipulation tasks.
-
Object-Focus Actor for Data-efficient Robot Generalization Dexterous Manipulation
Object-Focus Actor makes dexterous manipulation policies generalize to new object positions and backgrounds by focusing on the hand-object region and using relative poses and actions, needing only 10 to 30 demonstrations.
-
Adaptive Visuo-Tactile Fusion with Predictive Force Attention for Dexterous Manipulation
A force-guided attention module and future-force prediction auxiliary task improve visuo-tactile fusion for dexterous manipulation, reaching 93% average success in real robot trials.
-
CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-World
CordViP achieves strong real-world dexterous manipulation by feeding a diffusion policy with pose-tracked 3D object models and hand point clouds, pretrained on contact maps and arm-hand coordination.
-
TacPrint: A Wearable Fingertip Tactile Sensor for Human-to-Robot Contact Reproduction
A low-cost wearable fingertip sensor estimates dense contact-depth maps from 24 capacitive channels and uses them to substantially improve robot grasping and wiping in human-to-robot replay.
-
AXIS: A Growable Community-Driven Data Engine for Scalable Robot Manipulation
Pretraining π0.5 on the crowdsourced AXIS simulation dataset (207 tasks, 50K+ trajectories) raises downstream LIBERO-Plus success from 83.9% to 88.8% as the pretraining corpus grows from none to the full dataset.
-
The Latent Color Subspace: Emergent Order in High-Dimensional Chaos
FLUX.1’s VAE latent space contains an interpretable Hue–Saturation–Lightness structure that enables training-free color prediction and control via closed-form latent edits.
-
ImMimic: Cross-Domain Imitation from Human Videos via Mapping and Interpolation
A co-training framework that maps retargeted human hand trajectories to robot demonstrations with dynamic time warping and MixUp interpolation improves robot manipulation success rates and smoothness across four embodiments.
-
A Survey: Learning Embodied Intelligence from Physical Simulators and World Models
Embodied intelligence learning is reviewed through the complementary lenses of physical simulators and world models, with a proposed IR-L0 to IR-L4 robot capability taxonomy.
Discussion (0). Continue with ORCID to comment.