Pith. sign in

REVIEW 4 major objections 6 minor 73 references

Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A humanoid whose only interface is egocentric vision can search, grasp, carry, and place objects and open drawers — with no 3D object state — and the search emerges from training, not programming.

desk verdict Solid vision-driven humanoid control system with a genuinely new perception-as-interface task specification; the main open question is the PULSE-X latent action space's coverage of contact-rich dexterity. read the letter →

arxiv 2505.12278 v1 pith:7NPNAG3G submitted 2025-05-18 cs.RO cs.CV

Zhengyi Luo , Chen Tessler , Toru Lin , Ye Yuan , Tairan He , Wenli Xiao , Yunrong Guo , Gal Chechik
show 3 more authors
Kris Kitani Linxi Fan Yuke Zhu
This is my paper · ORCID
classification cs.ROcs.CV
keywords egocentricvisiondexterouswhole-bodycontrolvisualreinforcementlearningperception-as-interfacehumanoidactivesearchhouseholdmanipulationsimulatedhumanoids
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Perceptive Dexterous Control (PDC), which trains a simulated full-body humanoid with articulated hands to act solely from egocentric camera images and proprioception. Tasks are specified exclusively through visual cues drawn into the image — a bright green overlay on the target object, red on drawer handles, a 3D arrow marking the destination, and colored corner blocks signalling which hand to use and when to grasp or release — replacing the phase variables and 3D goal coordinates that prior whole-body controllers rely on. The authors show that one policy trained with reinforcement learning, acting through a motion-capture-derived latent prior, can search for, grasp, transport, and release objects and open drawers in procedurally generated kitchens, and that a stereo-vision version generalizes to unseen objects and scenes at success rates comparable to or better than a state-based oracle that always knows the object's position and shape. The paper's central demonstration is that human-like active search — scanning left and right, turning around, sweeping the countertop — emerges from training under partial observability rather than being explicitly designed, which matters because it points toward humanoids that need no hidden ground-truth state to act and whose tasks can be changed by changing the picture.

What carries the argument

The framework is carried by three pieces. First, perception-as-interface: task instructions are rendered into the agent's own camera image (a green object mask from instance segmentation, a red drawer-handle overlay, a 3D arrow for the goal, and colored 2D corner squares encoding handedness and grasp/release timing), so changing the task means changing the picture, not the policy input space. Second, PULSE-X: a conditional VAE distilled from a motion imitator trained on a large motion-capture archive, which defines a 48-dimensional latent action space; the high-level policy predicts a latent $z^{\text{PDC}}_t \in \mathbb{R}^{48}$, and the decoder $D_{\text{PULSE-X}}$ produces joint targets conditioned on proprioception, keeping motions human-like while cutting the control dimensionality from 153 joint targets to 48. Third, a lightweight CNN-GRU-MLP that ingests $128\times128$ RGB (or two $80\times80$ stereo images through a Siamese encoder, or RGB-D) plus proprioception, with the recurrent GRU layer supplying memory for the partially observable search task. Training uses PPO with a staged dense reward whose phases (approach, pre-grasp, grasp, transport, release) are scheduled by user-provided times $\lambda_{\text{start}}, \lambda_{\text{end}}$ and reference pre-grasp poses, plus early termination, and the kitchen policy is warm-started from the tabletop policy.

What would settle it

Decode the PULSE-X latent space exhaustively and check, for every pre-grasp the tasks require (each recorded grasp pose and the drawer-handle grip), whether some latent reconstructs it within task tolerance — a few centimeters at the wrist and fingertips while the body remains balanced. Any required pose that no latent reproduces is a task the framework cannot express regardless of perception quality; a positive control would add those motions to the prior's training data and show the policy then succeeds.

Watch

Extended reading notes

Core claim

PDC's central claim is that perception can serve as the complete task interface for dexterous whole-body control: if the image tells the agent what to pick up, where to take it, which hand to use, and when to contact and release, the policy needs no privileged state. Concretely, the paper trains a single CNN-GRU-MLP policy with proximal policy optimization whose output is a latent code in the action space of PULSE-X, a conditional variational autoencoder trained on large-scale motion-capture data; the decoder turns the latent and current proprioception into joint targets, so every movement is filtered through a human-motion prior. Evaluated on household objects, the stereo-vision policy lifts and places 96.4% of training objects and 91.8% of held-out objects, while the state-based oracle that receives exact object shape and pose reaches 99.1% on training objects but drops to 70.6% on the same held-out set — an inversion the authors attribute to vision capturing what generalizes while privileged shape features overfit. In the kitchen environment the policy locates the green-marked object in 95–99% of episodes, grasps it in about 80–85%, completes the carry-and-release trajectory in roughly 54–66%, and opens red-marked drawers at about 64% success, in both training and unseen kitchens. Alongside these numbers, the authors report the emergence of active search and whole-body coordination — head scans, 360-degree turns, leaning on the counter — as behaviors induced by the vision-driven setup rather than by explicit search logic.

Load-bearing premise

Every action passes through the latent space of PULSE-X, a prior trained on motion-capture data; if that latent space cannot express a motion the tasks demand — a finger curl inside a drawer handle, a lean the captures never contain — the whole framework fails no matter how well the vision works, and the paper does not measure that expressive coverage.

Editorial extensions

If this is right

  • A single policy body can be reused across tasks: the kitchen policy is warm-started from the tabletop policy and trained simultaneously for grasping and drawer-opening, so new skills can be added by fine-tuning without changing the architecture or the observation space.
  • Visual control generalizes to never-seen objects better than privileged control: on the held-out test objects, PDC's grasp success stays at 90–92% while the state-based oracle drops to 70.6%, implying that visual features transfer where canonical-shape encodings do not.
  • Stereo perception outperforms monocular RGB and RGB-D (96.4% vs 87.5% and 86.9% tabletop success), which the authors read as depth estimation emerging from the reaching and grasping task itself.
  • Training in randomized multi-object kitchens produces active search — scanning, turning, and countertop sweeps — that never appears in the simpler tabletop setting where the object is always in view.
  • Learning the visual encoder from scratch beats freezing a classification-pretrained encoder (90.1% vs 61.4–68.5% success), indicating that the representations that close the perception-action loop are task-specific, not generic classification features.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If perception-as-interface scales as its authors suggest, the same policy could accept any signal that can be drawn into the image — floating text instructions, a user's gaze, a cursor, even projected AR annotations — turning the humanoid into a general vision-conditioned motor system; the paper hints at floating text but does not test it.
  • The true expressive ceiling may be the motion prior, not the perception: since every action is a decoded PULSE-X latent, any hand-object interaction absent from the motion-capture archive (a precision pinch, an unusual grip) is unreachable no matter how clearly the vision system labels it, and the paper never measures the latent space's coverage of the interactions its tasks require.
  • The 'no privileged information' claim holds for the policy's observations only: the reward still uses exact 3D object positions, recorded pre-grasp poses, and a user-supplied grasp/release time schedule, so extending the framework to tasks without motion-capture equivalents would require replacing the staged reward rather than the perception interface.
  • The paper's own failure analysis — the agent does not attempt a second grasp after a miss, loses clarity when cabinets occlude the target, and shakes its head — points to the first testable improvements: a rewarded re-grasp policy and gaze stabilization, both compatible with the visual interface because the object mask persists in view.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Perceptive Dexterous Control (PDC), a reinforcement-learning framework for a simulated SMPL-X humanoid that uses egocentric vision as the sole task-specification interface. Tasks such as reaching, grasping, placing, and drawer opening are specified via visual overlays (object masks, a 3D target arrow, and colored corner indicators for hand use and grasp/release timing). The policy maps proprioception and pixel observations through a CNN-GRU to a 48-dimensional residual latent that is decoded by the PULSE-X motion prior into joint targets. The authors report success rates on a tabletop GRAB benchmark and in procedurally generated kitchen scenes, with ablations over vision modality, image resolution, pretrained encoders, distillation, the look-at reward, and the use of the motion prior. They claim that active search and whole-body coordination emerge from training in diverse scenes, that the vision-driven policy generalizes to unseen objects and scenes, and that it compares favorably to a modified state-space Omnigrasp oracle. The appendix provides per-object success breakdowns, implementation details, and additional qualitative examples.

Significance. If the central claims hold, this is a meaningful step toward vision-driven whole-body dexterous control for simulated humanoids: it demonstrates that a single egocentric-vision policy can handle search, reach, grasp, transport, placement, and drawer opening without direct state features, and it evaluates on an external benchmark (GRAB) with train/test splits and per-object results. The procedural kitchen generation, the use of multiple vision modalities, and the explicit documentation of the reward design are strengths. The paper also provides several honest limitation statements in Section 6. However, the significance is tempered by three load-bearing concerns: the action space is restricted by an unquantified PULSE-X latent manifold, all quantitative comparisons are single-run, and the 'emergent' search behavior is directly shaped by the look-at reward. With additional analysis and a modest amount of experimental repetition, the contribution would be solid.

major comments (4)
  1. [3.3, Eq. (3) and Appendix C.2] The action interface is the PULSE-X latent space: every command is a 48-d residual latent added to a learned prior mean and decoded by D_PULSE-X. The paper never quantifies whether this manifold can express the contact-rich whole-body configurations required by the kitchen tasks, such as pulling a drawer with fingertips, leaning over a counter to reach a far object, two-hand transport, or precision-grasping thin objects. The per-object results in Table 5 are consistent with a latent bottleneck: knife (39.5%), toothbrush (42.0%), and binoculars (31.0%) are the worst-performing objects, while large/simple objects are much higher. Please add a coverage analysis of the PULSE-X latent space for task-relevant poses (for example, decoding latents for the GRAB pre-grasps used in the reward and measuring end-effector pose error), or an ablation with a direct action policy or a larger latent space, to show that the reported failures are not imposed by the motion prior.
  2. [Tables 1, 2, 3, 4, 5] All success rates are reported from a single run, without seeds, standard deviations, or confidence intervals. Headline differences such as 'stereo outperforms RGB by 9%' (abstract) and the PULSE-X versus no-PULSE-X comparison (Table 3, R4 vs R8, 71.4% vs 90.1%) are within the range where run-to-run variance in large-scale RL is typically material. At least for the main comparisons, the authors should report mean and standard deviation over multiple seeds, or provide bootstrap intervals over evaluation episodes, so that the quantitative claims are statistically grounded.
  3. [Section 5, Eq. (2)] The paper states that the look-at reward 'helps shape the search behavior for the kitchen scene,' and the reward in Eq. (2) explicitly rewards looking at the object and then at the marker. Early termination and the user-provided time scheduler (lambda_start, lambda_end) further shape when and how the agent acts. Calling the resulting head-scanning behavior 'emergent' therefore overstates what arises without reward shaping. Please either qualify the emergence claim, or provide an ablation in the kitchen scene without the look-at reward that measures whether search behavior (e.g., time-to-first-fixation or fraction of episodes where the object enters view) still appears.
  4. [Section 4, baselines and Appendix C.1] The only baseline is a state-space Omnigrasp policy that the authors modified to add hand specification, placement, phase variables, and the same reward. The paper does not demonstrate that these modifications preserve the original method's intended behavior, and this oracle is not evaluated in the kitchen setting at all. The conclusion in Section 6 that PDC 'outperforms the state-based benchmark' therefore rests on a single author-modified comparison. Please add an unmodified or independently implemented baseline in at least one environment, or temper the claim to explicitly state that the comparison is against a modified state-space policy as implemented here.
minor comments (6)
  1. [Abstract and Section 1] The phrase 'without privileged state information (e.g., 3D object positions and geometries)' is overstated: the visual interface overlays object masks, 3D arrows, and hand-state color squares that are rendered from simulator state. Please clarify that the method uses privileged information encoded as visual cues, rather than purely natural egocentric perception.
  2. [Section 4.1] Succ_search is defined as whether the object 'comes into view,' which conflates active head movement with the object being visible. Reporting head-orientation statistics or time-to-first-fixation would make the search claim more concrete.
  3. [Appendix C.3] Numerical values for the reward weights w_hpe, w_hre, lambda_start, lambda_end, and the distance thresholds (0.2m, 0.25m, 2s) are not provided. These are needed for reproducibility.
  4. [Table 2] The kitchen results table reports only the PDC-Stereo configuration; no baseline or ablation is shown for the kitchen setting, which limits the reader's ability to contextualize the success rates.
  5. [Section 3.3 and Eq. (3)] The notation '1 has-contact' and the indicator functions in the reward are hard to parse; please define them clearly, and distinguish the prior mean mu_p_t from the prior distribution itself.
  6. [Appendix C.3] There is a grammatical typo: 'we uses a two-layer CNN' should be 'we use a two-layer CNN.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the action prior is re-implemented and externally benchmarked, and the reported emergent behaviors are empirical with disclosed shaping rewards.

full rationale

The central derivation chain is self-contained and does not reduce to its inputs. PDC's policy outputs a 48-dimensional latent z_PDC which is added to the prior mean and decoded by D_PULSE-X into joint targets (Appendix Eq. 3); although PULSE-X is a self-cited prior controller, the paper re-implements it in Appendix C.2 ('First, we train a motion imitator, PHC-X [33], on the cleaned AMASS dataset... Then, we distill the PHC-X policy into the PULSE-X CVAE using DAgger') and validates the IsaacLab implementation on the external AMASS benchmark in Table 6, so the self-citation is independently supported rather than load-bearing. The perception-as-interface inputs (green object mask, blue marker, colored hand indicators) are hand-specified inputs, not predicted or fitted quantities, and no success metric is used to fit any parameter; evaluation uses external GRAB objects with separate train/test splits and an Omnigrasp state-space oracle. The 'emergent search' claim is made with the look-at reward explicitly disclosed as a shaping reward, and the paper states that search 'emerges when the object is not always initialized in the view' rather than claiming search appears without any reward. The one conceptual caveat, that human-like motion quality is partly inherited from the AMASS-trained low-level controller, is acknowledged in Section 1 ('we leverage control priors learned from large-scale motion capture data') and is a design choice, not a circular derivation. The PULSE-X latent-coverage question is a correctness or robustness risk (whether the latent manifold can express all required contact poses) but not a circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The main free parameters are the hand-tuned reward weights and phase schedule, which are not reported with values. The axioms are the expressiveness of PULSE-X, the validity of GRAB pre-grasps, the reliability of the simulator, and the sufficiency of the visual interface.

free parameters (3)
  • w_hpe, w_hre (pre-grasp reward weights) = not specified in paper
    Used in r_pre-grasp to weight hand position and rotation matching; hand-tuned, values omitted.
  • lambda_start, lambda_end (grasp/release schedule) = not specified in paper
    User-provided time scheduler that defines approach, grasp, and release phases; affects reward segmentation.
  • Distance thresholds (pre-grasp, goal, timeout) = 0.2 m, 0.25 m, 2 s
    Hand-chosen thresholds that trigger reward phases and early termination.
assumptions (4)
  • domain assumption PULSE-X latent space can represent the actions needed for these tasks
    Section 3.3: policy outputs latent z_t decoded by PULSE-X; no coverage or expressiveness analysis is provided.
  • domain assumption GRAB pre-grasp poses are valid for kitchen scenes and unseen objects
    Pre-grasps from MoCap are used in the reward for all objects; if a pre-grasp pose is inappropriate, the reward misleads the policy.
  • domain assumption Physics simulator (IsaacLab) provides reliable contact and dynamics for these tasks
    All results are in simulation; real-world transfer is not demonstrated and the paper does not validate simulation fidelity.
  • ad hoc to paper Visual cues (mask, 3D arrow, corner colors) are a sufficient task specification
    This is the core 'perception-as-interface' design that the paper introduces; its sufficiency is the main premise being tested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning." pith.science (2026). https://pith.science/paper/7NPNAG3G

@misc{pith2026250512278,
  author       = {Pith},
  title        = {Pith review of: Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7NPNAG3G}},
  note         = {Machine review of arXiv:2505.12278}
}
read the original abstract

Human behavior is fundamentally shaped by visual perception -- our ability to interact with the world depends on actively gathering relevant information and adapting our movements accordingly. Behaviors like searching for objects, reaching, and hand-eye coordination naturally emerge from the structure of our sensory system. Inspired by these principles, we introduce Perceptive Dexterous Control (PDC), a framework for vision-driven dexterous whole-body control with simulated humanoids. PDC operates solely on egocentric vision for task specification, enabling object search, target placement, and skill selection through visual cues, without relying on privileged state information (e.g., 3D object positions and geometries). This perception-as-interface paradigm enables learning a single policy to perform multiple household tasks, including reaching, grasping, placing, and articulated object manipulation. We also show that training from scratch with reinforcement learning can produce emergent behaviors such as active search. These results demonstrate how vision-driven control and complex tasks induce human-like behaviors and can serve as the key ingredients in closing the perception-action loop for animation, robotics, and embodied AI.

Figures

Figures reproduced from arXiv: 2505.12278 by the authors.

Figure 1
Figure 1. Perceptive Dexterous Control (PDC) enables a humanoid equipped with egocentric vision to search for, reach, grasp, and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Kitchens: Our agent is trained in parallel on a large set of (randomly) procedurally generated kitchens. Each generated kitchen is structurally and visually different. Objects are spawned in random locations on the counter, and the agent starts in a random position and orientation within the scene. This diversity encourages the agent to learn general behaviors, such as search, and robust interaction. mation (object … view at source ↗
Figure 3
Figure 3. Perception-as-Interface: PDC instructs the policy through visual signals. We overlay the object of interest using a green mask, use a 3D blue marker to indicate target location, and use colored 2D squares (top corners) to inform the agent which hand to use and when to grasp and release. 3.2. The Visual Perception Interface To tackle the task of object search, grasp, goal reaching, release, and drawer opening, prior … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: We use perception and proprioception as input to the network, processed by a simple CNN-GRU-MLP architecture. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Tabletop: PDC is instructed directly from visual signals. A visual (top left and/or right) indicates in purple whether the corresponding hand should be prepared for contact. Changing to dark-blue indicates that contact should be made. Instructing the agent to use both …
Figure 6
Figure 6. Figure 6: Kitchen task: Here, the agent must master multiple skills. First, the agent must search and find the objective. A target object is marked in green, whereas a target drawer handle in red. Once found, the agent must approach the target before it can interact with it. ple…
Figure 7
Figure 7. Figure 7: Tabletop: PDC is instructed directly from visual signals. A visual (top left and/or right) indicates in purple whether the corresponding hand should be prepared for contact. Changing to dark-blue indicates contact should be made. Instructing the agent to use both hands…
Figure 8
Figure 8. Figure 8: Emergent search in the kitchen task: By training the agent in diverse and complex scenes, it learns generalizable behaviors and avoids overfitting. We observe that behaviors such as searching emerge. When the object is not in view, the agent scans the counter top and t…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 48 canonical work pages

  1. [1]

    Policy and value transfer in lifelong reinforcement learning

    David Abel, Yuu Jinnai, Sophie Yue Guo, George Konidaris, and Michael Littman. Policy and value transfer in lifelong reinforcement learning. InInternational conference on ma- chine learning, pages 20–29. PMLR, 2018. 3

    work page 2018

  2. [2]

    Optimal control of markov decision processes with incomplete state estimation.J

    Karl J Astrom. Optimal control of markov decision processes with incomplete state estimation.J. Math. Anal. Applic., 10: 174–205, 1965. 4

    work page 1965

  3. [3]

    Pmp: Learning to physically inter- act with environments using part-wise motion priors

    Jinseok Bae, Jungdam Won, Donggeun Lim, Cheol-Hui Min, and Young Min Kim. Pmp: Learning to physically inter- act with environments using part-wise motion priors. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1– 10, 2023. 2, 5

    work page 2023

  4. [4]

    Physically plausible full-body hand-object interaction synthesis.International Conference on 3D Vision (3DV), 2024

    Jona Braun, Sammy Christen, Muhammed Kocabas, Emre Aksan, and Otmar Hilliges. Physically plausible full-body hand-object interaction synthesis.International Conference on 3D Vision (3DV), 2024. 2, 3, 14

    work page 2024

  5. [5]

    Visual dexterity: In-hand reorientation of novel and complex object shapes.Science Robotics, 8(84):eadc9244, 2023

    Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Ed- ward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes.Science Robotics, 8(84):eadc9244, 2023. 2

    work page 2023

  6. [6]

    Vividex: Learning vision-based dex- terous manipulation from human videos.arXiv preprint arXiv:2404.15709, 2024

    Zerui Chen, Shizhe Chen, Etienne Arlaud, Ivan Laptev, and Cordelia Schmid. Vividex: Learning vision-based dex- terous manipulation from human videos.arXiv preprint arXiv:2404.15709, 2024

    arXiv 2024

  7. [7]

    Open-television: Teleoperation with immersive active visual feedback.arXiv preprint arXiv:2407.01512,

    Xuxin Cheng, Jialong Li, Shiqi Yang, Ge Yang, and Xiao- long Wang. Open-television: Teleoperation with immersive active visual feedback.arXiv preprint arXiv:2407.01512,

    arXiv

  8. [8]

    Learn- ing dexterous manipulation from exemplar object trajectories and pre-grasps

    Sudeep Dasari, Abhinav Gupta, and Vikash Kumar. Learn- ing dexterous manipulation from exemplar object trajectories and pre-grasps. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3889–3896. IEEE,

    work page

Show all 73 references
  1. [9]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 8

    2009

  2. [10]

    An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint ar...

    2010 arXiv

  3. [11]

    Sigmoid- weighted linear units for neural network function ap- proximation in reinforcement learning.arXiv preprint arXiv:1702.03118, 2017

    Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid- weighted linear units for neural network function ap- proximation in reinforcement learning.arXiv preprint arXiv:1702.03118, 2017. 14

    2017 arXiv

  4. [12]

    scene synthesizer: A python library for procedural scene generation in robot manipulation.Journal of Open Source Software, 2024

    Clemens Eppner, Adithyavairavan Murali, Caelan Garrett, Rowland O’Flaherty, Tucker Hermans, Wei Yang, and Di- eter Fox. scene synthesizer: A python library for procedural scene generation in robot manipulation.Journal of Open Source Software, 2024. 3

    2024

  5. [13]

    Latent space policies for hierarchical rein- forcement learning

    Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical rein- forcement learning. InInternational Conference on Machine Learning, pages 1851–1860. PMLR, 2018. 3

    2018

  6. [14]

    Learning agile soccer skills for a bipedal robot with deep reinforcement learning.arXiv preprint arXiv:2304.13653, 2023

    Tuomas Haarnoja, Ben Moran, Guy Lever, Sandy H Huang, Dhruva Tirumala, Markus Wulfmeier, Jan Hump- lik, Saran Tunyasuvunakool, Noah Y Siegel, Roland Hafner, Michael Bloesch, Kristian Hartikainen, Arunkumar Byra- van, Leonard Hasenclever, Yuval Tassa, Fereshteh Sadeghi, Nathan ...

    2023 arXiv

  7. [15]

    Deep residual learning for image recognition.arXiv preprint arXiv:1512.03385, 2015

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.arXiv preprint arXiv:1512.03385, 2015. 5

    2015 arXiv

  8. [16]

    Omnih2o: Universal and dexterous human-to- humanoid whole-body teleoperation and learning

    Tairan He, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous human-to- humanoid whole-body teleoperation and learning. InarXiv,

  9. [17]

    The interface theory of perception.Psychonomic bulletin & re- view, 22:1480–1506, 2015

    Donald D Hoffman, Manish Singh, and Chetan Prakash. The interface theory of perception.Psychonomic bulletin & re- view, 22:1480–1506, 2015. 3, 4

    2015

  10. [18]

    Rekep: Spatio-temporal reasoning of rela- tional keypoint constraints for robotic manipulation.arXiv preprint arXiv:2409.01652, 2024

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of rela- tional keypoint constraints for robotic manipulation.arXiv preprint arXiv:2409.01652, 2024. 3

    2024 arXiv

  11. [19]

    Dash: Modularized human ma- nipulation simulation with vision and language for embodied ai

    Yifeng Jiang, Michelle Guo, Jiangshan Li, Ioannis Exarchos, Jiajun Wu, and C Karen Liu. Dash: Modularized human ma- nipulation simulation with vision and language for embodied ai. InProceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation, pages 1–12, 2021. 4

    2021

  12. [20]

    Auto-encoding varia- tional bayes.2nd International Conference on Learning Rep- resentations, ICLR 2014 - Conference Track Proceedings, pages 1–14, 2014

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes.2nd International Conference on Learning Rep- resentations, ICLR 2014 - Conference Track Proceedings, pages 1–14, 2014. 14

    2014

  13. [21]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InProceedings of the IEEE/CVF international confer- ence on computer vision, pages 4015–4026, 2023. 4

    2023

  14. [22]

    Siamese neural networks for one-shot image recognition

    Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, pages 1–30. Lille, 2015. 5

    2015

  15. [23]

    Object motion guided human motion synthesis.ACM Transactions on Graphics (TOG), 42(6):1–11, 2023

    Jiaman Li, Jiajun Wu, and C Karen Liu. Object motion guided human motion synthesis.ACM Transactions on Graphics (TOG), 42(6):1–11, 2023. 7

    2023

  16. [24]

    Twisting lids off with two hands

    Toru Lin, Zhao-Heng Yin, Haozhi Qi, Pieter Abbeel, and Jitendra Malik. Twisting lids off with two hands. arXiv:2403.02338, 2024. 2

    2024 arXiv

  17. [25]

    Learning visuotactile skills with two multifingered hands.arXiv preprint arXiv:2404.16823,

    Toru Lin, Yu Zhang, Qiyang Li, Haozhi Qi, Brent Yi, Sergey Levine, and Jitendra Malik. Learning visuotactile skills with two multifingered hands.arXiv preprint arXiv:2404.16823,

    arXiv

  18. [26]

    Sim-to-real reinforcement learning for vision- based dexterous manipulation on humanoids.arXiv preprint arXiv:2502.20396, 2025

    Toru Lin, Kartik Sachdev, Linxi Fan, Jitendra Malik, and Yuke Zhu. Sim-to-real reinforcement learning for vision- based dexterous manipulation on humanoids.arXiv preprint arXiv:2502.20396, 2025. 2 9

    2025 arXiv

  19. [27]

    Sim-to-real reinforcement learning for vision- based dexterous manipulation on humanoids, 2025

    Toru Lin, Kartik Sachdev, Linxi Fan, Jitendra Malik, and Yuke Zhu. Sim-to-real reinforcement learning for vision- based dexterous manipulation on humanoids, 2025. 5

    2025

  20. [28]

    Mobile-television: Predictive motion priors for humanoid whole-body control.arXiv preprint arXiv:2412.07773,

    Chenhao Lu, Xuxin Cheng, Jialong Li, Shiqi Yang, Mazeyu Ji, Chengjing Yuan, Ge Yang, Sha Yi, and Xiaolong Wang. Mobile-television: Predictive motion priors for humanoid whole-body control.arXiv preprint arXiv:2412.07773,

    arXiv

  21. [29]

    Dextrah-g: Pixels-to-action dexterous arm-hand grasping with geometric fabrics.arXiv preprint arXiv:2407.02274, 2024

    Tyler Ga Wei Lum, Martin Matak, Viktor Makoviychuk, Ankur Handa, Arthur Allshire, Tucker Hermans, Nathan D Ratliff, and Karl Van Wyk. Dextrah-g: Pixels-to-action dexterous arm-hand grasping with geometric fabrics.arXiv preprint arXiv:2407.02274, 2024. 2, 3

    2024 arXiv

  22. [30]

    Omnigrasp: Grasping diverse objects with simulated humanoids

    Zhengyi Luo, Jinkun Cao, Sammy Christen, Alexander Win- kler, Kris M Kitani, and Weipeng Xu. Omnigrasp: Grasping diverse objects with simulated humanoids. InThe Thirty- eighth Annual Conference on Neural Information Processing Systems. 2, 3, 5, 6, 7, 12, 13, 14

  23. [31]

    Dynamics-regulated kinematic policy for egocentric pose es- timation.NeurIPS, 34:25019–25032, 2021

    Zhengyi Luo, Ryo Hachiuma, Ye Yuan, and Kris Kitani. Dynamics-regulated kinematic policy for egocentric pose es- timation.NeurIPS, 34:25019–25032, 2021. 7

    2021

  24. [32]

    Universal hu- manoid motion representations for physics-based control

    Zhengyi Luo, Jinkun Cao, Josh Merel, Alexander Winkler, Jing Huang, Kris Kitani, and Weipeng Xu. Universal hu- manoid motion representations for physics-based control. arXiv preprint arXiv:2310.04582, 2023. 3, 5

    2023 arXiv

  25. [33]

    Winkler, Kris Ki- tani, and Weipeng Xu

    Zhengyi Luo, Jinkun Cao, Alexander W. Winkler, Kris Ki- tani, and Weipeng Xu. Perpetual humanoid control for real- time simulated avatars. InInternational Conference on Com- puter Vision (ICCV), 2023. 7, 13, 14

    2023

  26. [34]

    Amass: Archive of motion capture as surface shapes.Proceedings of the IEEE International Conference on Computer Vision, 2019-Octob: 5441–5450, 2019

    Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Ger- ard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes.Proceedings of the IEEE International Conference on Computer Vision, 2019-Octob: 5441–5450, 2019. 5

    2019

  27. [35]

    Isaac gym: High performance gpu-based physics sim- ulation for robot learning.arXiv preprint arXiv:2108.10470,

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics sim- ulation for robot learning.arXiv preprint arXiv:2108.10470,

    arXiv

  28. [36]

    Pre- serving and combining knowledge in robotic lifelong rein- forcement learning.Nature Machine Intelligence, pages 1– 14, 2025

    Yuan Meng, Zhenshan Bing, Xiangtong Yao, Kejia Chen, Kai Huang, Yang Gao, Fuchun Sun, and Alois Knoll. Pre- serving and combining knowledge in robotic lifelong rein- forcement learning.Nature Machine Intelligence, pages 1– 14, 2025. 3

    2025

  29. [37]

    Hierarchical visuomotor control of humanoids

    Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvu- nakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne. Hierarchical visuomotor control of humanoids. arXiv preprint arXiv:1811.09656, 2018. 3

    2018 arXiv

  30. [38]

    Catch and carry: Reusable neural controllers for vision-guided whole-body tasks.ACM Trans

    Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. Catch and carry: Reusable neural controllers for vision-guided whole-body tasks.ACM Trans. Graph., 39, 2020. 2, 4

    2020

  31. [39]

    Orbit: A unified simulation framework for interactive robot learning environments.IEEE Robotics and Automation Letters, 8(6): 3740–3747, 2023

    Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yun- rong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A unified simulation framework for interactive robot lear...

    2023

  32. [40]

    A dataset of relighted 3D interacting hands

    Gyeongsik Moon, Shunsuke Saito, Weipeng Xu, Rohan Joshi, Julia Buffalini, Harley Bellan, Nicholas Rosen, Jesse Richardson, Mize Mallorie, Philippe Bree, Tomas Simon, Bo Peng, Shubham Garg, Kevyn McPhail, and Takaaki Shira- tori. A dataset of relighted 3D interacting hands. InN...

    2023

  33. [41]

    Pivot: Iterative visual prompt- ing elicits actionable knowledge for vlms.arXiv preprint arXiv:2402.07872, 2024

    Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompt- ing elicits actionable knowledge for vlms.arXiv preprint arXiv:2402.07872, 2024. 3

    2024 arXiv

  34. [42]

    Sharing lifelong reinforcement learning knowledge via modulating masks

    Saptarshi Nath, Christos Peridis, Eseoghene Ben-Iwhiwhu, Xinran Liu, Shirin Dora, Cong Liu, Soheil Kolouri, and Andrea Soltoggio. Sharing lifelong reinforcement learning knowledge via modulating masks. InConference on Life- long Learning Agents, pages 936–960. PMLR, 2023. 3

    2023

  35. [43]

    MIT press, 2004

    Alva No ¨e.Action in perception. MIT press, 2004. 1

    2004

  36. [44]

    A sensorimotor account of vision and visual consciousness.Behavioral and brain sci- ences, 24(5):939–973, 2001

    J Kevin O’regan and Alva No ¨e. A sensorimotor account of vision and visual consciousness.Behavioral and brain sci- ences, 24(5):939–973, 2001. 1

    2001

  37. [45]

    Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A A Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image.Proceedings of the IEEE Computer Society Conference on Computer Vision and Pat- ter...

    2019

  38. [46]

    Deeploco: dynamic locomotion skills us- ing hierarchical deep reinforcement learning.ACM Trans

    Xue Bin Peng, Glen Berseth, Kangkang Yin, and Michiel Van De Panne. Deeploco: dynamic locomotion skills us- ing hierarchical deep reinforcement learning.ACM Trans. Graph., 36:1–13, 2017. 3

    2017

  39. [47]

    Deepmimic: Example-guided deep reinforce- ment learning of physics-based character skills.ACM Trans- actions On Graphics (TOG), 37(4):1–14, 2018

    Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example-guided deep reinforce- ment learning of physics-based character skills.ACM Trans- actions On Graphics (TOG), 37(4):1–14, 2018. 5

    2018

  40. [48]

    Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters.arXiv preprint arXiv:2205.01906, 2022

    Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters.arXiv preprint arXiv:2205.01906, 2022. 3

    2022 arXiv

  41. [49]

    Ef- ficient learning on point clouds with basis point sets

    Sergey Prokudin, Christoph Lassner, and Javier Romero. Ef- ficient learning on point clouds with basis point sets. InPro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 4332–4341, 2019. 7, 13

    2019

  42. [50]

    General in- hand object rotation with vision and touch

    Haozhi Qi, Brent Yi, Sudharshan Suresh, Mike Lambeta, Yi Ma, Roberto Calandra, and Jitendra Malik. General in- hand object rotation with vision and touch. InConference on Robot Learning, pages 2549–2564. PMLR, 2023. 2

    2023

  43. [51]

    Dexmv: Imitation learning for dexterous manipulation from human videos

    Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Rui- han Yang, Yang Fu, and Xiaolong Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, pages 570–587. Springer, 2022

    2022

  44. [52]

    Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system.arXiv preprint arXiv:2307.04577, 2023

    Yuzhe Qin, Wei Yang, Binghao Huang, Karl Van Wyk, Hao Su, Xiaolong Wang, Yu-Wei Chao, and Dieter 10 Fox. Anyteleop: A general vision-based dexterous robot arm-hand teleoperation system.arXiv preprint arXiv:2307.04577, 2023. 2

    2023 arXiv

  45. [53]

    A reduction of imitation learning and structured prediction to no-regret online learning.arXiv preprint arXiv:1011.0686,

    Stephane Ross, Geoffrey J Gordon, and J Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning.arXiv preprint arXiv:1011.0686,

    arXiv

  46. [54]

    Proximal policy optimization algo- rithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Rad- ford, and Oleg Klimov. Proximal policy optimization algo- rithms, 2017. 4, 6

    2017

  47. [55]

    Lifelong machine learning systems: Beyond learning algorithms

    Daniel L Silver, Qiang Yang, and Lianghao Li. Lifelong machine learning systems: Beyond learning algorithms. In AAAI Spring Symposium: Lifelong Machine Learning, 2013. 3

    2013

  48. [56]

    Hand-object interaction pretraining from videos.arXiv preprint arXiv:2409.08273, 2024

    Himanshu Gaurav Singh, Antonio Loquercio, Carmelo Sfer- razza, Jane Wu, Haozhi Qi, Pieter Abbeel, and Jitendra Ma- lik. Hand-object interaction pretraining from videos.arXiv preprint arXiv:2409.08273, 2024. 2

    2024 arXiv

  49. [57]

    Dextrah-rgb: Visuomotor policies to grasp anything with dexterous hands.arXiv preprint arXiv:2412.01791, 2024

    Ritvik Singh, Arthur Allshire, Ankur Handa, Nathan Ratliff, and Karl Van Wyk. Dextrah-rgb: Visuomotor policies to grasp anything with dexterous hands.arXiv preprint arXiv:2412.01791, 2024. 2, 3

    2024 arXiv

  50. [58]

    Be- tween mdps and semi-mdps: A framework for temporal ab- straction in reinforcement learning.Artificial intelligence, 112(1-2):181–211, 1999

    Richard S Sutton, Doina Precup, and Satinder Singh. Be- tween mdps and semi-mdps: A framework for temporal ab- straction in reinforcement learning.Artificial intelligence, 112(1-2):181–211, 1999. 3

    1999

  51. [59]

    Grab: A dataset of whole-body human grasp- ing of objects

    Omid Taheri, Nima Ghorbani, Michael J Black, and Dim- itrios Tzionas. Grab: A dataset of whole-body human grasp- ing of objects. InComputer Vision–ECCV 2020: 16th Eu- ropean Conference, Glasgow, UK, August 23–28, 2020, Pro- ceedings, Part IV 16, pages 581–600. Springer, 2020. 7, 13

    2020

  52. [60]

    Calm: Conditional adversarial latent models for directable virtual characters

    Chen Tessler, Israel Yoni Kasten, Israel Yunrong Guo, and Canada Nvidia. Calm: Conditional adversarial latent models for directable virtual characters. 3

  53. [61]

    A deep hierarchical approach to lifelong learning in minecraft

    Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. A deep hierarchical approach to lifelong learning in minecraft. InProceedings of the AAAI conference on artificial intelligence, 2017. 3

    2017

  54. [62]

    Learning robot soccer from egocentric vision with deep rein- forcement learning.arXiv preprint arXiv:2405.02425, 2024

    Dhruva Tirumala, Markus Wulfmeier, Ben Moran, Sandy Huang, Jan Humplik, Guy Lever, Tuomas Haarnoja, Leonard Hasenclever, Arunkumar Byravan, Nathan Batchelor, et al. Learning robot soccer from egocentric vision with deep rein- forcement learning.arXiv preprint arXiv:2405.02425,...

    2024 arXiv

  55. [63]

    Unidexgrasp++: Im- proving dexterous grasping policy learning via geometry- aware curriculum and iterative generalist-specialist learning

    Weikang Wan, Haoran Geng, Yun Liu, Zikang Shan, Yaodong Yang, Li Yi, and He Wang. Unidexgrasp++: Im- proving dexterous grasping policy learning via geometry- aware curriculum and iterative generalist-specialist learning. InProceedings of the IEEE/CVF International Conference o...

    2023

  56. [64]

    Physhoi: Physics-based imita- tion of dynamic human-object interaction.arXiv preprint arXiv:2312.04393, 2023

    Yinhuai Wang, Jing Lin, Ailing Zeng, Zhengyi Luo, Jian Zhang, and Lei Zhang. Physhoi: Physics-based imita- tion of dynamic human-object interaction.arXiv preprint arXiv:2312.04393, 2023. 2, 5, 6

    2023 arXiv

  57. [65]

    Con- trol strategies for physically simulated characters performing two-player competitive sports.ACM Trans

    Jungdam Won, Deepak Gopinath, and Jessica Hodgins. Con- trol strategies for physically simulated characters performing two-player competitive sports.ACM Trans. Graph., 40:1–11,

  58. [66]

    Physics-based character controllers using conditional vaes

    Jungdam Won, Deepak Gopinath, and Jessica Hodgins. Physics-based character controllers using conditional vaes. ACM Trans. Graph., 41:1–12, 2022. 3, 14

    2022

  59. [67]

    Hierarchical planning and control for box loco-manipulation.Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(3):1– 18, 2023

    Zhaoming Xie, Jonathan Tseng, Sebastian Starke, Michiel van de Panne, and C Karen Liu. Hierarchical planning and control for box loco-manipulation.Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(3):1– 18, 2023. 2

    2023

  60. [68]

    Dexterous ma- nipulation from images: Autonomous real-world rl via sub- step guidance

    Kelvin Xu, Zheyuan Hu, Ria Doshi, Aaron Rovinsky, Vikash Kumar, Abhishek Gupta, and Sergey Levine. Dexterous ma- nipulation from images: Autonomous real-world rl via sub- step guidance. In2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5938–5945. IEEE,

  61. [69]

    Flow as the cross-domain manipulation interface.arXiv preprint arXiv:2407.15208, 2024

    Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gor- don Wetzstein, Manuela Veloso, and Shuran Song. Flow as the cross-domain manipulation interface.arXiv preprint arXiv:2407.15208, 2024. 3

    2024 arXiv

  62. [70]

    Intermimic: Towards universal whole-body control for physics-based human-object interactions, 2025

    Sirui Xu, Hung Yu Ling, Yu-Xiong Wang, and Liang-Yan Gui. Intermimic: Towards universal whole-body control for physics-based human-object interactions, 2025. 2, 5, 6

    2025

  63. [71]

    Humanvla: Towards vision-language directed object re- arrangement by physical humanoid.Advances in Neural In- formation Processing Systems, 37:18633–18659, 2025

    Xinyu Xu, Yizheng Zhang, Yong-Lu Li, Lei Han, and Cewu Lu. Humanvla: Towards vision-language directed object re- arrangement by physical humanoid.Advances in Neural In- formation Processing Systems, 37:18633–18659, 2025. 2

    2025

  64. [72]

    Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy

    Yinzhen Xu, Weikang Wan, Jialiang Zhang, Haoran Liu, Zikang Shan, Hao Shen, Ruicheng Wang, Haoran Geng, Yijia Weng, Jiayi Chen, et al. Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. InProceedings of the I...

    2023

  65. [73]

    OakInk: A large-scale knowledge reposi- tory for understanding hand-object interaction

    Lixin Yang, Kailin Li, Xinyu Zhan, Fei Wu, Anran Xu, Liu Liu, and Cewu Lu. OakInk: A large-scale knowledge reposi- tory for understanding hand-object interaction. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 7 11 Appendix A . Introduction 12 B...

    2022

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.