{"id":"0eb81748-09e7-4f30-9288-26d4bfbcfcd0","arxiv_id":"2505.12278","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PDC trains a single egocentric-vision policy that lets a simulated humanoid search for, grasp, and place objects and open drawers without privileged state information.","lead":"A simulated humanoid learns to search, grasp, and place household objects using only its own camera view and body awareness, without knowing object positions in advance. The paper's 'perception-as-interface' design lets one policy handle several tasks by reading visual markers, and the authors report emergent head-scanning search behavior.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PULSE-X latent coverage is the load-bearing bottleneck: every action is decoded from a 48-d latent trained on AMASS, and the paper never checks whether required contact-rich poses are expressible; low per-object kitchen success on thin/large objects is consistent with this risk.","rationale":"The central claim is that a single egocentric-vision policy can perform dexterous whole-body manipulation without privileged state. Two conditions must hold: the visual observations must carry enough task information, and the action space must be able to express the required motions. The first condition is supported by the perception-as-interface design, which the paper is explicit about; the second is the one that is least supported. All actuation passes through the PULSE-X decoder (Eq. 3), so the reachable action set is the decoder's output manifold, not the full 51x3 joint-target space. This is an upper bound on the entire framework's dexterity. The paper's own per-object numbers (Table 5) show pronounced failures on precisely the objects that call for contact-rich or wide-spread hand poses (knife, toothbrush, binoculars), which is exactly what a latent-coverage bottleneck would look like. The R4/R8 comparison shows PULSE-X helps a fixed-budget PPO run, but it does not measure whether the manifold contains the required actions; a direct-action policy could in principle surpass it with more compute. Therefore the concern is not merely speculative: it is a testable expressiveness assumption that the paper neither states nor checks. I agree with the reader's identification of this assumption. Other weaknesses (no error bars, missing kitchen baseline, AR overlays carrying task information) are real but secondary: error bars affect confidence in exact numbers, not feasibility; overlays are an explicit design choice; and the kitchen baseline absence weakens comparison but not the absolute demonstration. If the proposed latent-reachability check shows large minimum errors for hard objects, the verdict should move toward REJECT or UNVERDICTED for the strong dexterity claim; if the check passes, the conditional can be upgraded. For now, UNCHANGED is appropriate.","tokens_in":19808,"tokens_out":9441,"duration_ms":107157,"concrete_test":"For each of the 140 GRAB test grasps and a set of proprioceptive states sampled from successful and failed PDC kitchen rollouts, solve z* = argmin_z ||D_PULSE-X(z + mu_p(o_p), o_p) - q_pre-grasp||^2 with a loss on hand translation and orientation, using gradient descent over the 48-d latent. Compare the minimum achievable error to the thresholds actually used in the pre-grasp reward (||p_hand - p_pre-grasp|| <= 0.2 m, orientation term e^{-100||Delta_theta||^2}) for hard objects (knife, toothbrush, binoculars, mug). If the optimal decoded hand pose is far outside these thresholds, PULSE-X cannot express the required action and the framework's ceiling is confirmed; if all pre-grasps are well-approximated, the concern is resolved and the low success rates should be attributed to vision or reward design.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing bottleneck is the PULSE-X action interface, not the vision stack. Every control command is a 48-d latent z_PDC that is added to the prior mean and decoded by D_PULSE-X into joint targets (Appendix Eq. 3). D_PULSE-X was trained to imitate AMASS motions via DAgger from PHC-X; AMASS is dominated by locomotion and everyday body motion, not by forceful, contact-rich hand-object interactions such as pulling a drawer handle with fingertips, leaning over a counter to reach a far object, or precision-grasping thin objects. If the decoder's output manifold cannot express these configurations, no visual policy can succeed, no matter how good the perception is. The paper provides no coverage analysis of this manifold for the task-relevant pre-grasps and contact poses. The internal evidence is consistent with a bottleneck: per-object kitchen success (Table 5) drops to 31.0% for binoculars, 39.5% for knife, and 42.0% for toothbrush, while large/simple objects are near 80%. Table 3 R4 vs R8 shows PULSE-X helps on average at one training budget, but that does not establish that the latent space contains all required actions; a direct-action policy could in principle overtake with more compute or better optimization. The central claim of whole-body dexterity therefore rests on an unquantified expressiveness assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Perceptive Dexterous Control (PDC), a reinforcement-learning framework for a simulated SMPL-X humanoid that uses egocentric vision as the sole task-specification interface. Tasks such as reaching, grasping, placing, and drawer opening are specified via visual overlays (object masks, a 3D target arrow, and colored corner indicators for hand use and grasp/release timing). The policy maps proprioception and pixel observations through a CNN-GRU to a 48-dimensional residual latent that is decoded by the PULSE-X motion prior into joint targets. The authors report success rates on a tabletop GRAB benchmark and in procedurally generated kitchen scenes, with ablations over vision modality, image resolution, pretrained encoders, distillation, the look-at reward, and the use of the motion prior. They claim that active search and whole-body coordination emerge from training in diverse scenes, that the vision-driven policy generalizes to unseen objects and scenes, and that it compares favorably to a modified state-space Omnigrasp oracle. The appendix provides per-object success breakdowns, implementation details, and additional qualitative examples.","tokens_in":20162,"tokens_out":5791,"duration_ms":62709,"significance":"If the central claims hold, this is a meaningful step toward vision-driven whole-body dexterous control for simulated humanoids: it demonstrates that a single egocentric-vision policy can handle search, reach, grasp, transport, placement, and drawer opening without direct state features, and it evaluates on an external benchmark (GRAB) with train/test splits and per-object results. The procedural kitchen generation, the use of multiple vision modalities, and the explicit documentation of the reward design are strengths. The paper also provides several honest limitation statements in Section 6. However, the significance is tempered by three load-bearing concerns: the action space is restricted by an unquantified PULSE-X latent manifold, all quantitative comparisons are single-run, and the 'emergent' search behavior is directly shaped by the look-at reward. With additional analysis and a modest amount of experimental repetition, the contribution would be solid.","major_comments":[{"comment":"The action interface is the PULSE-X latent space: every command is a 48-d residual latent added to a learned prior mean and decoded by D_PULSE-X. The paper never quantifies whether this manifold can express the contact-rich whole-body configurations required by the kitchen tasks, such as pulling a drawer with fingertips, leaning over a counter to reach a far object, two-hand transport, or precision-grasping thin objects. The per-object results in Table 5 are consistent with a latent bottleneck: knife (39.5%), toothbrush (42.0%), and binoculars (31.0%) are the worst-performing objects, while large/simple objects are much higher. Please add a coverage analysis of the PULSE-X latent space for task-relevant poses (for example, decoding latents for the GRAB pre-grasps used in the reward and measuring end-effector pose error), or an ablation with a direct action policy or a larger latent space, to show that the reported failures are not imposed by the motion prior.","section":"3.3, Eq. (3) and Appendix C.2"},{"comment":"All success rates are reported from a single run, without seeds, standard deviations, or confidence intervals. Headline differences such as 'stereo outperforms RGB by 9%' (abstract) and the PULSE-X versus no-PULSE-X comparison (Table 3, R4 vs R8, 71.4% vs 90.1%) are within the range where run-to-run variance in large-scale RL is typically material. At least for the main comparisons, the authors should report mean and standard deviation over multiple seeds, or provide bootstrap intervals over evaluation episodes, so that the quantitative claims are statistically grounded.","section":"Tables 1, 2, 3, 4, 5"},{"comment":"The paper states that the look-at reward 'helps shape the search behavior for the kitchen scene,' and the reward in Eq. (2) explicitly rewards looking at the object and then at the marker. Early termination and the user-provided time scheduler (lambda_start, lambda_end) further shape when and how the agent acts. Calling the resulting head-scanning behavior 'emergent' therefore overstates what arises without reward shaping. Please either qualify the emergence claim, or provide an ablation in the kitchen scene without the look-at reward that measures whether search behavior (e.g., time-to-first-fixation or fraction of episodes where the object enters view) still appears.","section":"Section 5, Eq. (2)"},{"comment":"The only baseline is a state-space Omnigrasp policy that the authors modified to add hand specification, placement, phase variables, and the same reward. The paper does not demonstrate that these modifications preserve the original method's intended behavior, and this oracle is not evaluated in the kitchen setting at all. The conclusion in Section 6 that PDC 'outperforms the state-based benchmark' therefore rests on a single author-modified comparison. Please add an unmodified or independently implemented baseline in at least one environment, or temper the claim to explicitly state that the comparison is against a modified state-space policy as implemented here.","section":"Section 4, baselines and Appendix C.1"}],"minor_comments":[{"comment":"The phrase 'without privileged state information (e.g., 3D object positions and geometries)' is overstated: the visual interface overlays object masks, 3D arrows, and hand-state color squares that are rendered from simulator state. Please clarify that the method uses privileged information encoded as visual cues, rather than purely natural egocentric perception.","section":"Abstract and Section 1"},{"comment":"Succ_search is defined as whether the object 'comes into view,' which conflates active head movement with the object being visible. Reporting head-orientation statistics or time-to-first-fixation would make the search claim more concrete.","section":"Section 4.1"},{"comment":"Numerical values for the reward weights w_hpe, w_hre, lambda_start, lambda_end, and the distance thresholds (0.2m, 0.25m, 2s) are not provided. These are needed for reproducibility.","section":"Appendix C.3"},{"comment":"The kitchen results table reports only the PDC-Stereo configuration; no baseline or ablation is shown for the kitchen setting, which limits the reader's ability to contextualize the success rates.","section":"Table 2"},{"comment":"The notation '1 has-contact' and the indicator functions in the reward are hard to parse; please define them clearly, and distinguish the prior mean mu_p_t from the prior distribution itself.","section":"Section 3.3 and Eq. (3)"},{"comment":"There is a grammatical typo: 'we uses a two-layer CNN' should be 'we use a two-layer CNN.'","section":"Appendix C.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's technical core is promising and the evaluation design is better than most in this area, but the three load-bearing issues—latent-space coverage, lack of multiple seeds, and the overstatement of emergence—should be addressed before publication. The author overlap with PULSE-X and Omnigrasp is not problematic per se, but the paper should make the dependency on PULSE-X more critical in the main text rather than treating it as a black box. I would encourage the editor to ask for a focused revision rather than a full rewrite."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a well-executed system paper that deserves a serious referee. It demonstrates something prior work didn't: a single egocentric-vision policy, trained with RL, that searches, grasps, places, and opens drawers in varied kitchen scenes without privileged state. That's real. The perception-as-interface idea — using masks, an AR-style arrow, and colored corner indicators for task specification — is the most interesting contribution and likely to be picked up by others.\n\nWhat I like: the evaluation is more honest than much of this subfield. They report per-object success on GRAB train/test and on held-out kitchen scenes. They ablate resolution, pretrained encoders, motion prior, and look-at reward. The observation that stereo beats RGBD is nice. The claim about emergent search is partly supported by video, and the authors are transparent that the look-at reward and object spawn distribution shape it. Open-sourcing is promised.\n\nWhere I'd push back: the load-bearing concern is the PULSE-X bottleneck. Every action is a 48-d latent perturbing a prior mean from an AMASS-trained CVAE. The paper never checks whether this manifold covers the precise, force-requiring hand-object poses needed for drawer pulling or thin-object grasps. The per-object kitchen results are consistent with that worry — binoculars at 31%, knife and toothbrush around 40%. That doesn't invalidate the paper, but the authors should either add a coverage analysis or a direct-action baseline. A second soft spot: there are no variance estimates or multiple seeds, and the only baseline is a state-space oracle they modified. A vision-based baseline would sharpen the comparison. Minor: some of the 'emergent' search is reward-shaped; they admit this.\n\nNone of this is fatal. The central claim — vision-only, whole-body dexterous humanoid control in varied scenes via a single policy — holds up as a capability demonstration. The limitations section is candid about re-grasping and vision failures.\n\nWho is this for? People working on humanoid loco-manipulation, simulation-to-real transfer, and perception-driven character control. It's not a breakthrough that reshapes the field, but it's a genuinely useful result with reproducible enough details to build on.\n\nI'd send it to peer review. The authors should be asked to address the latent-space coverage question and report variance; but the contribution itself deserves referee time.","headline":"Solid vision-driven humanoid control system with a genuinely new perception-as-interface task specification; the main open question is the PULSE-X latent action space's coverage of contact-rich dexterity.","tokens_in":20704,"tokens_out":2901,"would_cite":true,"duration_ms":26371,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A humanoid whose only interface is egocentric vision can search, grasp, carry, and place objects and open drawers — with no 3D object state — and the search emerges from training, not programming.","keywords":["egocentric vision","dexterous whole-body control","visual reinforcement learning","perception-as-interface","humanoid control","active search","household manipulation","simulated humanoids"],"falsifier":"Decode the PULSE-X latent space exhaustively and check, for every pre-grasp the tasks require (each recorded grasp pose and the drawer-handle grip), whether some latent reconstructs it within task tolerance — a few centimeters at the wrist and fingertips while the body remains balanced. Any required pose that no latent reproduces is a task the framework cannot express regardless of perception quality; a positive control would add those motions to the prior's training data and show the policy then succeeds.","tokens_in":19627,"feed_emoji":"🦾","tokens_out":20033,"duration_ms":184726,"temperature":0.7,"pith_summary":"The paper introduces Perceptive Dexterous Control (PDC), which trains a simulated full-body humanoid with articulated hands to act solely from egocentric camera images and proprioception. Tasks are specified exclusively through visual cues drawn into the image — a bright green overlay on the target object, red on drawer handles, a 3D arrow marking the destination, and colored corner blocks signalling which hand to use and when to grasp or release — replacing the phase variables and 3D goal coordinates that prior whole-body controllers rely on. The authors show that one policy trained with reinforcement learning, acting through a motion-capture-derived latent prior, can search for, grasp, transport, and release objects and open drawers in procedurally generated kitchens, and that a stereo-vision version generalizes to unseen objects and scenes at success rates comparable to or better than a state-based oracle that always knows the object's position and shape. The paper's central demonstration is that human-like active search — scanning left and right, turning around, sweeping the countertop — emerges from training under partial observability rather than being explicitly designed, which matters because it points toward humanoids that need no hidden ground-truth state to act and whose tasks can be changed by changing the picture.","feed_headline":"Vision alone teaches a humanoid to search, grasp, and open drawers","feed_subtitle":"One egocentric-vision policy handles reach, grasp, place, and drawers with no 3D object state; search emerges on its own.","key_machinery":"The framework is carried by three pieces. First, perception-as-interface: task instructions are rendered into the agent's own camera image (a green object mask from instance segmentation, a red drawer-handle overlay, a 3D arrow for the goal, and colored 2D corner squares encoding handedness and grasp/release timing), so changing the task means changing the picture, not the policy input space. Second, PULSE-X: a conditional VAE distilled from a motion imitator trained on a large motion-capture archive, which defines a 48-dimensional latent action space; the high-level policy predicts a latent $z^{\\text{PDC}}_t \\in \\mathbb{R}^{48}$, and the decoder $D_{\\text{PULSE-X}}$ produces joint targets conditioned on proprioception, keeping motions human-like while cutting the control dimensionality from 153 joint targets to 48. Third, a lightweight CNN-GRU-MLP that ingests $128\\times128$ RGB (or two $80\\times80$ stereo images through a Siamese encoder, or RGB-D) plus proprioception, with the recurrent GRU layer supplying memory for the partially observable search task. Training uses PPO with a staged dense reward whose phases (approach, pre-grasp, grasp, transport, release) are scheduled by user-provided times $\\lambda_{\\text{start}}, \\lambda_{\\text{end}}$ and reference pre-grasp poses, plus early termination, and the kitchen policy is warm-started from the tabletop policy.","core_discovery":"PDC's central claim is that perception can serve as the complete task interface for dexterous whole-body control: if the image tells the agent what to pick up, where to take it, which hand to use, and when to contact and release, the policy needs no privileged state. Concretely, the paper trains a single CNN-GRU-MLP policy with proximal policy optimization whose output is a latent code in the action space of PULSE-X, a conditional variational autoencoder trained on large-scale motion-capture data; the decoder turns the latent and current proprioception into joint targets, so every movement is filtered through a human-motion prior. Evaluated on household objects, the stereo-vision policy lifts and places 96.4% of training objects and 91.8% of held-out objects, while the state-based oracle that receives exact object shape and pose reaches 99.1% on training objects but drops to 70.6% on the same held-out set — an inversion the authors attribute to vision capturing what generalizes while privileged shape features overfit. In the kitchen environment the policy locates the green-marked object in 95–99% of episodes, grasps it in about 80–85%, completes the carry-and-release trajectory in roughly 54–66%, and opens red-marked drawers at about 64% success, in both training and unseen kitchens. Alongside these numbers, the authors report the emergence of active search and whole-body coordination — head scans, 360-degree turns, leaning on the counter — as behaviors induced by the vision-driven setup rather than by explicit search logic.","pith_inferences":["If perception-as-interface scales as its authors suggest, the same policy could accept any signal that can be drawn into the image — floating text instructions, a user's gaze, a cursor, even projected AR annotations — turning the humanoid into a general vision-conditioned motor system; the paper hints at floating text but does not test it.","The true expressive ceiling may be the motion prior, not the perception: since every action is a decoded PULSE-X latent, any hand-object interaction absent from the motion-capture archive (a precision pinch, an unusual grip) is unreachable no matter how clearly the vision system labels it, and the paper never measures the latent space's coverage of the interactions its tasks require.","The 'no privileged information' claim holds for the policy's observations only: the reward still uses exact 3D object positions, recorded pre-grasp poses, and a user-supplied grasp/release time schedule, so extending the framework to tasks without motion-capture equivalents would require replacing the staged reward rather than the perception interface.","The paper's own failure analysis — the agent does not attempt a second grasp after a miss, loses clarity when cabinets occlude the target, and shakes its head — points to the first testable improvements: a rewarded re-grasp policy and gaze stabilization, both compatible with the visual interface because the object mask persists in view."],"forward_implications":["A single policy body can be reused across tasks: the kitchen policy is warm-started from the tabletop policy and trained simultaneously for grasping and drawer-opening, so new skills can be added by fine-tuning without changing the architecture or the observation space.","Visual control generalizes to never-seen objects better than privileged control: on the held-out test objects, PDC's grasp success stays at 90–92% while the state-based oracle drops to 70.6%, implying that visual features transfer where canonical-shape encodings do not.","Stereo perception outperforms monocular RGB and RGB-D (96.4% vs 87.5% and 86.9% tabletop success), which the authors read as depth estimation emerging from the reaching and grasping task itself.","Training in randomized multi-object kitchens produces active search — scanning, turning, and countertop sweeps — that never appears in the simpler tabletop setting where the object is always in view.","Learning the visual encoder from scratch beats freezing a classification-pretrained encoder (90.1% vs 61.4–68.5% success), indicating that the representations that close the perception-action loop are task-specific, not generic classification features."],"supporting_citations":[{"why":"supplies the privileged-state oracle (Omnigrasp) PDC is compared against, as well as the humanoid setup and the reference pre-grasp pipeline for rewards.","marker":"[30]"},{"why":"defines PULSE-X, the motion-capture-derived latent action space through which every PDC action must pass.","marker":"[32]"},{"why":"provides the motion imitator from which the PULSE-X decoder is distilled, fixing the repertoire of expressible motions.","marker":"[33]"},{"why":"is the motion-capture archive that PULSE-X is trained to reproduce, the source of the human-motion prior.","marker":"[34]"},{"why":"is the prior vision-conditioned whole-body controller using phase variables and privileged object position, the design that perception-as-interface replaces.","marker":"[38]"},{"why":"is the proximal policy optimization algorithm used to train the controller from pixels.","marker":"[54]"},{"why":"supplies the household objects and the recorded whole-body pre-grasps used for reward design and for train/test evaluation.","marker":"[59]"},{"why":"provides the instance segmentation whose masks are alpha-blended onto the image to mark the target object.","marker":"[21]"},{"why":"procedurally generates the diverse randomized kitchens used for training and testing the multi-task policy.","marker":"[12]"}],"fun_headline_variants":["Egocentric vision alone yields dexterous humanoid control","One visual policy handles reach, grasp, place, and drawers","Single vision policy learns to search and grasp without 3D state","Vision-only training makes humanoid search and manipulate","No 3D state needed: vision-only humanoid performs tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every action passes through the latent space of PULSE-X, a prior trained on motion-capture data; if that latent space cannot express a motion the tasks demand — a finger curl inside a drawer handle, a lean the captures never contain — the whole framework fails no matter how well the vision works, and the paper does not measure that expressive coverage.","fun_headline_variants_meta":{"raw":{"variants":["Egocentric vision alone yields dexterous humanoid control","One visual policy handles reach, grasp, place, and drawers","Single vision policy learns to search and grasp without 3D state","Vision-only training makes humanoid search and manipulate","No 3D state needed: vision-only humanoid performs tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00059,"raw_usage":{"total_tokens":2827,"prompt_tokens":1064,"completion_tokens":1763,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":1679}},"tokens_in":680,"tokens_out":1763,"duration_ms":12955,"temperature":1.0,"reasoning_tokens":1679,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:36:34.483349+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Decode the PULSE-X latent space exhaustively and check, for every pre-grasp the tasks require (each recorded grasp pose and the drawer-handle grip), whether some latent reconstructs it within task tolerance — a few centimeters at the wrist and fingertips while the body remains balanced. Any required pose that no latent reproduces is a task the framework cannot express regardless of perception quality; a positive control would add those motions to the prior's training data and show the policy then succeeds.","supporting_citations":[{"cited_title":"Omnigrasp: Grasping diverse objects with simulated humanoids","cited_arxiv_id":null,"evidence_quote":"supplies the privileged-state oracle (Omnigrasp) PDC is compared against, as well as the humanoid setup and the reference pre-grasp pipeline for rewards."},{"cited_title":"Winkler, Kris Ki- tani, and Weipeng Xu","cited_arxiv_id":null,"evidence_quote":"provides the motion imitator from which the PULSE-X decoder is distilled, fixing the repertoire of expressible motions."},{"cited_title":"Amass: Archive of motion capture as surface shapes.Proceedings of the IEEE International Conference on Computer Vision, 2019-Octob: 5441–5450, 2019","cited_arxiv_id":null,"evidence_quote":"is the motion-capture archive that PULSE-X is trained to reproduce, the source of the human-motion prior."},{"cited_title":"Catch and carry: Reusable neural controllers for vision-guided whole-body tasks.ACM Trans","cited_arxiv_id":null,"evidence_quote":"is the prior vision-conditioned whole-body controller using phase variables and privileged object position, the design that perception-as-interface replaces."},{"cited_title":"Proximal policy optimization algo- rithms, 2017","cited_arxiv_id":null,"evidence_quote":"is the proximal policy optimization algorithm used to train the controller from pixels."},{"cited_title":"Grab: A dataset of whole-body human grasp- ing of objects","cited_arxiv_id":null,"evidence_quote":"supplies the household objects and the recorded whole-body pre-grasps used for reward design and for train/test evaluation."},{"cited_title":"scene synthesizer: A python library for procedural scene generation in robot manipulation.Journal of Open Source Software, 2024","cited_arxiv_id":null,"evidence_quote":"procedurally generates the diverse randomized kitchens used for training and testing the multi-task policy."}],"review_version":1}