{"id":"1c49c05c-c4ae-48b7-a8d6-96dc45801da0","arxiv_id":"2507.19817","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Ag2x2 learns coordination-aware, agent-agnostic visual representations from human videos and uses them as rewards to acquire bimanual manipulation skills in simulation, reaching 73.5% success across 13 tasks.","lead":"Ag2x2 trains a visual representation from everyday human cooking videos, keeping hand positions while removing the person, and uses that representation as a reward signal to teach two simulated robot arms 13 bimanual tasks without human demonstrations or hand-engineered rewards. The authors report a 73.5% average success rate, above their strongest baselines, but the small number of evaluation runs and modified benchmark environments leave the comparison open to question.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reward Eq. (2) is computed in an embedding that also receives the goal's desired end-effector coordinates, so the 16.2-point hand-channel gain may reflect direct coordinate matching, not a transferable coordination representation.","rationale":"The paper's central claim is that preserving hand positions in an agent-agnostic representation enables zero-shot bimanual skill acquisition without engineered rewards. The reward construction in Eq. (2) makes this claim hard to evaluate: the same coordinates used to encode current and goal states are also the target of the reward. Even a model that merely encodes the position tokens would receive a shaped reward for reaching the goal coordinates, independent of any human-video pretraining. The Ag2x2-H ablation removes the hand channel from both current and goal inputs, so it cannot separate representation quality from reward informativeness. I therefore read the 73.5% versus 57.3% comparison as not yet establishing the proposed mechanism. The suggested test isolates the reward contribution while leaving the current-state hand channel and all pretrained weights intact. If the method remains successful, the representation carries transferable coordination knowledge; if not, the headline gain is a reward-engineering artifact. This is an experimentally addressable confound, so a conditional verdict is appropriate rather than outright rejection. The reader's weakest assumption (human-to-robot correspondence) is adjacent but distinct; this concern applies even if the human-to-robot mapping is perfect.","tokens_in":12170,"tokens_out":8058,"duration_ms":94711,"concrete_test":"Re-run the 13 Table I tasks with only the goal's hand-coordinate channel ablated: at every step set the two goal hand positions equal to the current hand positions, keeping the goal image and all pretrained weights unchanged. If average success stays near 73.5%, the hand representation transfers; if it drops toward the 57.3% Ag2x2-H level, the Table I gain is explained by direct coordinate matching in Eq. (2) rather than by coordination-aware pretraining.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section III.B-C defines the visual encoder Fϕ over RGB plus the 2D coordinates of both hands, and the reward in Eq. (2) is a monotone function of S(z_t,z_g)=cosine(Fϕ(o_t), Fϕ(g)). The goal g explicitly contains '2D projections of desired end-effector positions in camera space' (Section III.C). Hence R increases whenever current proxy positions match the specified goal positions, because the position tokens enter both embeddings. The Ag2x2-H ablation (Section IV.C) removes hand positions from the reward, so the 16.2-point gap conflates two effects: (1) a transferable coordination representation learned from human videos, and (2) a task-specific end-effector goal-coordinate reward channel. The latter is an engineered per-task supervision signal, yet the abstract claims 'without human demonstrations or engineered rewards' and the title claims zero-shot. No procedure is given for obtaining the desired end-effector projections for a new task, so the claimed generality rests on an unspecified task-specific specification.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Ag2x2, a two-stage framework for bimanual manipulation. In the first stage, a ViT-Large encoder is fine-tuned with a time-contrastive objective on human egocentric videos from EpicKitchen in which the human body is inpainted, while the 2D positions of the two hands are preserved and injected as learned position tokens. In the second stage, a PPO policy is trained in IsaacGym with proxy-sphere agents and a tilted reward that measures embedding similarity between the current observation and a goal specification consisting of a goal image plus 2D projections of desired end-effector positions. The policy outputs proxy actions that are converted to joint commands for two Franka arms via inverse kinematics. The paper evaluates the method on 13 bimanual tasks from Bi-DexHands and PerAct2, reporting a 73.5% overall success rate, surpassing baselines (R3M, VIP, Ag2Manip, Eureka) and an expert-reward baseline, with an ablation (Ag2x2-H) that removes hand position information and drops to 57.3%. The paper also demonstrates that 12 generated trajectories can train an imitation policy that generalizes to novel rope configurations.","tokens_in":12322,"tokens_out":5558,"duration_ms":69171,"significance":"If the reported results are robust, the paper makes a useful contribution: it provides evidence that time-contrastive pretraining on human videos can transfer to bimanual robot control, that preserving hand-position cues helps coordination, and that autonomously generated demonstrations can seed imitation learning. The breadth of the evaluation (13 tasks, including deformable-object manipulation) and the auxiliary analyses (trajectory smoothness, task-progress consistency, and a proprioception control) are strengths. However, the central claims of zero-shot skill acquisition, absence of engineered rewards, and superiority over expert-designed rewards are weakened by the fact that the goal specification contains task-specific end-effector coordinates that enter the same embedding used by the reward, and by the absence of variance reporting in the headline results. These issues are addressable with additional controls and more detailed statistics, but they are load-bearing for the paper's main conclusions.","major_comments":[{"comment":"The reward is computed as a monotone function of embedding similarity S(z_t, z_g), where z_t = F_phi(o_t) and z_g = F_phi(g). According to Section III.B, F_phi takes as input the RGB image concatenated with 2D hand coordinates, and according to Section III.C the goal g explicitly contains \"2D projections of desired end-effector positions in camera space.\" Therefore, whenever the current observation includes current end-effector positions, the reward directly rewards matching those positions to the desired end-effector projections. This is a task-specific, engineered coordinate-matching signal, and it is not a consequence of the coordination-aware representation learned from human videos. The 16.2-point gain of Ag2x2 over Ag2x2-H in Table I therefore conflates two effects: (a) transferable coordination information learned from human videos, and (b) direct reward feedback on end-effector goal coordinates. To support the abstract's claim of learning \"without ... engineered rewards\" and to attribute the ablation gain correctly, the paper should provide a control in which hand-position tokens are present in the observation but the goal contains no hand coordinates, or remove desired end-effector projections from the goal entirely. It should also explain how the desired end-effector projections are obtained for a new task, since the current text gives no procedure.","section":"Section III.C, Eq. (2)"},{"comment":"Success rates are reported only as counts out of 9 per task, with no variance, confidence intervals, or per-seed breakdown. The headline comparison of 73.5% (Ag2x2) versus 63.2% (expert reward) and 56.4% (Ag2Manip) involves differences of roughly 10-17 successes across 117 trials, and several task-level differences are only 1-2 successes (e.g., task a: Ag2x2 7 vs. Ag2x2-H 7; task g: Ag2x2 6 vs. expert 5). Without repeated-seed statistics or a significance test, it is not possible to conclude that Ag2x2 reliably outperforms the expert-reward baseline or the hand-removed ablation. The paper should report mean and standard deviation over the three seeds, per-seed counts, and an appropriate statistical comparison, especially for the central claim that Ag2x2 surpasses expert-designed rewards.","section":"Table I, Section IV.B"},{"comment":"The PerAct2 tasks are ported to IsaacGym with assets converted from .ttm to .urdf/.dae, and when \"direct asset migration was not possible, we substituted comparable alternatives to maintain task integrity.\" Because both the assets and the simulation environment differ from the original PerAct2 benchmark, the statement that Ag2x2 achieves higher success rates than those reported in PerAct2 is not a direct comparison. The paper should specify which assets were substituted, quantify how the task geometry changed, and provide the exact success criteria and error margins used, so that the 13-task results are reproducible and comparable to the original benchmarks.","section":"Section IV.A"},{"comment":"The reward scaling parameter alpha is set to 3.0 \"based on ablation studies\" conducted on the same 13-task evaluation suite. This means the reward function contains at least one hyperparameter tuned on the benchmark used to demonstrate the \"no engineered rewards\" claim. The paper should report the ablation over alpha (including per-task success rates) and discuss the sensitivity of the main results to this choice. Without this, it is unclear whether the strong performance depends on a carefully selected constant that was chosen using the test tasks.","section":"Section III.D"}],"minor_comments":[{"comment":"The notation in the contrastive loss, particularly the subscript \"o‰c_l\", is confusing; please make explicit that l indexes a frame from a different video than i, j, and k, and define all symbols in one place.","section":"Eq. (1)"},{"comment":"The column headers are inconsistent: the \"Overall\" row appears above the per-task columns, and the meaning of \"Avg.\" is not uniform across tables. Clarify the table structure so that the reader can map each column to a task.","section":"Table I and Table IV"},{"comment":"The name \"PerAct 2\" is written inconsistently as \"PerAct2\" and \"PerAct 2\" in the text, captions, and references; please standardize.","section":"Throughout"},{"comment":"The Spearman rank correlation in Table III is described as measuring \"task progress consistency,\" but it actually measures correlation between frame index and visual similarity to the final goal image, which is a property of the reward landscape, not of the policy's causal progress. Please rephrase to avoid implying that the policy itself is being directly measured.","section":"Section IV.E"},{"comment":"The limitation about single static goal images is clearly stated and appreciated; consider also mentioning that the current framework requires the final end-effector positions to be specified, consistent with Major Comment 1, and discuss how those positions could be inferred automatically for truly zero-shot operation.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"The paper builds directly on the authors' prior Ag2Manip work, and the incremental novelty is appropriate for a workshop or conference contribution but should be more carefully positioned. The main risk is that the reported hand-channel gain may be an artifact of the goal containing desired end-effector coordinates that enter the reward embedding; a control experiment that removes those coordinates from the goal would either validate or invalidate the central claim. The lack of variance reporting is also concerning for a claim of state-of-the-art performance. The evaluation is otherwise broad and the additional analyses are useful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Ag2x2 is a sensible, incremental extension of Ag2Manip that adds hand-position tokens to an agent-agnostic visual representation and shows a real empirical gain on bimanual RL benchmarks. The central claim is probably in the right ballpark, but the paper's own evaluation conflates the learned representation with a task-specific coordinate-matching reward, and the write-up overstates the zero-shot and no-engineered-rewards framing.\n\nWhat's new and good: The idea of keeping hand positions while erasing the human is a clean response to Ag2Manip's limitation. The two-stage training (visual adaptation then hand-token integration) is straightforward. The 16.2-point drop when hand features are removed is a meaningful ablation. Thirteen tasks across two benchmarks, including a rope task, is a decent spread. The imitation learning section is a nice bonus that shows the learned policies generate usable demonstrations. The paper is clearly written and the related work is fair.\n\nWhere it gets soft: The reward in Eq. (2) is cosine similarity between embeddings of current state and goal, and the goal contains 2D projections of desired end-effector positions. Since the encoder also takes hand positions as input, the reward contains a direct matching term between current and goal hand coordinates. That is task-specific supervision, not a transferable coordination representation. The Ag2x2-H ablation removes hand tokens from both representation and reward, so the 16.2-point gap conflates two effects. The proprioception experiment (Table IV) does not fully resolve this because it adds proprioception to all models while the representation difference remains. This matters because the paper claims no human demonstrations or engineered rewards, but the goal spec includes end-effector targets, and no procedure is given for obtaining them for a new task. I'd call this an interpretation flaw, not a fatal one. The gains are probably real, but the mechanism is less clean than claimed. The stress-test note about this is on point.\n\nOther concerns are standard but real: 9 runs per task, no error bars; PerAct2 tasks ported to IsaacGym with substituted assets, so cross-benchmark comparison to PerAct2's reported numbers is not same-protocol; alpha=3.0 picked from ablations; no code or data release. The 'zero-shot' in the title is doing heavy lifting since you still need a per-task goal image and hand targets, plus RL training time per task.\n\nBottom line: The core idea is worth taking seriously, and the results are promising enough to deserve a proper review. It needs a rewrite to separate the coordinate-matching reward from the learned representation, add error bars, and run a same-protocol comparison. I'd send it out and expect major revisions. I'd also bring it to the reading group because the reward-gaming issue is a good discussion point.","headline":"A solid incremental extension with a real empirical win, but the reward channel is partly doing task-specific work that the 'no engineered rewards' framing understates.","tokens_in":12909,"tokens_out":4892,"would_cite":true,"duration_ms":49486,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that keeping the 2D positions of the hands, while erasing the human body, makes agent-agnostic visual representations work for zero-shot bimanual manipulation, reaching 73.5% success on 13 simulated tasks and beating…","keywords":["bimanual manipulation","agent-agnostic representation","visual reward shaping","time-contrastive learning","zero-shot skill acquisition","hand position tokens","deformable object manipulation","reinforcement learning"],"falsifier":"A direct test is to retrain Ag2x2 on the same videos while jittering the hand-position coordinates with Gaussian noise of increasing variance; if success stays near 73.5%, precise hand positions are not the active ingredient, while a collapse toward the 57.3% ablation level would confirm they are. A complementary test swaps the two simulated arms for arms with different kinematics or re-projects the goal end-effector positions through a different camera view and checks whether the 73.5% figure survives; both tests target the human-to-robot correspondence that the reward channel depends on.","tokens_in":11926,"feed_emoji":"🤖","tokens_out":6168,"duration_ms":66251,"temperature":0.7,"pith_summary":"Ag2x2's central claim is that agent-agnostic visual representations, which erase the human from demonstration videos so that skills transfer to robots, can be extended to bimanual manipulation if the 2D positions of the two hands are preserved as extra input tokens. The paper trains a time-contrastive encoder on inpainted egocentric cooking frames augmented with hand-position coordinates, then uses embedding similarity between the current frame and a goal image as a reward for two simulated robot arms. Across 13 tasks from Bi-DexHands and PerAct2, it reports a 73.5% average success rate, above its strongest zero-shot baseline and above a policy trained with expert-designed rewards. The ablation without hand positions drops to 57.3%, isolating the hand-position channel as the source of the coordination signal.","feed_headline":"Hand positions unlock zero-shot bimanual robot skills at 73.5%","feed_subtitle":"Erasing the human but keeping 2D hand positions beats expert rewards on 13 simulated tasks.","key_machinery":"The load-bearing object is the hand-aware agent-agnostic dataset: egocentric human video frames with the person segmented and inpainted away, augmented by a 4-channel signal giving the 2D position of each hand, computed as the mean of 21 projected hand keypoints, with learned variables for occluded or out-of-frame hands. A vision transformer encoder maps the concatenation of image and hand-position tokens into a shared latent space, trained with a time-contrastive loss so that temporally close states and goal states embed close together. During policy learning, embedding similarity to the goal image with a tilted reward shapes exploration, and two free-floating proxy spheres act as the end-effectors whose trajectories are converted to robot joint commands by inverse kinematics. The hand-position token is what lets the representation encode where each effector is without encoding what the human looks like, which the paper identifies as the missing ingredient for bimanual coordination.","core_discovery":"On the paper's own terms, Ag2x2 establishes that a coordination-aware but agent-agnostic representation—human bodies removed, but two 2D hand-position coordinates injected as learned tokens—suffices to acquire bimanual manipulation skills without demonstrations or engineered rewards. With goal images plus projected end-effector positions as the only task specification, reinforcement learning on proxy agents reaches a 73.5% average success rate on 13 diverse tasks, outperforming expert-designed reward functions (63.2%) and the previous agent-agnostic method (56.4%). The paper also reports that removing the hand channel causes a consistent 16.2 percentage point drop and that the learned policies can generate demonstration data: 12 trajectories of rope straightening suffice to train an imitation policy that generalizes to unseen rope configurations.","pith_inferences":["A natural next test, not run in the paper, is to corrupt the hand positions with increasing noise during pretraining; if performance is robust to jitter, the claim that precise positions carry coordination would need refinement, and if it collapses, the positional channel is doing the work.","The reward signal is defined in 2D camera space, so deploying Ag2x2 on a real robot would require projecting the robot's own end-effectors through a calibrated camera; the paper's simulated experiments do not test this correspondence under real camera noise or occlusion.","The time-contrastive embedding can be discontinuous for discrete state transitions (the paper notes a negative progress correlation on button pressing despite 100% success), so tasks with sudden visual changes may need a supplementary discrete-state signal.","Because only end-effector positions are retained, the same 'erase the agent, keep the positions' recipe could in principle extend to articulated fingers, though the paper explicitly leaves dexterous manipulation to future work."],"forward_implications":["Zero-shot bimanual skill acquisition becomes feasible from unlabeled human video alone, removing the two main scaling bottlenecks: expert demonstrations and task-specific reward engineering.","Hand-position information, not backbone choice or proprioception during policy learning, is the decisive factor: removing it costs 16.2 percentage points, while adding explicit proprioceptive rewards does not close the gap.","Deformable-object bimanual tasks such as rope straightening, typically hard for engineered rewards, become learnable from the same representation.","Ag2x2-generated rollouts are usable as training data: 12 trajectories produced an imitation policy that generalizes to unseen rope configurations, offering a route to scalable data collection.","On most PerAct2 tasks, the method exceeds the original benchmark's reported success despite not using the benchmark's expert waypoints, suggesting the embedding-similarity reward is a strong substitute for intermediate waypoint guidance."],"supporting_citations":[{"why":"introduces the agent-agnostic visual and action representation that Ag2x2 extends; also the strongest zero-shot baseline and the ablation's architectural reference.","marker":"[12]"},{"why":"provides the Bi-DexHands benchmark and six bimanual tasks used in evaluation.","marker":"[1]"},{"why":"provides the PerAct2 benchmark and seven bimanual tasks, plus the reported success rates Ag2x2 compares against.","marker":"[13]"},{"why":"supplies the egocentric household videos from which the hand-aware agent-agnostic training dataset is built.","marker":"[33]"},{"why":"supplies the hand keypoints whose projected mean becomes the 2D hand-position channel.","marker":"[34]"},{"why":"performs the segmentation used to erase the human from video frames.","marker":"[36]"},{"why":"performs the video inpainting that fills erased human regions to create agent-agnostic frames.","marker":"[37]"},{"why":"defines feasible graspable poses for the proxy agents, determining when a grasp is valid.","marker":"[38]"},{"why":"provides the PPO algorithm used to train the bimanual policies.","marker":"[39]"}],"fun_headline_variants":["Hand cues boost zero-shot bimanual skills to 73.5%","Agent-agnostic bimanual AI hits 73.5% with hand positions","Zero-shot bimanual: hand coordinates beat engineered rewards","With hand positions, zero-shot bimanual reaches 73.5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that 2D hand positions extracted from egocentric human cooking videos, projected through estimated camera poses, serve as a meaningful reward signal for coordinating two simulated robot arms whose body and kinematics are completely different from a human's.","fun_headline_variants_meta":{"raw":{"variants":["Hand cues boost zero-shot bimanual skills to 73.5%","Agent-agnostic bimanual AI hits 73.5% with hand positions","Zero-shot bimanual: hand coordinates beat engineered rewards","With hand positions, zero-shot bimanual reaches 73.5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000797,"raw_usage":{"total_tokens":3496,"prompt_tokens":925,"completion_tokens":2571,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":2490}},"tokens_in":541,"tokens_out":2571,"duration_ms":19425,"temperature":1.0,"reasoning_tokens":2490,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:58:52.368518+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to retrain Ag2x2 on the same videos while jittering the hand-position coordinates with Gaussian noise of increasing variance; if success stays near 73.5%, precise hand positions are not the active ingredient, while a collapse toward the 57.3% ablation level would confirm they are. A complementary test swaps the two simulated arms for arms with different kinematics or re-projects the goal end-effector positions through a different camera view and checks whether the 73.5% figure survives; both tests target the human-to-robot correspondence that the reward channel depends on.","supporting_citations":[{"cited_title":"Ag2manip: Learning novel manipulation skills with agent- agnostic visual and action representations,","cited_arxiv_id":null,"evidence_quote":"introduces the agent-agnostic visual and action representation that Ag2x2 extends; also the strongest zero-shot baseline and the ablation's architectural reference."},{"cited_title":"Bi-dexhands: Towards human-level bimanual dexterous manipulation,","cited_arxiv_id":null,"evidence_quote":"provides the Bi-DexHands benchmark and six bimanual tasks used in evaluation."},{"cited_title":"Peract2: Benchmarking and learning for robotic bimanual manipulation tasks,","cited_arxiv_id":null,"evidence_quote":"provides the PerAct2 benchmark and seven bimanual tasks, plus the reported success rates Ag2x2 compares against."},{"cited_title":"The EPIC-KITCHENS dataset: Collection, challenges and baselines,","cited_arxiv_id":null,"evidence_quote":"supplies the egocentric household videos from which the hand-aware agent-agnostic training dataset is built."},{"cited_title":"Reconstructing hands in 3d with transformers,","cited_arxiv_id":null,"evidence_quote":"supplies the hand keypoints whose projected mean becomes the 2D hand-position channel."},{"cited_title":"Open-vocabulary panoptic segmentation with text-to-image diffusion models,","cited_arxiv_id":null,"evidence_quote":"performs the segmentation used to erase the human from video frames."},{"cited_title":"Towards an end- to-end framework for flow-guided video inpainting,","cited_arxiv_id":null,"evidence_quote":"performs the video inpainting that fills erased human regions to create agent-agnostic frames."},{"cited_title":"Graspnet-1billion: A large-scale benchmark for general object grasping,","cited_arxiv_id":null,"evidence_quote":"defines feasible graspable poses for the proxy agents, determining when a grasp is valid."}],"review_version":1}