{"id":"3d2b457e-2d1d-4fdc-8178-280cbc1a8dde","arxiv_id":"2502.09886","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Video2Policy automatically builds simulated manipulation tasks from internet videos and trains RL policies using LLM-generated rewards, reaching 88% average simulation success and 47% sim2real success.","lead":"This paper presents a pipeline that converts internet videos of human hands manipulating objects into simulated robot tasks, then uses GPT-4o to write reward functions and reinforcement learning to train policies. The authors report 88% average success across 12 simulated tasks and a 47% success rate when a trained lifting policy is transferred to a real robot.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unquantified selection in the video-to-task pipeline: with a 42% average module failure rate (Table 4) and only 12 curated tasks reported, the 88% average success may not establish an automated data engine.","rationale":"The paper's strongest claim has two parts: (1) Video2Policy policies outperform LLM-only baselines, and (2) internet videos can serve as an automated data engine. The first part is supported by Table 1, but its scope is modest (12 tasks, 10 episodes per task, high variance on several tasks). The second, more consequential part is where the load-bearing weakness lies. The authors themselves report a 42% average module failure rate in Table 4, with the object detector failing on 60% of SSv2 videos. This makes the absence of end-to-end yield statistics critical: we do not know how many videos were fed into the pipeline to obtain the 100 reconstructed scenes or the 12 evaluated tasks. If the displayed results are survivors of heavy filtering and manual intervention, the headline 'automated data engine' is not established. The paper does provide useful evidence: it reports failure rates rather than hiding them, ablations show the value of visual information and reward iteration, and the sim2real experiment, though small, is a genuine attempt to close the loop. The concern is therefore not internal inconsistency but a missing measurement of end-to-end yield. That measurement is directly testable by re-running on a random sample and reporting per-stage pass/fail counts. Until that is done, the conditional verdict is appropriate; no change to the reader's verdict is needed.","tokens_in":19454,"tokens_out":4824,"duration_ms":51723,"concrete_test":"Run the complete Video2Policy pipeline end-to-end on a pre-registered random sample of N=100 SSv2 videos not used in the paper, with no manual frame selection, no manual object-name provision, and no filtering on policy success. Record per-stage pass rates (Grounding DINO, SAM-2, InstantMesh, FoundationPose, GPT-4o code runnability) and the distribution of final policy success under a pre-specified success criterion written by an independent evaluator before training. If the number of videos producing a policy above 50% success is below, say, 20, or if the average success computed over all attempted videos (treating pipeline failures as 0) is far below 88%, then the automated-data-engine claim is not supported. Publish the attempted-video list and per-stage failure counts to rule out selection bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that internet RGB videos can serve as an automated data engine for training generalist policies in simulation. For that to hold, the full chain object detection -> segmentation -> mesh reconstruction -> 6D tracking -> VLM code generation must convert a useful fraction of random videos into runnable simulation tasks with solvable reward structures. The paper's own robustness analysis (Appendix A.2, Table 4) reports a 42% average failure rate across these modules, with Grounding DINO failing on 60% of SSv2 videos and FoundationPose failing on 55%. Yet Table 1 reports results for only 9 SSv2 tasks plus 3 self-recorded videos, and the paper never states how many source videos were attempted, how many were discarded at each stage, or how the 12 tasks were selected. The phrase 'we reconstruct over 100 videos on 9 different tasks' does not say whether those 100 reconstructions were the successful survivors of a larger pool. If the reported tasks are a favorable subset, the 88% average success is a conditional statement about curated instances, not evidence for scalable automation. Manual interventions (choosing first vs. last frame for mesh reconstruction, providing object names for self-recorded videos) further weaken the 'automated' claim. The evaluation also uses manually written ground-truth success functions without independent verification, so reported success rates may be sensitive to leniency in the success criteria.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Video2Policy, a pipeline that turns internet RGB videos into simulation tasks for robot policy learning. The pipeline reconstructs object meshes and 6D poses from videos, uses GPT-4o to generate scene, reset, success, observation, and reward code, then trains RL policies with iterative in-context reward reflection. The authors report an average 88% success rate over 12 tasks (9 from SSv2, 3 self-recorded), a 75% success rate for a behavior-cloned generalist policy on 10 unseen lifting videos, and 47% success in a real-robot deployment. The central claim is that internet videos can serve as an automated data engine for generating diverse, grounded manipulation tasks in simulation.","tokens_in":19646,"tokens_out":4624,"duration_ms":49189,"significance":"If the central claim held, the work would be a meaningful step toward scalable, video-grounded simulation data for robotics, combining off-the-shelf vision models with LLM-generated task and reward code. The paper deserves credit for reporting a component-wise robustness analysis (Appendix A.2, Table 4), ablating each stage of code generation and reward reflection, checking that baselines are not disadvantaged by reset functions (Table 9), and attempting a real-robot transfer. However, as presented, the evidence is not strong enough to establish the 'automated data engine' claim: the video-to-task funnel is unquantified, per-task evaluation is based on only 10 episodes with high variance, success functions are hand-written without independent validation, and manual interventions remain in the pipeline. The relative comparisons against LLM-only baselines are suggestive, but the absolute performance claims and the scalability inference require substantially more evidence.","major_comments":[{"comment":"The central 'automated data engine' claim is not supported by an end-to-end yield analysis. Table 4 reports an average module failure rate of 42%, with Grounding DINO failing on 60% of SSv2 videos and FoundationPose on 55%, yet the paper never states how many source videos were attempted, how many survived each pipeline stage, or how the 9 SSv2 tasks and 3 self-recorded tasks in Table 1 were selected. Without this funnel information, the 88% average success rate is a conditional statement about a potentially curated subset rather than evidence that a useful fraction of arbitrary internet videos can be converted into runnable, solvable simulation tasks.","section":"Appendix A.2, Table 4; Section 3.1"},{"comment":"The evaluation protocol is too weak to support the headline quantitative claims. Success rates are averaged over only 10 evaluation episodes per task with three seeds, and several entries have large standard deviations (e.g., 'Push sth. with sth.' 0.43 ± 0.40, 'Throw Garlic into Bowl.' 0.70 ± 0.36, 'Tip sth. over' Eureka 0.67 ± 0.47). With 10 episodes, a single episode changes the success rate by 10%, so the reported differences between Video2Policy and Eureka (0.88 vs. 0.71) are not statistically established, and the near-zero task 'Cover sth. with sth.' (0.07 ± 0.05) indicates that some generated tasks are essentially unsolved and should be discussed separately rather than averaged into the headline number.","section":"Section 4, Experimental Setup; Table 1"},{"comment":"The absolute success rates rest on hand-written success functions that are not independently validated. The paper states that 'we write the ground-truth success function for each generated task' (Section 4), and the only validation in Fig. 10 is a correlation between GPT-4o-generated success functions and these same hand-written ones. Since the manual success functions define the evaluation itself, the reported absolute success rates may be sensitive to the leniency or strictness of the authors' criteria; only the relative comparison across methods, which share the same success functions, is protected.","section":"Section 4, Evaluation metric; Appendix A.3"},{"comment":"The pipeline is not fully automated as claimed. Object names and captions for the three self-recorded videos are provided manually, and the choice of first versus last frame for mesh reconstruction is made manually for occluded SSv2 objects. The manuscript should quantify the amount and nature of human intervention per task, and should either remove the 'automated' wording or report an automation rate that excludes these manual steps.","section":"Section 3.1; Section 4, Scene Generation"}],"minor_comments":[{"comment":"The simulator name is misspelled as 'Issac Gym'; it should be 'Isaac Gym'.","section":"Section 4, Experimental Setup"},{"comment":"The phrase 'generalist police' should be 'generalist policies'.","section":"Section 1, Introduction"},{"comment":"The coefficient description reads 'correlation coefficient qual to 0.83'; this should be 'equal to 0.83'.","section":"Appendix A.3, Fig. 10 caption"},{"comment":"The sentence 'as Dimage and Dimage are calculated within the same camera coordinate system' appears to contain a typo; the second quantity should likely be Dmesh.","section":"Section 3.1, Size estimation"},{"comment":"Table 6's caption repeats the D1-distance description from Table 5, but Table 6 reports predicted versus ground-truth object sizes; the caption should be corrected accordingly.","section":"Appendix A.2, Tables 5 and 6"},{"comment":"No code, data, or pre-trained models are released, and the paper does not state an availability plan; given the complexity of the pipeline, this limits reproducibility and should be addressed.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The conditional verdict in the reader's report is appropriate. The main gap is the unquantified video-to-task selection funnel combined with small evaluation episodes; these are fixable by additional experiments and reporting, so major revision is the right call rather than rejection. I would also encourage the editor to request data/code release as a condition of the revision, since the pipeline has many components and the current experiments cannot be independently reproduced."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the core idea is genuinely new—using monocular internet video to extract meshes, sizes, and 6D poses as visual grounding for LLM-generated task and reward code—and the paper is refreshingly honest about its pipeline's failure rates. But the headline \"88% average success\" is a statement about 12 curated tasks, not an automated data engine, because the paper never reports how many videos it started with or how many were discarded at each stage.\n\nWhat the paper does well: it makes a clear advance over text-only task generation (Gensim, RoboGen, Eureka). The ablations show visual information matters: removing it drops average success from 87% to 75%, and removing success-function picking drops it to 57%. The appendix includes a robustness table (Table 4) with measured per-module failure rates—DINO 60%, FoundationPose 55%, average 42%—which is more transparency than this line of work usually offers. The sim2real transfer, though small (3 objects, 47% success), is a genuine external prediction because the evaluation success functions are hand-written and separate from the learned rewards. So circularity is not the problem here.\n\nThe soft spot is selection. Twelve tasks are reported, but the paper doesn't say how many source videos were attempted, how many reconstructions failed, or how the final set was chosen. With a 42% average module failure rate, the reported tasks could be the survivors of a much larger pool. Manual interventions—choosing the first vs last frame for mesh reconstruction, feeding object names for self-recorded videos—also stretch the word \"automated.\" The evaluation uses manually written success functions; the reported correlation (0.83) between generated and ground-truth success functions is useful, but it doesn't validate the ground truth itself. Ten evaluation episodes per task, with tasks like \"cover something\" at 7% and \"throw garlic\" at 70±36%, means the 88% average is fragile. No code or data released, which doesn't help.\n\nNone of this kills the idea. The method is plausible, the writing is clear, and the limitations are acknowledged. The paper is for researchers working on automated task generation and sim2real policy learning. It deserves a serious referee, but the referee should ask for the full selection funnel, more episodes, and released artifacts. The \"automated data engine\" claim needs to be scaled down to \"pipeline that can generate tasks from curated videos.\"","headline":"Genuinely novel pipeline that uses internet video-derived geometry and 6D poses to ground LLM-generated robot tasks, with honest reporting of module failures, but the 88% average success rests on a curated 12-task subset with no disclosed selection funnel.","tokens_in":20234,"tokens_out":2670,"would_cite":true,"duration_ms":24332,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By reconstructing internet videos into simulation tasks, Video2Policy trains policies that reach 88% average success, beating LLM-only baselines and transferring a generalist lifting policy to a real robot.","keywords":["simulation","internet videos","reinforcement learning","task generation","reward design","vision-language model","robot manipulation","sim2real"],"falsifier":"Randomly sample 100 SSv2 videos, run the full Video2Policy pipeline end to end, and evaluate the produced policies with hand-written ground-truth success functions; if the fraction of videos yielding a successful policy is far below the 88% average (given the 42% per-module failure rate), the claim that internet videos can scale as a data engine fails.","tokens_in":19201,"feed_emoji":"🎥","tokens_out":11894,"duration_ms":96700,"temperature":0.7,"pith_summary":"Internet RGB videos of everyday human manipulation can serve as an automated data engine for training robot policies in simulation. The paper's Video2Policy pipeline first turns each video into a simulated tabletop scene, reconstructing object meshes, their real-world sizes, and their 6D pose tracks, then has a vision-language model (GPT-4o) write the full task code: reset, success, observation, and reward functions. Policies are trained with PPO, and the reward function is refined iteratively from training logs, an in-context reward-reflection loop. On 12 reconstructed tasks the learned policies reach 88% average success, beating Code-as-Policy, RoboGen, and Eureka baselines, and the pipeline can collect expert trajectories to train a generalist behavior-cloned policy that transfers to a real robot.","feed_headline":"Internet videos yield simulated robot policies that hit 88% success","feed_subtitle":"Everyday clips become simulated tasks, then a generalist lifting policy transfers to a real robot.","key_machinery":"The central mechanism is the task JSON, a compact description of each video that bundles the reconstructed object meshes, their scaled sizes, 6D pose tracks, and the video caption. A vision-language model (GPT-4o) reads this JSON and writes executable task code covering six parts: scene information, reset function, success function, observation function, observation-space function, and reward function. The reward is then refined through in-context reward reflection: PPO trains $N = 8$ candidate rewards per round, training logs and per-component analyses are fed back to the VLM, and the best candidate is selected for the next round; the training reward is the sum of the learned reward and the generated success signal weighted by $\\lambda = 100$.","core_discovery":"The paper claims that internet videos can be a scalable source of grounded, task-level diversity for robotic simulation, unlike text-only LLM generation that hallucinates uninteresting tasks. Video2Policy reconstructs the manipulated objects and their relative motion from each video into a task JSON, and the VLM generates executable task code including a success function informed by visual priors, not just text captions. On 12 tasks from SSv2 and self-recorded videos, the method reaches 88% average success, compared with 71% for Eureka, 45% for RoboGen, and 34% for Code-as-Policy. The authors further show that 100 reconstructed lifting tasks can train an image-based behavior-cloned policy that generalizes to 10 unseen objects (75% success) and succeeds on 47% of real robot trials after sim-to-real transfer.","pith_inferences":["Because the paper's own robustness analysis reports an average 42% vision-module failure rate, the end-to-end yield on arbitrary internet video is likely lower than the 88% policy-level success; practical deployment would require cheap filtering of reconstruction failures before training.","The success-function correlation result (0.83 with visual information vs 0.56 without) suggests the pipeline's ceiling is tied to perception quality, so swapping in stronger detection, mesh, or tracking models should raise both reconstruction yield and policy success.","The framework could be tested on behaviors without a clear final success state (for example, continuous or two-agent interactions) by having the VLM infer intermediate success criteria from the trajectory, an extension the paper does not address.","Since the generalist policy was evaluated only on lifting, a natural next test is whether the same data engine scales to multiple behaviors trained in one policy from diverse video collections."],"forward_implications":["If internet RGB videos are a reliable task source, robotic training data can scale with the internet rather than teleoperation or hand-designed benchmarks.","The ablations show the iterative reward-reflection step is the largest contributor, with average success dropping from 87% to 48% when it is removed.","A general policy lifted from 100 reconstructed lifting tasks generalizes to 10 unseen objects at 75% success, including objects from unseen categories.","The learned generalist policy transfers to a real robot with 47% success when using segmentation-mask input and domain randomization.","Success on unseen tasks rises from 13% to 75% as the number of training tasks grows from 10 to 100, indicating the data engine scales with video supply."],"supporting_citations":[{"why":"Supplies the SSv2 dataset whose captioned human manipulation videos are the input task source.","marker":"(Goyal et al., 2017)"},{"why":"Grounding DINO detects the manipulated objects in the first frame using text prompts.","marker":"(Liu et al., 2023)"},{"why":"SAM-2 segments each detected object across frames to produce masks for reconstruction and tracking.","marker":"(Ravi et al., 2024)"},{"why":"InstantMesh reconstructs 3D meshes from a single video frame.","marker":"(Xu et al., 2024)"},{"why":"UniDepth estimates camera intrinsics and metric depth used to scale meshes to real object sizes.","marker":"(Piccinelli et al., 2024)"},{"why":"FoundationPose tracks 6D object poses from mesh and depth inputs.","marker":"(Wen et al., 2024)"},{"why":"The GPT-4 family (including GPT-4o) generates the task code, success function, and reward function from the video-derived JSON.","marker":"(Achiam et al., 2023)"},{"why":"Eureka contributes the iterative in-context reward-reflection mechanism and serves as the closest baseline without video-derived success functions.","marker":"(Ma et al., 2023)"},{"why":"RoboGen is an LLM-only baseline that Video2Policy outperforms, especially on multi-object and dynamic tasks.","marker":"(Wang et al., 2023c)"},{"why":"Code-as-Policy is the LLM-as-policy baseline that fails on dynamic and multi-object tasks.","marker":"(Liang et al., 2023)"}],"fun_headline_variants":["Internet videos power robot policies to 88% success","Video2Policy turns clips into simulated robot tasks","From human videos to robot skills: 88% success rate","Watch and learn: internet videos train robot policies","Scaling robot learning with everyday internet videos"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach rests on the video-reconstruction stage supplying an accurate task JSON (object meshes, sizes, and 6D pose tracks) that the VLM then uses to write the task code; the paper's own robustness analysis reports a 42% average module failure rate, so this premise is fragile.","fun_headline_variants_meta":{"raw":{"variants":["Internet videos power robot policies to 88% success","Video2Policy turns clips into simulated robot tasks","From human videos to robot skills: 88% success rate","Watch and learn: internet videos train robot policies","Scaling robot learning with everyday internet videos"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000451,"raw_usage":{"total_tokens":2257,"prompt_tokens":915,"completion_tokens":1342,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":1282}},"tokens_in":531,"tokens_out":1342,"duration_ms":10185,"temperature":1.0,"reasoning_tokens":1282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T20:09:59.990632+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomly sample 100 SSv2 videos, run the full Video2Policy pipeline end to end, and evaluate the produced policies with hand-written ground-truth success functions; if the fraction of videos yielding a successful policy is far below the 88% average (given the 42% per-module failure rate), the claim that internet videos can scale as a data engine fails.","supporting_citations":[],"review_version":1}