Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videos

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By reconstructing internet videos into simulation tasks, Video2Policy trains policies that reach 88% average success, beating LLM-only baselines and transferring a generalist lifting policy to a real robot.

desk verdict Genuinely novel pipeline that uses internet video-derived geometry and 6D poses to ground LLM-generated robot tasks, with honest reporting of module failures, but the 88% average success rests on a curated 12-task subset with no disclosed selection funnel. read the letter →

arxiv 2502.09886 v1 pith:DASG5RF3 submitted 2025-02-14 cs.RO cs.AIcs.LG

Weirui Ye , Fangchen Liu , Zheng Ding , Yang Gao , Oleh Rybkin , Pieter Abbeel This is my paper · ORCID
classification cs.ROcs.AIcs.LG
keywords simulationinternetvideosreinforcementlearningtaskgenerationrewarddesignvision-languagemodelrobotmanipulationsim2real
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Internet RGB videos of everyday human manipulation can serve as an automated data engine for training robot policies in simulation. The paper's Video2Policy pipeline first turns each video into a simulated tabletop scene, reconstructing object meshes, their real-world sizes, and their 6D pose tracks, then has a vision-language model (GPT-4o) write the full task code: reset, success, observation, and reward functions. Policies are trained with PPO, and the reward function is refined iteratively from training logs, an in-context reward-reflection loop. On 12 reconstructed tasks the learned policies reach 88% average success, beating Code-as-Policy, RoboGen, and Eureka baselines, and the pipeline can collect expert trajectories to train a generalist behavior-cloned policy that transfers to a real robot.

What carries the argument

The central mechanism is the task JSON, a compact description of each video that bundles the reconstructed object meshes, their scaled sizes, 6D pose tracks, and the video caption. A vision-language model (GPT-4o) reads this JSON and writes executable task code covering six parts: scene information, reset function, success function, observation function, observation-space function, and reward function. The reward is then refined through in-context reward reflection: PPO trains $N = 8$ candidate rewards per round, training logs and per-component analyses are fed back to the VLM, and the best candidate is selected for the next round; the training reward is the sum of the learned reward and the generated success signal weighted by $\lambda = 100$.

What would settle it

Randomly sample 100 SSv2 videos, run the full Video2Policy pipeline end to end, and evaluate the produced policies with hand-written ground-truth success functions; if the fraction of videos yielding a successful policy is far below the 88% average (given the 42% per-module failure rate), the claim that internet videos can scale as a data engine fails.

Watch

Extended reading notes

Core claim

The paper claims that internet videos can be a scalable source of grounded, task-level diversity for robotic simulation, unlike text-only LLM generation that hallucinates uninteresting tasks. Video2Policy reconstructs the manipulated objects and their relative motion from each video into a task JSON, and the VLM generates executable task code including a success function informed by visual priors, not just text captions. On 12 tasks from SSv2 and self-recorded videos, the method reaches 88% average success, compared with 71% for Eureka, 45% for RoboGen, and 34% for Code-as-Policy. The authors further show that 100 reconstructed lifting tasks can train an image-based behavior-cloned policy that generalizes to 10 unseen objects (75% success) and succeeds on 47% of real robot trials after sim-to-real transfer.

Load-bearing premise

The whole approach rests on the video-reconstruction stage supplying an accurate task JSON (object meshes, sizes, and 6D pose tracks) that the VLM then uses to write the task code; the paper's own robustness analysis reports a 42% average module failure rate, so this premise is fragile.

Editorial extensions

If this is right

  • If internet RGB videos are a reliable task source, robotic training data can scale with the internet rather than teleoperation or hand-designed benchmarks.
  • The ablations show the iterative reward-reflection step is the largest contributor, with average success dropping from 87% to 48% when it is removed.
  • A general policy lifted from 100 reconstructed lifting tasks generalizes to 10 unseen objects at 75% success, including objects from unseen categories.
  • The learned generalist policy transfers to a real robot with 47% success when using segmentation-mask input and domain randomization.
  • Success on unseen tasks rises from 13% to 75% as the number of training tasks grows from 10 to 100, indicating the data engine scales with video supply.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper's own robustness analysis reports an average 42% vision-module failure rate, the end-to-end yield on arbitrary internet video is likely lower than the 88% policy-level success; practical deployment would require cheap filtering of reconstruction failures before training.
  • The success-function correlation result (0.83 with visual information vs 0.56 without) suggests the pipeline's ceiling is tied to perception quality, so swapping in stronger detection, mesh, or tracking models should raise both reconstruction yield and policy success.
  • The framework could be tested on behaviors without a clear final success state (for example, continuous or two-agent interactions) by having the VLM infer intermediate success criteria from the trajectory, an extension the paper does not address.
  • Since the generalist policy was evaluated only on lifting, a natural next test is whether the same data engine scales to multiple behaviors trained in one policy from diverse video collections.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Video2Policy, a pipeline that turns internet RGB videos into simulation tasks for robot policy learning. The pipeline reconstructs object meshes and 6D poses from videos, uses GPT-4o to generate scene, reset, success, observation, and reward code, then trains RL policies with iterative in-context reward reflection. The authors report an average 88% success rate over 12 tasks (9 from SSv2, 3 self-recorded), a 75% success rate for a behavior-cloned generalist policy on 10 unseen lifting videos, and 47% success in a real-robot deployment. The central claim is that internet videos can serve as an automated data engine for generating diverse, grounded manipulation tasks in simulation.

Significance. If the central claim held, the work would be a meaningful step toward scalable, video-grounded simulation data for robotics, combining off-the-shelf vision models with LLM-generated task and reward code. The paper deserves credit for reporting a component-wise robustness analysis (Appendix A.2, Table 4), ablating each stage of code generation and reward reflection, checking that baselines are not disadvantaged by reset functions (Table 9), and attempting a real-robot transfer. However, as presented, the evidence is not strong enough to establish the 'automated data engine' claim: the video-to-task funnel is unquantified, per-task evaluation is based on only 10 episodes with high variance, success functions are hand-written without independent validation, and manual interventions remain in the pipeline. The relative comparisons against LLM-only baselines are suggestive, but the absolute performance claims and the scalability inference require substantially more evidence.

major comments (4)
  1. [Appendix A.2, Table 4; Section 3.1] The central 'automated data engine' claim is not supported by an end-to-end yield analysis. Table 4 reports an average module failure rate of 42%, with Grounding DINO failing on 60% of SSv2 videos and FoundationPose on 55%, yet the paper never states how many source videos were attempted, how many survived each pipeline stage, or how the 9 SSv2 tasks and 3 self-recorded tasks in Table 1 were selected. Without this funnel information, the 88% average success rate is a conditional statement about a potentially curated subset rather than evidence that a useful fraction of arbitrary internet videos can be converted into runnable, solvable simulation tasks.
  2. [Section 4, Experimental Setup; Table 1] The evaluation protocol is too weak to support the headline quantitative claims. Success rates are averaged over only 10 evaluation episodes per task with three seeds, and several entries have large standard deviations (e.g., 'Push sth. with sth.' 0.43 ± 0.40, 'Throw Garlic into Bowl.' 0.70 ± 0.36, 'Tip sth. over' Eureka 0.67 ± 0.47). With 10 episodes, a single episode changes the success rate by 10%, so the reported differences between Video2Policy and Eureka (0.88 vs. 0.71) are not statistically established, and the near-zero task 'Cover sth. with sth.' (0.07 ± 0.05) indicates that some generated tasks are essentially unsolved and should be discussed separately rather than averaged into the headline number.
  3. [Section 4, Evaluation metric; Appendix A.3] The absolute success rates rest on hand-written success functions that are not independently validated. The paper states that 'we write the ground-truth success function for each generated task' (Section 4), and the only validation in Fig. 10 is a correlation between GPT-4o-generated success functions and these same hand-written ones. Since the manual success functions define the evaluation itself, the reported absolute success rates may be sensitive to the leniency or strictness of the authors' criteria; only the relative comparison across methods, which share the same success functions, is protected.
  4. [Section 3.1; Section 4, Scene Generation] The pipeline is not fully automated as claimed. Object names and captions for the three self-recorded videos are provided manually, and the choice of first versus last frame for mesh reconstruction is made manually for occluded SSv2 objects. The manuscript should quantify the amount and nature of human intervention per task, and should either remove the 'automated' wording or report an automation rate that excludes these manual steps.
minor comments (6)
  1. [Section 4, Experimental Setup] The simulator name is misspelled as 'Issac Gym'; it should be 'Isaac Gym'.
  2. [Section 1, Introduction] The phrase 'generalist police' should be 'generalist policies'.
  3. [Appendix A.3, Fig. 10 caption] The coefficient description reads 'correlation coefficient qual to 0.83'; this should be 'equal to 0.83'.
  4. [Section 3.1, Size estimation] The sentence 'as Dimage and Dimage are calculated within the same camera coordinate system' appears to contain a typo; the second quantity should likely be Dmesh.
  5. [Appendix A.2, Tables 5 and 6] Table 6's caption repeats the D1-distance description from Table 5, but Table 6 reports predicted versus ground-truth object sizes; the caption should be corrected accordingly.
  6. [General] No code, data, or pre-trained models are released, and the paper does not state an availability plan; given the complexity of the pipeline, this limits reproducibility and should be addressed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the video-to-task pipeline is a composition of external modules, and the reported sim2real and unseen-object results are genuine external checks.

full rationale

The paper's derivation chain is not circular. Scene reconstruction (Grounding DINO, SAM-2, InstantMesh, FoundationPose) and task-code generation (GPT-4o) are external components whose outputs feed the RL pipeline; no output is defined in terms of the final reported success, and no equation reduces to the input by construction. The training reward is the sum of a GPT-generated reward and a GPT-generated success indicator (R_hat + lambda * R_{0|1}, lambda=100), while evaluation uses hand-written "ground-truth" success functions that are stated to be inaccessible during training. The appendix reports a correlation of r=0.834 between generated and manual success functions, which shows they are similar but not identical; this is a potential evaluation-validity concern, not circularity. The general-policy experiments on 10 unseen videos (75% simulation success) and the real-robot deployment (47% success) are predictions on held-out instances and therefore provide independent support. The robustness analysis (Table 4, 42% average module failure) and manual interventions (choosing first versus last frame for mesh reconstruction, providing object names for self-recorded videos) weaken the scalability claim but are external-validity and selection concerns, not circular reasoning. Self-citations (Ye et al. 2021; Wang et al. 2024) are only used as general RL motivation and are not load-bearing. No circular step meets the required standard of exhibiting a specific reduction or a fitted parameter renamed as a prediction.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to the central success metric; the pipeline relies on external model outputs (depth, poses) and hyperparameters (e.g., lambda equals 100) that do not encode the target results. The central claim rests on a chain of vision and language model assumptions rather than on a parametric derivation.

assumptions (4)
  • domain assumption Isaac Gym physics engine provides sufficient physical fidelity for tabletop manipulation learning.
    All task generation and RL training take place in Isaac Gym; if the physics is too inaccurate, policies may not transfer or even solve the simulation tasks. Section 4 Experimental Setup invokes this.
  • domain assumption The vision sub-pipeline (Grounding DINO, SAM-2, InstantMesh, UniDepth, FoundationPose) yields accurate enough detections, masks, meshes, sizes, and 6D poses from monocular video.
    These outputs compose the task JSON that drives all downstream code generation. The paper's own robustness table (App. A.2) reports an average module failure rate of 42%, so this is load-bearing and fragile.
  • domain assumption GPT-4o can translate the task JSON and captions into executable Isaac Gym task code, including reset, success, observation, and reward functions.
    The entire method depends on LLM code generation. The paper handles imperfect outputs by sampling and picking, but still assumes a capable model. Section 3.2.
  • domain assumption The human-written ground-truth success functions correctly define task completion and are used consistently for evaluation.
    Reported success rates are measured against these hand-written functions. If they are too lenient or incorrect, the headline numbers are misleading. Section 4.1 states that the ground-truth success function is written for each generated task.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videos." pith.science (2026). https://pith.science/paper/DASG5RF3

@misc{pith2026250209886,
  author       = {Pith},
  title        = {Pith review of: Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DASG5RF3}},
  note         = {Machine review of arXiv:2502.09886}
}
read the original abstract

Simulation offers a promising approach for cheaply scaling training data for generalist policies. To scalably generate data from diverse and realistic tasks, existing algorithms either rely on large language models (LLMs) that may hallucinate tasks not interesting for robotics; or digital twins, which require careful real-to-sim alignment and are hard to scale. To address these challenges, we introduce Video2Policy, a novel framework that leverages internet RGB videos to reconstruct tasks based on everyday human behavior. Our approach comprises two phases: (1) task generation in simulation from videos; and (2) reinforcement learning utilizing in-context LLM-generated reward functions iteratively. We demonstrate the efficacy of Video2Policy by reconstructing over 100 videos from the Something-Something-v2 (SSv2) dataset, which depicts diverse and complex human behaviors on 9 different tasks. Our method can successfully train RL policies on such tasks, including complex and challenging tasks such as throwing. Finally, we show that the generated simulation data can be scaled up for training a general policy, and it can be transferred back to the real robot in a Real2Sim2Real way.

Figures

Figures reproduced from arXiv: 2502.09886 by the authors.

Figure 1
Figure 1. The Video2Policy framework can leverage internet videos to generate simulation tasks and learn policies for them automatically, which can be considered a data engine for generalist policies. 2024). RoboGen (Wang et al., 2023c) leverages LLM￾generated reward functions for RL. However, it is hard to scale as it requires manual success functions. However, since we leverage both text prompts and explicit visual prior kn… view at source ↗
Figure 2
Figure 2. Some visualization of the tasks generated from SSv2 Video Dataset. the video, which will be fed into GPT-4o for code generation. We utilize FoundationPose (Wen et al., 2024) in model￾based setups to estimate the position and orientation of the objects. This model takes the object mesh, predicted camera intrinsics, and the depth information from each frame as inputs. Finally, we automatically generate a URDF file for… view at source ↗
Figure 3
Figure 3. Examples of the Segmentation Mask Observation between the simulation and the real, which can better bridge the sim2real gap. 2023), we want to investigate how to learn a general policy from the internet videos, which directly outputs the exe￾cutable actions of the robot rather than the un-executable future videos (Du et al., 2024; Qin et al., 2023) or language tokens (Liang et al., 2023; Brohan et al., 2023). We con… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: V2P achieves better performance across iteration. 4.2. Policy Generalization Analysis from diverse videos To further demonstrate the scalability of our framework, we target training a general policy from diverse videos. As mentioned in Sec. 3.3, we regard the Video2Pol…
Figure 5
Figure 5. Figure 5: Performance of the trained general policy on 10 unseen task instances. BC-V2P outperforms BC-CoP on 9 of 10 significantly. focus on one single behavior, lifting up. Specifically, we sample 100 videos concerning the lifting behavior from SSv2 dataset, generate the scene…
Figure 6
Figure 6. Figure 6: Scalability of the general policy model (BC-V2P) to￾wards the number of training tasks on lifting behavior. Generalization to Unseen Videos The performances of the models are shown in [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: An example of the generated code for the task Insert. Examples of Code Evolutions We provide examples of modifying the reward function based on the reward logs. During reward iterations, we provide examples of how humans adapt and refine the reward functions. We add th…
Figure 8
Figure 8. Figure 8: An example of the newly generated reward function for the task Throw. ... ### Analysis of Policy Feedback 1. **Distance from Gripper to Garlic (dist_grasp_reward)** - **Observation**: The values fluctuate around 0.1, with a minimum of 0.0018 and a maximum of 0.1320. Th…
Figure 9
Figure 9. Figure 9: An example of the reward analysis for the task Throw. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Correlation analysis of generated success functions and the manual ground-truth functions. success functions with visual information can be over-confident, as the data points lie below the y=x curve. Instead, it can be noise with caption only when generating the succe…
Figure 11
Figure 11. Figure 11: An example of how GPT-4o picks better code for less hallucination. The generated codes do not reset cheating. Since the task codes are generated under different prompts for all the methods, we conduct ablation studies by choosing the same ’reset’ function from our met…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SimLauncher: Launching Sample-Efficient Real-world Robotic Reinforcement Learning via Simulation Pre-training

    cs.RO 2025-07 conditional novelty 5.0 of 10

    Simulation-pretrained policies, with digital-twin demos for critic bootstrapping and action proposals, cut real-world RL training time while reaching near-perfect success on three manipulation tasks.

Reference graph

Works this paper leans on

17 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    The distance between the gripper and the garlic

    work page

  2. [2]

    Throw the garlic towards the bowl

    work page

  3. [3]

    # Reward Components:

    Ensure the garlic reaches the bowl. # Reward Components:

    work page

  4. [4]

    bowl_pos

    The final position of the garlic being inside the bowl. """ batch_size = states[’eef_pos’].shape[0] rewards = torch.zeros(batch_size, device=states[’eef_pos’].device, dtype=torch.float32) reward_components = {} # 1. Adjust Distance from gripper to garlic reward dist_eef_to_garlic = torch.norm(states[’garlic_pos’] - states[’eef_pos’], dim=-1) dist_grasp_re...

    work page

  5. [5]

    The garlic’s distance to the bowl

    work page

  6. [6]

    The velocity of the garlic towards the bowl

    work page

  7. [8]

    The last value is 0.1272

    **Distance from Gripper to Garlic (dist_grasp_reward) ** - **Observation**: The values fluctuate around 0.1, with a minimum of 0.0018 and a maximum of 0.1320. The last value is 0.1272. - **Suggestion**: The fluctuation indicates variability, but the impact may be insufficient for success. Adjust the scale or temperature slightly for better sensitivity

    work page

  8. [9]

    - **Suggestion**: Increase the temperature parameter for better gradient

    **Distance from Garlic to Bowl (dist_garlic_to_bowl) ** - **Observation**: The values are quite low (0.0000 - 0.0029), indicating this component is not significantly contributing. - **Suggestion**: Increase the temperature parameter for better gradient

    work page

Show all 17 references
  1. [10]

    The last value is 0.1746

    **Velocity Reward (vel_reward) ** - **Observation**: The values show variability and seem to be contributing to the learning process. The last value is 0.1746. - **Suggestion**: It is effective but might need a slight adjustment in temperature and scaling

  2. [11]

    - **Suggestion**: Rewriting or significantly adjusting this component is necessary

    **Final Position Reward (final_position_reward) ** - **Observation**: This component consistently shows zeros, indicating it’s not achieved in any of the episodes. - **Suggestion**: Rewriting or significantly adjusting this component is necessary

  3. [12]

    It fluctuates with no episodes achieving success

    **Total Reward (total_reward) ** - **Observation**: The total reward has shown improvement over time but doesn’t lead to success. It fluctuates with no episodes achieving success. - **Suggestion**: Improve the component scales to better balance the total reward for successful ...

  4. [13]

    card_pos

    We can find that the segmentation and mesh reconstruction parts are more robust than the others. Table 5. D1 distance between the predicted depth by Unidep (Piccinelli et al., 2024) and the ground-truth depth in Sliding video. Full size Center region (0.8x crop) Object boundin...

    2024

  5. [14]

    This added information is highly useful for learning tasks involving object manipulation

    Inclusion of the End-Effector-Card Distance in Observations: This is beneficial because it provides an additional critical feature that can help the agent understand its relative position to the target object. This added information is highly useful for learning tasks involvin...

  6. [15]

    Enhanced Success Criteria: ... The second implementation refines the success criteria by adding a condition that the card’s linear velocity should be minimal (states[’card_vel_linear’].norm(dim=-1) < 0.01), ensuring that the card is not just lifted but also stable. This is a m...

  7. [16]

    Dynamic and Informative Observations:

  8. [17]

    Figure 11

    Observation Space Adjustment: ... Figure 11. An example of how GPT-4o picks better code for less hallucination. The generated codes do not reset cheating. Since the task codes are generated under different prompts for all the methods, we conduct ablation studies by choosing th...

  9. [318]

    "" insert fork into storage box

    PMLR, 2023. Chen, Z., Kiami, S., Gupta, A., and Kumar, V . Genaug: Retargeting behaviors to unseen situations via generative augmentation. arXiv preprint arXiv:2302.06671, 2023. Chen, Z., Walsman, A., Memmel, M., Mo, K., Fang, A., Vemuri, K., Wu, A., Fox, D., and Gupta, A. Urd...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.