{"id":"f27116fe-9135-4dbc-bdcd-897fee5f479c","arxiv_id":"2501.04268","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A VLM fine-tuned on video-derived synthetic policy code outperforms GPT-4o zero-shot on RLBench and LIBERO robotic manipulation benchmarks.","lead":"This paper introduces RoboPro, a vision-language model that writes executable robot control programs from a camera image and a plain-language instruction, trained on 115k synthetic code samples generated from robot videos. RoboPro beat GPT-4o in zero-shot manipulation success rates on RLBench, LIBERO, and a real Franka robot, without task-specific fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on unverified synthetic training code: Video2Code's 115k generated programs are neither executed nor filtered, so the model's gains could reflect API-library familiarity rather than learned procedural knowledge.","rationale":"I agree with the reader's identification of the weakest assumption: the correctness of Video2Code's synthetic code is the linchpin of the method. The paper provides no evidence that the generated code is executable or semantically aligned with the videos, and it applies no filtering. The ablation only shows that the data is useful for learning API syntax, not that it conveys procedural knowledge. The strongest claim (surpassing GPT-4o by 11.6%) depends on RoboPro having learned something beyond API familiarity; if the training targets are broken, the reported performance could be an artifact of the API library's heuristics rather than genuine video-derived knowledge. This is a concrete, testable gap, and it is more fundamental than the secondary issues (missing RoboCodeX comparison, lack of error bars, no release) because it threatens the causal mechanism of the contribution. The reader's CONDITIONAL verdict remains appropriate: the method is promising but the central claim is not yet adequately supported. I would maintain CONDITIONAL pending the proposed verification of the synthetic data quality.","tokens_in":16583,"tokens_out":6357,"duration_ms":61095,"concrete_test":"Sample 500 generated training examples (video segment + instruction + generated code) from the Video2Code pipeline. For each, execute the code in the RLBench simulator (or a lightweight mock of the API library) using the same scene/observation extraction as training, and measure (a) fraction that runs to completion without exception, and (b) fraction whose execution achieves the instruction's success criterion. Report these numbers separately from the downstream RoboPro evaluation. If the execution success rate is below ~50% (or if the validity rate is below ~80%), then Video2Code is not producing 'high-quality runtime code', and the performance gains in Table I should be re-attributed or re-evaluated. This test directly addresses whether the SFT targets are reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that RoboPro's zero-shot performance stems from procedural knowledge learned from video-derived policy code. The entire data source is Video2Code, which uses Gemini-1.5-Flash to turn DROID video keyframes into natural-language plans and DeepSeek-Coder-V2 to translate those plans into code. Section III-B states that this yields '115k runtime code data' but reports no metric for how many of these programs are syntactically valid, execute without error, or actually accomplish the task. No filtering or verification is applied before SFT. The only evidence for data utility is the ablation (Table V), where removing Video2Code drops RLBench success from 42.7% to 0.4%. But that comparison conflates two effects: learning the API definition/usage syntax and learning correct procedural ordering. A model that has never seen the API library is at a severe disadvantage; its failure says little about the semantic correctness of the Video2Code targets. If a large fraction of the 115k programs are hallucinated or logically broken, the language-modeling objective trains RoboPro to reproduce those flaws. The reported success on RLBench could then be explained by the API library's powerful heuristics (which already embody much of the task structure, e.g., generate sweep path, generate pour path) and by the model learning the most common API-call skeleton for each instruction, rather than by acquiring genuine task knowledge from video content. The paper's error analysis (Fig. 4) also cannot rescue this: it examines downstream execution errors, not the correctness of the synthetic training labels. Without a direct measurement of Video2Code's output quality, the causal story—'videos in-the-wild provide procedural knowledge that transfers to simulators and real robots'—is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RoboPro, a vision-language model that observes wrist-camera RGB-D images and a free-form instruction, and outputs Python policy code to be executed through a predefined API library of perception and control modules. To obtain training data, the authors introduce Video2Code, a pipeline that uses Gemini-1.5-Flash to convert 50,000 DROID videos into natural-language plans and DeepSeek-Coder-V2 to translate those plans into 115,000 code snippets; these are combined with general visual-instruction data for supervised fine-tuning of CodeQwen-1.5-7B with a SigLIP vision encoder. The paper evaluates RoboPro on nine RLBench tasks, eight LIBERO tasks, and eight real-world tasks, reporting that it outperforms GPT-4o by 11.6% on RLBench and 17.4% on LIBERO, and that it is robust to API renaming, API refactoring, and unseen task-specific skills.","tokens_in":16861,"tokens_out":6360,"duration_ms":62578,"significance":"If the reported results hold, the contribution is significant: Video2Code offers a scalable way to convert in-the-wild videos into policy-code training data, and RoboPro would demonstrate that a 7B open vision-language model can generate executable policy code that generalizes zero-shot across simulators and a real robot, outperforming a proprietary generalist VLM. The paper also provides an early study of robustness to API format changes and unseen skill sets. However, the current evidence is not yet sufficient to support the state-of-the-art claim: the reported margins are within statistical noise for the sample sizes, the closest prior method (RoboCodeX) is not benchmarked, and the synthetic training code is not validated for executability or correctness.","major_comments":[{"comment":"All results are point estimates from 25 (RLBench), 30 (LIBERO), or 10 (real-world) episodes, with no confidence intervals, standard errors, or multiple seeds. On RLBench, a single episode changes a task average by 4 percentage points, so the headline gap of 11.6% (42.7 vs 31.1) is within roughly two standard errors of the difference, and several per-task gaps are 4-8 points. The statements that RoboPro \"significantly outperforms\" GPT-4o (Section IV-A.2 and the Abstract) are therefore not supported by the reported statistics. Please provide Wilson intervals or other confidence intervals, run multiple seeds, or perform paired episode-level comparisons when the same episodes are used across methods.","section":"Section IV-A.1 and Tables I, II, IV"},{"comment":"The paper identifies RoboCodeX as the closest prior multimodal policy-code generation method, and says the primary comparison group is code-generation methods, yet RoboCodeX is not evaluated in any of Tables I-IV. Without a same-protocol comparison under the same API library, the claims of state-of-the-art zero-shot performance (Abstract and Section IV-A.1) are not established. The authors should add a RoboCodeX comparison, or explicitly justify why it cannot be run under the same evaluation protocol and soften the state-of-the-art claim accordingly.","section":"Section IV-A and Section II-B"},{"comment":"Section III-B reports collecting 115k runtime code data via Gemini-1.5-Flash and DeepSeek-Coder-V2, but no executability check, syntactic validation, or task-success verification is reported for these programs. The ablation in Table V (42.7 vs 0.4 on RLBench when Video2Code is removed) is consistent with the model learning API syntax and naming, since the no-Video2Code model never saw the API library during fine-tuning; it does not establish that the synthetic programs encode correct procedural knowledge. Please provide quantitative quality statistics for the 115k programs (e.g., syntax validity, API-call coverage, human or automated correctness judgments) and include a control trained on non-executable or API-skeleton-only code to separate API familiarity from procedural learning.","section":"Section III-B and Table V"},{"comment":"The real-world evaluation uses 10 tests per task, so per-task differences such as 80 vs 70 or 60 vs 60 are not meaningful without error bars or a significance test. Likewise, the API-renaming and API-refactoring results in Table I are single-run RLBench averages of 42.7 and 40.4, a 2.3-point gap that is within noise for 25 episodes per task. These robustness claims should either be presented with uncertainty quantification or be described as preliminary observations rather than definitive evidence of robustness.","section":"Section IV-B.1 and Section IV-C"}],"minor_comments":[{"comment":"The phrase \"Our primary comparison tagets\" contains a typo and should read \"Our primary comparison targets.\"","section":"Section IV-A"},{"comment":"The table captions contain the garbled phrase \"GREYED ON NEED SUPERVISED TRAINING\" and \"GREYED ON\"; these should read \"GREYED OUT: REQUIRES SUPERVISED TRAINING\" or similar.","section":"Tables I and II"},{"comment":"The sentence \"the results on tow simulation platforms\" should read \"the results on two simulation platforms.\"","section":"Appendix V-D"},{"comment":"The prompt text contains the typo \"excecute\" (also as \"Begin to excecute the task\"); it should be \"execute.\"","section":"Listings 1-3"},{"comment":"The phrase \"changes in API formation\" should be \"changes in API format.\"","section":"Section IV-B.1"},{"comment":"The error breakdown lacks definitions, counts, and an annotation procedure, which makes it difficult to verify the claim that RoboPro reduces logical errors relative to GPT-4o.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a robotics venue and the proposed pipeline is interesting, but the evidentiary basis for the central state-of-the-art claim needs substantial strengthening. The absence of any verification of the 115k synthetic programs is a particular gap: if many generated programs are syntactically invalid or logically wrong, the reported success could be driven by the hand-designed API library rather than by procedural knowledge learned from videos. These issues are fixable with additional experiments and analysis, so I am recommending major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Video2Code is the real contribution here, and it's a good one. The idea of using off-the-shelf Gemini-1.5-Flash and DeepSeek-Coder-V2 to convert DROID videos into policy code without human annotation is cheap and scalable. RoboCodeX needed human-annotated code and hand-built simulation; RoboPro's data pipeline directly attacks that bottleneck. The evaluation is broader than most code-as-policies papers: RLBench, LIBERO, a real robot, plus robustness to API renaming/refactoring and unseen skill sets. That is real effort and worth credit.\n\nThe soft spots are real, though. Every success rate is a point estimate from 25/30 episodes (or 10 in the real world) with no confidence intervals, no seeds, no significance tests. An 11.6-point gap on RLBench is not obviously meaningful without variance. They never benchmark against RoboCodeX, the closest prior method. And the 115k synthetic programs from Video2Code are never checked for executability or semantic correctness. The ablation that removes Video2Code (RLBench drops to 0.4%) conflates learning API syntax with learning procedural knowledge; a model that has never seen the API library will fail regardless of label quality. So that ablation alone does not validate the synthetic code.\n\nWhere I'd push back on the harshest reading: the robustness results (Table I, Table III) suggest the model learned something deeper than a memorized skeleton. If RoboPro had just memorized API call order for seen instructions, renaming function names or changing argument types would likely degrade it more than the observed 42.7→40.4 on refactoring. And it beats GPT-4o on tasks with new task-specific APIs. So the 'it is all API familiarity' worry is plausible but not established; the procedural-knowledge story is still standing, just under-supported.\n\nMissing code/data is a real issue for a SOTA claim. The arXiv version points to a website but releases nothing I can see.\n\nVerdict: this deserves peer review. A good referee should demand confidence intervals, a RoboCodeX comparison, and a direct measurement of Video2Code output quality (e.g., parse rate, execution accuracy on a few held-out sim tasks). If those come back positive, we have a solid contribution. Send it out.","headline":"Video2Code is a genuinely useful data-curation idea, but the SOTA claim needs confidence intervals and a RoboCodeX baseline before I'd bet on it.","tokens_in":17445,"tokens_out":4261,"would_cite":true,"duration_ms":38448,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7B vision-language model trained on code synthesized from in-the-wild videos generates executable policy code for robotic manipulation in a zero-shot manner, outperforming GPT-4o on RLBench, LIBERO, and real-world tasks.","keywords":["robotic manipulation","zero-shot generalization","policy code generation","vision-language model","video-to-code synthesis","code as policy","robot foundation model"],"falsifier":"Inspect the synthesized training corpus: execute or statically analyze the 115k Video2Code programs against the API library in a simulator and count how many run without error and implement the stated instruction. A finding that the large majority are non-executable or semantically wrong, while RoboPro still scores well downstream, would undercut the claim that it learned procedural knowledge from video-derived code; a clearer version is to train RoboPro on the same data with API-call order scrambled and check whether the RLBench success rate collapses to the no-Video2Code baseline.","tokens_in":16385,"feed_emoji":"🤖","tokens_out":12153,"duration_ms":101348,"temperature":0.7,"pith_summary":"RoboPro aims to show that a single open 7B vision-language model can generate executable policy code for robotic manipulation in a zero-shot manner, with no fine-tuning on the target robot or environment. The paper's central claim is that training on 115k code samples synthesized from in-the-wild robot videos lets RoboPro outperform the much larger GPT-4o by 11.6% on RLBench and 17.4% on LIBERO, and reach 72.5% average success on eight real-world tasks. If true, this matters because it replaces expensive runtime code collection and manual simulation annotations with an automatic video-to-code pipeline, potentially scaling robot foundation models from video data alone. The paper also claims the learned skill knowledge transfers to renamed or refactored API formats and to entirely unseen skill sets.","feed_headline":"Video-trained robot model beats GPT-4o on zero-shot tasks","feed_subtitle":"A 7B open model trained on synthetic code from videos surpasses GPT-4o in simulation and the real world.","key_machinery":"The load-bearing mechanism is Video2Code, an automatic two-stage data curation pipeline. Stage one uses Gemini-1.5-Flash on 16 key frames per video to extract a concise robot-centric natural-language plan; stage two feeds that plan, the original instruction, and API definitions to DeepSeek-Coder-V2 to generate executable policy code. The API library itself is the second piece: it splits into perception modules, which ground objects and physical properties via GroundingDINO and AnyGrasp, and control modules, which provide heuristic paths for sliding, sweeping, pouring, wiping, and joint rotation. Because RoboPro is trained to call these APIs through code, the authors argue the model internalizes atomic skills independently of their names or signatures. The model is a SigLIP-L vision encoder connected through a two-layer MLP to CodeQwen-1.5-7B, trained in three stages, the last being supervised fine-tuning on the 115k Video2Code samples mixed with general visual instruction data.","core_discovery":"RoboPro is a vision-language model that takes a wrist-camera RGB-D image, a free-form instruction, and an API library definition, and outputs Python code that calls atomic perception and control functions to produce trajectories. The authors' discovery is that operational videos, not robot demonstrations annotated with code, can serve as the source of training data: Video2Code uses a draft VLM to turn 50k DROID videos into step-by-step natural-language plans, then a code-specialized LLM translates the plans into executable policy code against the API library, yielding 115k aligned image-instruction-code triples. Fine-tuned on this data, RoboPro generates code whose logical ordering of API calls is substantially more accurate than that of GPT-4o, which the authors identify as the main source of its success on RLBench and LIBERO. The authors frame this as evidence that procedural knowledge implicit in videos is transferable to novel tasks, environments, and skill definitions.","pith_inferences":["Beyond the paper, if Video2Code scales as claimed, any large video corpus containing manipulation, including human-hand or egocentric video, could in principle be distilled into policy-code training data, potentially removing the robot-specific data bottleneck for embodied foundation models.","Beyond the paper, because the paper does not verify or filter the synthesized code before training, a direct test of the mechanism is whether RoboPro trained on API-call orders that have been deliberately scrambled loses its advantage; if it does not, the gains may reflect API-format familiarity rather than learned procedural sequencing.","Beyond the paper, the method depends on a hand-designed API library, so each new embodiment or sensor suite likely needs a new library; an implicit next step is to learn or adapt the API definitions themselves from data.","Beyond the paper, the authors point toward navigation as a next domain, and the same recipe of video-to-plan-to-code could be tested there with minimal changes to the pipeline."],"forward_implications":["Zero-shot policy code generation from a 7B open model can exceed the manipulation success of a frontier proprietary VLM on both RLBench and LIBERO, without any task-specific training.","The same API library and code-format prompts work across two simulators and a real Franka arm, so code generation transfers across embodiments and environments.","RoboPro's success on RLBench rises as the fraction of Video2Code data grows from 25% to 100%, indicating that scaling automatically synthesized code data is a viable route to improve such models.","Because performance on unseen task-specific APIs (Water Plants, Hit Ball, Scoop Cube) stays above GPT-4o, the procedural knowledge learned from videos transfers to skill sets never seen in training.","Removing Video2Code data collapses RLBench success from 42.7% to 0.4%, showing the video-derived code data is the main driver of the gain."],"supporting_citations":[{"why":"DROID dataset supplies the 50k instructional videos that Video2Code distills into training code.","marker":"[8]"},{"why":"Gemini-1.5-Flash serves as the draft VLM that extracts natural-language plans from video key frames.","marker":"[22]"},{"why":"DeepSeek-Coder-V2 serves as the code LLM that translates plans into executable policy code.","marker":"[23]"},{"why":"GroundingDINO implements the open-set object detection behind the perception API calls.","marker":"[24]"},{"why":"AnyGrasp implements the grasp-pose prediction behind the control API calls.","marker":"[25]"},{"why":"Code-as-Policies is the language-only code generation baseline that RoboPro outperforms by 19.1% on RLBench.","marker":"[18]"},{"why":"GPT-4o is the state-of-the-art VLM baseline that RoboPro claims to surpass on all three evaluation settings.","marker":"[30]"},{"why":"PerAct is the supervised training-based baseline on RLBench that RoboPro's zero-shot result is compared against.","marker":"[9]"},{"why":"RLBench is the simulation benchmark where RoboPro reports the 42.7% success rate and the 11.6-point gain over GPT-4o.","marker":"[31]"},{"why":"LIBERO is the second simulation benchmark where RoboPro reports 52.4% versus GPT-4o's 35.0%.","marker":"[32]"}],"fun_headline_variants":["Robot coder from videos beats GPT-4o zero-shot","RoboPro: video-synthesized code outperforms GPT-4o","Zero-shot robot control via video-derived policy code","From in-the-wild videos to robot code: RoboPro tops GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the 115k programs synthesized by Gemini and DeepSeek-Coder are executable and logically correct enough to teach RoboPro real procedural skill; the paper does not verify or filter these programs before training, so if most are flawed the reported gains could come from API and prompt design rather than from video-derived knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Robot coder from videos beats GPT-4o zero-shot","RoboPro: video-synthesized code outperforms GPT-4o","Zero-shot robot control via video-derived policy code","From in-the-wild videos to robot code: RoboPro tops GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1612,"prompt_tokens":947,"completion_tokens":665,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":588}},"tokens_in":563,"tokens_out":665,"duration_ms":6320,"temperature":1.0,"reasoning_tokens":588,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:36:33.276896+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the synthesized training corpus: execute or statically analyze the 115k Video2Code programs against the API library in a simulator and count how many run without error and implement the stated instruction. A finding that the large majority are non-executable or semantically wrong, while RoboPro still scores well downstream, would undercut the claim that it learned procedural knowledge from video-derived code; a clearer version is to train RoboPro on the same data with API-call order scrambled and check whether the RLBench success rate collapses to the no-Video2Code baseline.","supporting_citations":[{"cited_title":"Hello gpt-4o,","cited_arxiv_id":null,"evidence_quote":"GPT-4o is the state-of-the-art VLM baseline that RoboPro claims to surpass on all three evaluation settings."},{"cited_title":"Libero: Benchmarking knowledge transfer for lifelong robot learning,","cited_arxiv_id":null,"evidence_quote":"LIBERO is the second simulation benchmark where RoboPro reports 52.4% versus GPT-4o's 35.0%."}],"review_version":1}