{"id":"03bdf803-4868-48aa-b785-d7f5738b6d18","arxiv_id":"2501.18733","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A robot framework combining GPT-4V planning with a 3D feature-field skill policy improves long-horizon kitchen manipulation accuracy over LLM baselines, according to small real-robot trials.","lead":"LMM-3DP is a robot system that uses a visual language model to plan tasks step by step and a 3D-aware policy trained on a few human demos to carry out each step. It was tested in a real kitchen and reported better success rates than several language-model baselines, though the experiments are small-scale.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 1.45x low-level and 1.5x high-level improvement ratios are not reproducible from the paper's own tables, and the introduction's 'first step 50%' contradicts Table VII.","rationale":"The most load-bearing concern is that the paper's headline numbers cannot be derived from the reported data. The reader's weakest assumption focused on the lack of error bars and the small number of trials, which is a valid statistical concern. However, the internal inconsistency is more decisive: even without considering variance, the claimed 1.5x improvement in high-level planning does not match Table IV or Table VII averages, and the first-step accuracy cited in the introduction contradicts Table VII. This directly threatens the central claim of the paper, since the abstract's quantitative comparison is the main takeaway. I agree with the reader that the paper should not be accepted as-is; the verdict CONDITIONAL remains appropriate, but the required revisions should include a precise, reproducible computation of the claimed improvement ratios or a correction of those claims. The concrete test would settle whether the numbers are merely misreported or truly absent.","tokens_in":12511,"tokens_out":7928,"duration_ms":65836,"concrete_test":"Recompute all accuracy averages from Tables II, IV, VII, and VIII using the stated trial counts, and check whether any consistent definition yields 1.45x low-level and 1.5x high-level improvements. Specifically: compute Ours vs each baseline in Table II using Grasp+Place only and using all available skills; compute Ours vs each baseline in Table IV for all tasks and in Table VII for long-horizon tasks; verify the intro's 50% first-step claim against Table VII parentheses. If the ratios do not reproduce, the abstract's quantitative claims must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims in the abstract and conclusion are not supported by the tables. For low-level control, Table II compares only Grasp and Place: Ours averages (90+65)/2=77.5%, OWL-v2 (60+45)/2=52.5% (ratio 1.48), Voxposer (60+50)/2=55% (ratio 1.41). This is roughly 1.45x, but it is not a defined comparison because the two baselines have different ratios and the other three skills (Turn, Open, Close) have no baseline. For high-level planning, Table IV gives Ours 92% vs SayCan 50% (ratio 1.84) and Voxposer 88% (ratio 1.05); Table VII gives Ours 56.7% vs SayCan+OWL-v2 6.7% (ratio 8.5) and Voxposer 10% (ratio 5.7). No calculation reproduces the claimed ~1.5x. The introduction states the baseline has first-step average accuracy of 50%, but Table VII shows first-step averages of 40% for SayCan+OWL-v2 and 66.7% for Voxposer. The paper also states 'each reported accuracy rate is obtained with 10 trials' yet describes 5 trials per object/location for pick and place. These inconsistencies mean the headline claim lacks a reproducible basis.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LMM-3DP, a framework that combines a GPT-4V-based high-level planner with a language-conditioned 3D feature-field imitation policy for real-world robot manipulation. The planner supports visual feedback, a critic agent with self-feedback, memory, and reattempts, while the low-level policy fuses DINO-derived semantic features with PointNext geometry features to predict keyframe actions. The system is evaluated on five low-level skills, high-level planning tasks, and three long-horizon kitchen tasks, with ablations on visual feedback and the critic agent. The abstract and conclusion claim a 1.45x success-rate increase in low-level control and an approximate 1.5x improvement in high-level planning accuracy over LLM-based baselines.","tokens_in":12797,"tokens_out":4983,"duration_ms":42520,"significance":"If the empirical claims were fully supported, LMM-3DP would be a useful demonstration of closed-loop LMM planning combined with 3D semantic feature-field policies for long-horizon manipulation. The integration of visual feedback, a critic, and memory into a GPT-4V planner is a plausible recipe for improving robustness, and the 3D policy builds on credible prior work (GNFactor, DNAct). The paper also ships a project page with demo videos, which is a plus. However, the central quantitative claims are not reproducible from the tables, the baselines for low-level control are not trained policies, and no statistical support is provided for any reported accuracy rate. The significance of the work therefore depends on a revision that corrects the reporting and narrows the claims.","major_comments":[{"comment":"The headline claims of \"1.45x success rate increase in low-level control\" and \"approximate 1.5x improvement in high-level planning accuracy\" are not reproducible from the paper's own tables. For low-level control, Table II compares only Grasp and Place: Ours averages 77.5%, OWL-v2 52.5% (ratio 1.48), and Voxposer 55% (ratio 1.41); these ratios differ by baseline and cover only two of the five skills. For high-level planning, Table IV gives Ours 92%, SayCan 50% (ratio 1.84), and Voxposer 88% (ratio 1.05). No calculation from these tables yields the advertised 1.5x planning improvement, and the low-level ratio is not a well-defined single number. The abstract and conclusion should state the specific comparisons actually made.","section":"Abstract and Conclusion vs. Tables II and IV"},{"comment":"The introduction states that \"our baseline only has an overall average accuracy of 7% and first step average accuracy of 50%\" for long-horizon tasks. Table VII reports overall averages of 6.7% for SayCan+OWL-v2 and 10% for Voxposer, which roughly round to 7%, but the first-step averages are 40% for SayCan+OWL-v2 and 66.7% for Voxposer. The claimed 50% first-step accuracy does not match either baseline. This discrepancy must be corrected, and the basis for any aggregate baseline number should be explicitly computed from Table VII.","section":"Introduction vs. Table VII"},{"comment":"The paper states that \"each reported accuracy rate is obtained with 10 trials,\" but the very next sentence says \"Pick is tested 5 times for each of 5 objects, place 5 times for each of 4 locations, and other skills 10 times total.\" Thus the pick and place rates in Tables I and II are based on 25 and 20 trials, respectively, not 10. This contradiction affects the interpretation of all skill accuracies, and the trial-count reporting should be revised to be unambiguous and accurate.","section":"Section IV-A, first paragraph"},{"comment":"The low-level baselines are OWL-v2, an open-vocabulary object detector, and Voxposer, a planning method with value maps; neither is a trained imitation policy. Claiming a \"1.45x success rate increase in low-level control\" against these alternatives is not a comparison of low-level control policies, and the paper's wording implies a policy-level improvement. Moreover, Turn, Open, and Close skills are not evaluated against any baseline in Table II. The authors should either add proper imitation-learning baselines or clearly reframe the comparison as \"our learned policy versus non-learned affordance-based approaches\" and avoid the general \"low-level control\" claim.","section":"Section IV-A, Table II and low-level baselines"},{"comment":"All accuracy rates are proportions from 10 trials per condition (or fewer for some subtrials) with no error bars, confidence intervals, or significance tests. For example, in Table II, the Grasp difference of 90% vs. 60% over 10 trials is not statistically significant at the 0.05 level (Fisher's exact test two-sided p ≈ 0.30). The word \"significant\" in the abstract is therefore unsupported. The paper should report confidence intervals or exact tests, or soften the language to \"observed improvement\" without statistical significance claims.","section":"Section IV-A and Tables VII/VIII"}],"minor_comments":[{"comment":"The loss equation is malformed: the equation number appears in the middle of the expression, and the line breaks split the equation into fragments. Please rewrite it as a single, properly numbered equation.","section":"Equation (1)"},{"comment":"The notation \"arot ∈ 0, 1(360/5)3\" is unclear. Please define the rotation discretization explicitly, e.g., 72 bins per axis, and clarify the dimension of the prediction.","section":"Section III-B"},{"comment":"The phrase \"Due to the emergency of LMMs\" should be \"Due to the emergence of LMMs.\"","section":"Related Work"},{"comment":"The sentence \"Our results demonstrate that our method significantly outperforms the baseline (see Table II)\" uses \"significantly\" in a statistical sense without any supporting test. Please either add a test or rephrase to \"outperforms the baseline in these trials.\"","section":"Section IV-A"},{"comment":"The text says \"Our planning part, with visual feedback and a critic agent, has nearly 100% accuracy,\" but Table IV shows an average of 92% and Table VIII shows 70% for the close-left-drawer condition. Please align this statement with the reported numbers.","section":"Section IV-A, Long Horizon Tasks"},{"comment":"The row labeled \"1st kitchen (1 kitchen checkpoint)\" is confusing: it appears to repeat the single-kitchen results from Table I, not a separate checkpoint. Please clarify the experimental condition.","section":"Table V"}],"recommendation":"major_revision","confidential_remarks":"The paper describes a plausible and potentially useful system, but the empirical reporting is not yet at the standard required for the claims made. The headline ratios are not reproducible, the trial-count statement is internally contradictory, and the low-level baseline comparison is not apples-to-apples. These are fixable with a careful re-analysis and a rewrite of the abstract and conclusion; they are not fatal to the underlying idea. I would encourage the editor to treat this as a major revision rather than a rejection, provided the authors can supply corrected numbers, add uncertainty estimates, and recalibrate their claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one if you work on long-horizon manipulation with foundation models. The paper's real contribution is a clean system design: GPT-4V as a visual planner with a critic agent and memory, feeding a language-conditioned 3D transformer policy that fuses DINO features with PointNext geometry. That integration is new, and the ablations on the critic and visual feedback are genuinely informative. The writing is clear and the related work is honest about debts to GNFactor, DNAct, and ACT3D.\n\nThe problem is the evidence. The abstract and conclusion claim a 1.45x low-level and 1.5x high-level improvement, but those numbers don't emerge from any single table. The low-level comparison in Table II covers only Grasp and Place; the other three skills have no baselines. The high-level comparison gives different ratios depending on whether you use SayCan or Voxposer (1.84x vs 1.05x in Table IV, 8.5x vs 5.7x in Table VII). The introduction's 'first step 50%' doesn't match Table VII, which shows 46.7% and 66.7% for the two baselines. Every accuracy comes from 10 trials with no variance or significance, and the paper itself contradicts its trial counts: Section IV says each rate comes from 10 trials, but pick and place are described as 5 trials per object/location. The low-level baselines are detector/value-map methods, not trained imitation policies, so the comparison blurs planning and control.\n\nNone of this kills the central idea. The system is sensible, the ablations suggest the critic and closed-loop replanning help, and the limitations paragraph is honest. But the headline claims, as stated, are not reproducible from the paper's own tables, and the absence of uncertainty quantification is a real problem.\n\nWho should read it: people building LMM-planning-plus-skill systems will find the architecture a useful reference point. The evaluation template is also a cautionary example.\n\nIt deserves a serious referee, but with the expectation of major revision: redo the numbers, add error bars or more trials, and make the baseline comparisons apples-to-apples.","headline":"A sensible LMM-plus-3D-policy integration whose headline numbers don't survive close reading; worth peer review, but the evaluation needs major revision.","tokens_in":13372,"tokens_out":2290,"would_cite":false,"duration_ms":19724,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Integrating a GPT-4V planner with a 3D feature-field policy lifts robot manipulation success by 1.45x and planning accuracy by 1.5x over LLM baselines.","keywords":["large multimodal models","robot manipulation","3D feature fields","imitation learning","long-horizon planning","GPT-4V","kitchen manipulation","critic agent"],"falsifier":"Rerun the three long-horizon tasks with 50 trials per condition and compute binomial confidence intervals for the success rates; if the 95% intervals for LMM-3DP and the baselines overlap substantially, the claimed 1.45x and 1.5x advantages are not established.","tokens_in":12281,"feed_emoji":"🤖","tokens_out":5181,"duration_ms":40042,"temperature":0.7,"pith_summary":"This paper tries to establish that robot manipulation in real-world kitchens can be made far more robust by combining a visual-language planner that can see and self-correct with a low-level policy that understands 3D geometry and semantics. The authors build LMM-3DP, which uses GPT-4V to generate and revise step-by-step language plans, a critic agent that checks each plan against the current scene, and a memory of past critiques; the low level is a language-conditioned 3D transformer that predicts keyframe poses from a fused semantic-geometric point cloud. Across five skills, composition tests, and three long-horizon tasks, the framework reports a 1.45x success-rate increase in low-level control and roughly 1.5x higher planning accuracy over LLM-based baselines, and 56.5% overall accuracy on long-horizon tasks where a baseline gets 7% (50% on the first step). If these numbers hold, the framework's value is showing that closed-loop visual feedback and a reusable 3D skill library can together overcome the classic failure modes of LLM planners, which ignore scene state and cannot adapt when the world changes mid-task.","feed_headline":"GPT-4V planner + 3D feature field: robot kitchens get 1.5x better","feed_subtitle":"A self-correcting visual planner with a 3D skill policy beats LLM baselines on long-horizon manipulation in a real kitchen.","key_machinery":"The load-bearing mechanism is the semantic-aware 3D feature field built by back-projecting DINO 2D features onto geometry features from PointNext, which is then attended jointly with CLIP language embeddings inside a 3D transformer (a '3D feature field transformer'). This unified representation is what lets a single imitation-learned policy perform multiple skills—grasp, place, turn, open, close—from a few demonstrations and to compose pick-and-place skills across unseen object-location pairs. On the planning side, the GPT-4V planner operates in a closed loop with a critic agent and a memory of human-summarised critiques; the critic takes only the visual observation and proposed plan, so it can veto steps that contradict the scene (e.g., open a drawer that is already open, or pick an object that is not present). The two sides are integrated by having the language embeddings that represent the high-level plan attend to the 3D feature field in the same transformer, so the low-level action is conditioned on both the instruction and the current 3D scene.","core_discovery":"On its own terms, the paper's central discovery is that the gap between high-level reasoning and low-level control can be closed by making the planner multimodal and self-improving and by grounding the controller in a 3D semantic feature field. The planner receives visual feedback after each skill, so it can re-plan on environment disturbances; a critic that does not see the human instruction catches plans misaligned with observations; and stored critiques serve as memory for future planning. The controller back-projects self-supervised 2D features (DINO) into 3D space, fuses them with PointNext geometry features, and uses a 3D transformer in which language embeddings are jointly attended with the point cloud to output the next keyframe pose. The paper reports 90% grasp, 65% place, 80% turn, 40% open, 100% close without distractors, drop of roughly 10-20 points with distractors, and 56.5% average on long-horizon tasks, versus 7% for a SayCan+OWL-v2 baseline and 10% for Voxposer on the same three tasks. The mistakes that remain are attributed to the low-level policy accumulating errors, while the planning component with visual feedback and critic reaches near 100% accuracy in its ablations.","pith_inferences":["If the reported ratios hold under larger trials, a practical implication is that expensive per-task engineering for low-level skills can be replaced by a small demonstration library plus a generic vision-language planner, lowering the marginal cost of adding new kitchen tasks.","The critic-agent design suggests a testable extension: replacing GPT-4V with an open-weight LMM and re-measuring the 1.5x planning gain would show whether the improvement is tied to GPT-4V's specific visual reasoning or generalizes across visual-language models.","The 10-trial evaluation means the 1.45x and 1.5x claims are compatible with wide confidence intervals; a direct replication with 50+ trials per condition would sharpen whether the true effect is as large as stated.","The two-kitchen drop of 10-20 points hints that the semantic 3D representation may be overfitting to visual texture or layout, and an extension is to finetune on a third kitchen to see whether the drop shrinks with more diverse data."],"forward_implications":["Robots using this stack can recover from failed low-level attempts by re-planning rather than repeating the same mistake, as shown in the 'reattempt after failure' experiments with injected noise.","The framework can handle a human instruction that does not match the scene by using a critic that ignores the instruction and checks feasibility against the visual observation.","Low-level skills learned from as few as 5-10 demonstrations per task generalize to new objects and locations, as shown by the pick/place composition table where milk is placed in the sink and drawer despite not being trained there.","Training on two kitchen settings lowers per-skill accuracy by 10-20 points, indicating a measurable generalization cost when scene diversity is increased.","On tasks requiring visual state awareness—placing a pan in a drawer that is initially closed—the planner reaches 80% accuracy versus 50% or 0% for baselines, because only the LMM can see the drawer state."],"supporting_citations":[{"why":"SayCan is the LLM-based planning baseline that LMM-3DP is compared against on high-level planning and long-horizon tasks.","marker":"[1]"},{"why":"Voxposer is the SoTA LLM-based manipulation baseline that the paper claims to surpass in planning accuracy and long-horizon success.","marker":"[16]"},{"why":"GPT-4 is the LMM used as the planner; it supplies the visual reasoning capability that the whole high-level loop depends on.","marker":"[17]"},{"why":"PointNext provides the geometric point cloud features in the low-level policy's two-branch 3D representation.","marker":"[19]"},{"why":"DINO is the self-supervised vision transformer whose 2D features are back-projected into 3D to give the policy its semantic understanding.","marker":"[48]"},{"why":"CLIP's language encoder produces the embedding that conditions the policy on the current instruction or plan.","marker":"[50]"},{"why":"Act3D supplies the coarse-to-fine query point resampling method used for refining keyframe pose prediction.","marker":"[51]"},{"why":"OWL-v2 is the open-vocabulary detector used as the affordance baseline in the skill comparison and long-horizon tasks.","marker":"[52]"}],"fun_headline_variants":["LMM planner + 3D skill policy: 1.5x better robot manipulation","Self-correcting LMM planner + 3D field: 1.5x manipulation gain","Robot kitchen success up 1.5x using LMM planner and 3D skills","LMM-3DP: Visual planner and 3D skill policy improve manipulation 1.5x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every reported accuracy rate comes from only 10 trials, and the paper treats these proportions as reliable estimates of true success probabilities, without any confidence intervals or significance tests.","fun_headline_variants_meta":{"raw":{"variants":["LMM planner + 3D skill policy: 1.5x better robot manipulation","Self-correcting LMM planner + 3D field: 1.5x manipulation gain","Robot kitchen success up 1.5x using LMM planner and 3D skills","LMM-3DP: Visual planner and 3D skill policy improve manipulation 1.5x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000735,"raw_usage":{"total_tokens":3361,"prompt_tokens":1096,"completion_tokens":2265,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":712,"completion_tokens_details":{"reasoning_tokens":2165}},"tokens_in":712,"tokens_out":2265,"duration_ms":12876,"temperature":1.0,"reasoning_tokens":2165,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T22:41:15.344693+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the three long-horizon tasks with 50 trials per condition and compute binomial confidence intervals for the success rates; if the 95% intervals for LMM-3DP and the baselines overlap substantially, the claimed 1.45x and 1.5x advantages are not established.","supporting_citations":[{"cited_title":"Pointnext: Revisiting pointnet++ with improved training and scaling strategies,","cited_arxiv_id":null,"evidence_quote":"PointNext provides the geometric point cloud features in the low-level policy's two-branch 3D representation."},{"cited_title":"Act3d: 3d feature field transformers for multi-task robotic manipulation,","cited_arxiv_id":null,"evidence_quote":"Act3D supplies the coarse-to-fine query point resampling method used for refining keyframe pose prediction."},{"cited_title":"Scaling open-vocabulary object detection,","cited_arxiv_id":null,"evidence_quote":"OWL-v2 is the open-vocabulary detector used as the affordance baseline in the skill comparison and long-horizon tasks."}],"review_version":1}