{"id":"74b690ea-6c67-48a0-b929-55ccaf9d3c88","arxiv_id":"2411.18276","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new synthetic dataset with material-randomized stereo images and part-level action poses improves depth estimation and articulated object manipulation in simulation and real-world tests.","lead":"This paper introduces a large synthetic dataset for robot interactions with household articulated objects, including simulated infrared camera images and billions of labeled robot hand poses. The authors train a depth-repair model and a grasp-pose model on it, and report real-world success on opening and closing objects at 61 percent, versus about 32 percent for the best baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The real-world transfer claim rests on D415 simulator fidelity, yet real-depth evidence is qualitative and the sole quantitative real check is 31 trials.","rationale":"The reader identified the D415 rendering fidelity as the weakest assumption, and my read agrees. The paper's central claim has two real-world pillars: qualitative depth images and a 31-trial manipulation experiment. The manipulation experiment is encouraging but small, and the depth module's contribution is inferred from ablations rather than from quantitative real-depth evaluation. This is not a rejection of the dataset or framework; the synthetic pipeline is plausible and domain randomization can cover some sensor gap, but the evidence as presented does not yet establish how much of the simulated depth improvement transfers. A real-world depth benchmark with ground truth would settle this directly. I do not see a stronger internal inconsistency: the data-split sentence in Section V-A is ambiguous but not clearly fatal, and the pose-prediction simulation results are internally consistent. The absence of released data and code is a reproducibility concern, but it is secondary to the fidelity question because even with code, the simulator-to-real transfer would remain the load-bearing issue. The reader's CONDITIONAL verdict already reflects this uncertainty, so no verdict change is needed.","tokens_in":11420,"tokens_out":4824,"duration_ms":50586,"concrete_test":"Capture real D415 RGB/IR/raw-depth frames for the 7 real-world instances plus additional transparent/reflective household objects, and obtain ground-truth depth from a high-accuracy scanner or CAD alignment. Recompute MAE, REL, and delta-1.25 for raw depth, pre-trained D3RoMa, and D3RoMa fine-tuned on GAPartManip. If fine-tuning does not substantially reduce real-world depth error on transparent/specular parts, the simulated Table I gains are not evidence for zero-shot real-world depth perception.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GAPartManip enables zero-shot sim-to-real depth perception and manipulation depends on the fidelity of the Isaac Sim D415 replica described in Section III-B. The simulator replicates the D415 layout and projects a similar shadow pattern, and material parameters are randomized, but the paper provides no quantitative validation that the simulated IR images and resulting depth corruption match a real D415 on transparent or specular surfaces. Table I shows dramatic simulated depth improvements after fine-tuning on GAPartManip, but this evaluation is self-referential: both training and test images come from the same rendering pipeline. The real-world depth evidence in Fig. 5 is qualitative only, with no ground-truth depth comparison, so the largest simulated gain (84% MAE reduction) is not confirmed on real sensor data. The only quantitative real-world evidence is the manipulation success rate in Table III, which reports 61.29% versus 32.25% over just 31 open/close trials, with no error bars or per-instance breakdown. Because the depth module is load-bearing (removing it drops success from 61.29% to 38.70%), the whole framework's real-world edge depends on whether the simulated depth artifacts actually transfer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GAPartManip, a large synthetic dataset for articulated object manipulation containing 918 object instances across 19 categories, 241,680 rendered RGB/IR/depth samples, and 8 billion scene-level actionable interaction pose annotations. The data generation pipeline combines physics-based simulation of an Intel RealSense D415 structured-light camera with domain-randomized materials and a CUDA-accelerated pose annotation procedure. The authors also propose a modular manipulation framework comprising a D3RoMa-based depth reconstruction module, a Part-aware EcoGrasp pose prediction module, and a CuRobo-based local planner. Evaluations report that fine-tuning depth and pose models on GAPartManip substantially improves simulated depth metrics (Table I) and simulated actionable-pose precision (Table II), and that the full framework achieves a 61.29% real-world open/close success rate over 31 trials versus 32.25% for the best baseline (Table III).","tokens_in":11668,"tokens_out":5565,"duration_ms":50102,"significance":"If the reported results hold up, GAPartManip would be a valuable resource for the community: it is the first articulated-object manipulation dataset to combine material-randomized IR rendering with dense scene-level actionable pose annotations, and the proposed framework demonstrates a promising route to zero-shot sim-to-real manipulation of transparent and reflective parts. The paper also contributes a substantial engineering effort in the CUDA-accelerated pose filtering pipeline (about 150x speedup) and a modular system where each component is evaluated separately and in ablation. The main external validation is the real-world manipulation experiment, which compares against several existing methods. However, the strength of this contribution is currently limited by the thin real-world evidence and the lack of quantitative validation of the simulated D415 sensor model.","major_comments":[{"comment":"The real-world depth evaluation is qualitative only. The paper states that the fine-tuned models \"demonstrate strong depth estimation performance in real-world scenarios\" and that \"our model significantly outperforms the fine-tuned RAFT-Stereo and the raw depth,\" but Figure 5 shows no quantitative comparison (no ground-truth depth, no error metric) on real D415 data. Since the depth module is load-bearing (Table III: removing it drops overall success from 61.29% to 38.70%), the simulated depth gains in Table I (e.g., MAE 0.317 to 0.050) are not confirmed on real sensor data. Please provide quantitative real-world depth evaluation, for example by comparing against a high-accuracy depth sensor or manual ground truth on a small set of real scenes, or at least report per-scene depth errors for the cases shown in Figure 5.","section":"Section V-A and Fig. 5"},{"comment":"The fidelity of the simulated D415 structured-light camera is asserted but not validated. Section III-B states that the authors \"replicate the layout of the D415 imaging system\" and \"project a similar shadow pattern,\" and that material parameters are randomized, but no experiment demonstrates that the resulting simulated IR images and depth corruptions quantitatively match a real D415 on transparent, specular, or metal surfaces. The simulation results in Table I are measured on test images produced by the same rendering pipeline that generated the training data, so they do not by themselves establish sim-to-real transfer. A concrete validation would be to capture real D415 IR and raw depth for a small set of objects (or a re-created scene) and compare the simulated and real sensor outputs, or to measure real-world depth reconstruction error against a reference sensor before and after fine-tuning.","section":"Section III-B"},{"comment":"The real-world manipulation results are based on only 31 total trials (14 open, 17 close) and are reported without error bars, confidence intervals, or a significance test. With n=31, the 95% confidence interval for a 61.29% success rate is roughly [42%, 78%], and for the 32.25% baseline it is roughly [17%, 51%]; the intervals overlap. The claim of a \"significant performance boost\" relative to baselines is therefore not statistically supported. Please report per-instance trial counts, confidence intervals, or a two-proportion test, and if possible increase the number of trials, especially for the Ours vs. GSNet comparison.","section":"Section V-C and Table III"},{"comment":"The baseline comparison in the real-world experiment is underspecified. The text says \"we compare our method with three baselines: Where2Act, AO-Grasp, GSNet, and, like in Sec. V-B, we extend the Where2Act interaction pipeline to finish our tasks.\" It is not clear whether AO-Grasp and GSNet are also given the same post-grasping motion direction estimator, the same CuRobo planner, and the same mechanism for executing open/close actions, or whether they only provide a grasp pose with no articulated-object interaction. If the baselines are not integrated with the same interaction pipeline, the comparison may be unfair. Please describe the exact setup for each baseline, including how the open/close action is determined after grasping.","section":"Section V-C"}],"minor_comments":[{"comment":"The percentage improvements in the text are incorrect. For RAFT-Stereo, MAE goes from 0.618 to 0.309, which is a 50% error reduction (a 100% improvement, not 150%). For D3RoMa, MAE goes from 0.317 to 0.050, which is an 84% error reduction (about 534% improvement, not 600%). Please correct the percentages.","section":"Section V-A"},{"comment":"The word \"TODO\" appears in the framework diagram. This looks like an unfinished annotation and should be removed before publication.","section":"Figure 4"},{"comment":"There are two citation placeholders marked \"[?]\" in the related work. These should be filled in with the intended references.","section":"Section II-A and II-B"},{"comment":"The column header for the Novel split reads \"P P 0.4 P0.8\" while the Seen and Unseen splits read \"P P 0.8 P0.4\". If this is not a deliberate reordering, it is a typo and should be corrected.","section":"Table II"},{"comment":"The paper states the real-world experiments use \"7 distinct instances, including StorageFurniture, Box, and Microwave\" but does not report how many trials were conducted per instance or per object. Providing a per-instance breakdown would help readers assess where the method succeeds and fails.","section":"Section V-C"}],"recommendation":"major_revision","confidential_remarks":"The paper describes a potentially useful dataset and a modular pipeline, but the current real-world validation is too thin to support the strong SOTA claims: the real-world depth result is qualitative, the manipulation success is based on 31 trials without statistics, and the D415 simulator fidelity is not quantitatively checked. The leftover \"TODO\" and missing references also suggest the manuscript was not fully polished. I would encourage the authors to add quantitative real-world depth evaluation, expand or statistically analyze the real-world trials, clarify the baseline integration, and correct the percentage calculations. With those additions, the paper could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good honest take on a dataset paper that is better than its presentation. The core artifact is GAPartManip: 918 objects across 19 categories, ~240K rendered frames with both RGB and IR stereo for a simulated D415, plus 8B scene-level actionable poses. That combination—articulated objects, material randomization, and IR rendering—is genuinely new. The modular framework built on it, fine-tuning D3RoMa for depth and adapting EconomicGrasp to predict actioness, is reasonable. Simulation numbers are large and internally consistent: depth MAE drops 84% and pose precision roughly doubles after fine-tuning. Credit where due: this is a serious attempt to address a real problem, imperfect depth on transparent and reflective parts.\n\nNow the soft spots. The real-world evaluation is the load-bearing evidence for 'material-agnostic' and it is thin: 31 trials, no confidence intervals, and the depth module's real-world improvement is shown only qualitatively. The simulated depth evaluation uses test images from the same rendering pipeline that produced the training data, so part of the 84% MAE gain is self-referential. The D415 replica in Isaac Sim is plausible but not quantitatively validated against a real sensor on transparent or specular materials. The paper also has a placeholder citation in the related work and a 'TODO' in Figure 4—minor, but it suggests a final polish was skipped. Dataset and code are promised but not released, and the paper's usefulness as a benchmark depends on that release.\n\nNone of these are fatal on their own. The real-world manipulation results, though small, favor the proposed pipeline over strong baselines, and the ablation removing the depth module shows it matters. The concern that the D415 simulator fidelity is unproven is real, but it is a limitation to be fixed with more data, not a flaw in the core idea. If the data ships, this could become a standard training and evaluation resource for articulated-object manipulation.\n\nWho should read it: anyone working on sim-to-real depth for structured-light cameras, or on part-centric grasping. It deserves a serious referee—conditional on the authors releasing the data and tightening the real-world evaluation. I would not desk-reject it; I would send it out.","headline":"Useful new dataset and a plausible framework for material-agnostic articulated object manipulation, but the real-world evaluation is too thin to carry the sim-to-real claim as it stands.","tokens_in":12191,"tokens_out":2153,"would_cite":false,"duration_ms":21025,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GAPartManip claims that a synthetic dataset with material-randomized IR rendering and part-level interaction pose labels enables zero-shot sim-to-real manipulation of articulated objects, achieving 61.29% real-world success.","keywords":["articulated object manipulation","sim-to-real transfer","depth estimation","actionable interaction pose","structured light camera","domain randomization","part-centric dataset","material-agnostic perception"],"falsifier":"Collect real Intel RealSense D415 IR and depth captures of transparent-lidded and reflective-handled articulated objects under lighting conditions not represented in the 20 simulated scenes, and measure the depth error of the GAPartManip-trained depth model against the raw sensor output; if the model is not consistently better than raw depth on these materials, the material-randomized IR simulation is not capturing the sensor's real failure modes.","tokens_in":11251,"feed_emoji":"🤖","tokens_out":8158,"duration_ms":72002,"temperature":0.7,"pith_summary":"GAPartManip is a synthetic dataset built to test a data-centric hypothesis: that the gap between simulation and real-world manipulation of articulated objects is mostly a depth-sensing gap, and that it can be closed by training on infrared images that reproduce how structured-light cameras fail on transparent and reflective materials. The paper reports that fine-tuning a diffusion-based depth estimator and a part-aware grasp pose predictor on this dataset sharply reduces depth error in simulation and raises real-world open/close manipulation success to 61.29%, compared with 32.25% for the best baseline. The claim is not a new algorithm but a new data resource: 918 objects from 19 household categories, 240K rendered images, and over 8 billion scene-level actionable interaction poses.","feed_headline":"Synthetic infrared data lifts real-world robot success to 61.3%","feed_subtitle":"Training depth and part-aware pose networks on 240K randomized renders beats baselines without any real robot data.","key_machinery":"The load-bearing mechanism is the material-randomized IR rendering pipeline built on NVIDIA Isaac Sim, which replicates the Intel RealSense D415's structured-light imaging layout (IR projector, two IR cameras, projected shadow pattern) and randomizes diffuse, transparent, specular, and metal material parameters, ambient lighting, backgrounds, and joint states across 20 indoor scenes. The second mechanism is the CUDA-accelerated scene-level pose annotation pipeline, which projects part-level antipodal grasps into the scene and filters them by single-view visibility and collision, reducing annotation time from five minutes to under two seconds per part and producing over 8 billion actionable poses. Together these two mechanisms turn a simulator into a training distribution that is deliberately material-agnostic, so models trained on it do not assume clean depth or reflective surfaces.","core_discovery":"The central claim is that a large-scale part-centric dataset with physics-based IR rendering and scene-level actionable interaction pose annotations is sufficient to make articulated object manipulation work zero-shot in the real world, even when object parts are transparent or reflective. On the depth side, fine-tuning D3RoMa on GAPartManip improves simulated MAE from 0.317 to 0.050 and EPE from 2.82 to 0.69, and qualitative real-world captures show cleaner depth on transparent lids and small handles. On the pose side, the proposed Part-aware EcoGrasp, trained on the dataset's actioness labels, reaches 55.33% precision on seen instances and 41.65% on novel instances, versus 24.72% and 14.56% for the untrained EconomicGrasp baseline. Combined with a motion-direction estimator and a local planner, the full framework achieves 61.29% success across 31 real-world open/close trials, against 32.25% for the best baseline.","pith_inferences":["Editorial inference: if the D415 simulation is faithful, the same material-randomized IR pipeline could transfer to other structured-light depth sensors, because the domain randomization targets sensor physics rather than category-specific appearance; this is testable by fine-tuning the same models on synthetic IR from a different camera model.","Editorial inference: the reported real-world advantage rests on 31 trials across 7 object instances, so a larger multi-scene evaluation would be needed to confirm the magnitude of the gap over baselines, which the paper does not provide.","Editorial inference: the pose annotations are designed for a single parallel-jaw gripper with antipodal scoring, so extending the dataset to suction or multi-finger grippers would require re-annotation, though the part-level actionable pose representation is a plausible starting point."],"forward_implications":["Fine-tuning depth networks on GAPartManip cuts simulated mean depth error roughly sixfold (MAE from 0.317 to 0.050), a direct consequence of the material-randomized IR data.","Pose predictors trained on the dataset learn to score parts by actionability rather than graspability alone, so predicted poses open and close articulated objects instead of merely holding them; the paper reports 55.33% precision on seen instances versus 24.72% for the same architecture without dataset training.","The full pipeline transfers zero-shot to a real robot arm and Intel RealSense D415, reaching 61.29% success on 31 open/close trials, more than the 32.25% of the best baseline.","Because the dataset covers 19 categories with seen, similar, and novel instance splits, models trained on it generalize to object instances not used in training."],"supporting_citations":[{"why":"Supplies the object assets, 19 categories, part semantics, and the pre-trained motion-direction estimator that GAPartManip inherits and builds upon.","marker":"[9]"},{"why":"Provides the diffusion-based stereo depth estimator architecture that is fine-tuned on GAPartManip and used as the depth reconstruction module.","marker":"[14]"},{"why":"Supplies the antipodal grasp scoring, pose sampling strategy, and precision metrics used to create the dataset's actionable pose annotations.","marker":"[33]"},{"why":"Provides the EconomicGrasp architecture that is adapted into Part-aware EcoGrasp by estimating actioness instead of graspness.","marker":"[40]"},{"why":"Prior work on stereo IR rendering of transparent and specular objects with domain randomization, informing the IR simulation pipeline.","marker":"[15]"},{"why":"The simulator used to build the photo-realistic RGB/IR rendering pipeline and replicate the D415 imaging layout.","marker":"[38]"},{"why":"Learning-based stereo baseline whose fine-tuned version also improves on GAPartManip, supporting the dataset's depth benefit.","marker":"[43]"},{"why":"Rigid-object grasp pose baseline compared in simulation and real-world trials, and fine-tuned on the dataset in experiments.","marker":"[45]"}],"fun_headline_variants":["Dataset with 240K renders enables zero-shot robot manipulation","Part-centric dataset beats depth and pose baselines on real robots","Material-agnostic dataset trains robots to open transparent objects","Synthetic IR data makes real robot manipulation zero-shot","GAPartManip: 240K renders for robust articulated object control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the simulator's modeled failures of the Intel RealSense D415 on transparent and reflective surfaces being close enough to the real camera's failures that models trained on synthetic IR images transfer to real IR images.","fun_headline_variants_meta":{"raw":{"variants":["Dataset with 240K renders enables zero-shot robot manipulation","Part-centric dataset beats depth and pose baselines on real robots","Material-agnostic dataset trains robots to open transparent objects","Synthetic IR data makes real robot manipulation zero-shot","GAPartManip: 240K renders for robust articulated object control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000724,"raw_usage":{"total_tokens":3250,"prompt_tokens":955,"completion_tokens":2295,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":2210}},"tokens_in":571,"tokens_out":2295,"duration_ms":15329,"temperature":1.0,"reasoning_tokens":2210,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:21:24.315526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect real Intel RealSense D415 IR and depth captures of transparent-lidded and reflective-handled articulated objects under lighting conditions not represented in the 20 simulated scenes, and measure the depth error of the GAPartManip-trained depth model against the raw sensor output; if the model is not consistently better than raw depth on these materials, the material-randomized IR simulation is not capturing the sensor's real failure modes.","supporting_citations":[{"cited_title":"Gapartnet: Cross-category domain-generalizable object perception and manipulation via generalizable and actionable parts,","cited_arxiv_id":null,"evidence_quote":"Supplies the object assets, 19 categories, part semantics, and the pre-trained motion-direction estimator that GAPartManip inherits and builds upon."},{"cited_title":"D3roma: Disparity diffusion-based depth sensing for material-agnostic robotic manipulation,","cited_arxiv_id":null,"evidence_quote":"Provides the diffusion-based stereo depth estimator architecture that is fine-tuned on GAPartManip and used as the depth reconstruction module."},{"cited_title":"Graspnet-1billion: A large- scale benchmark for general object grasping,","cited_arxiv_id":null,"evidence_quote":"Supplies the antipodal grasp scoring, pose sampling strategy, and precision metrics used to create the dataset's actionable pose annotations."},{"cited_title":"An Economic Framework for 6-DoF Grasp Detection","cited_arxiv_id":"2407.08366","evidence_quote":"Provides the EconomicGrasp architecture that is adapted into Part-aware EcoGrasp by estimating actioness instead of graspness."},{"cited_title":"Asgrasp: Generalizable transparent object reconstruction and 6-dof grasp detection from rgb-d active stereo camera,","cited_arxiv_id":null,"evidence_quote":"Prior work on stereo IR rendering of transparent and specular objects with domain randomization, informing the IR simulation pipeline."},{"cited_title":"Gpu-accelerated robotic simulation for distributed reinforce- ment learning,","cited_arxiv_id":null,"evidence_quote":"The simulator used to build the photo-realistic RGB/IR rendering pipeline and replicate the D415 imaging layout."},{"cited_title":"Raft-stereo: Multilevel recurrent field transforms for stereo matching,","cited_arxiv_id":null,"evidence_quote":"Learning-based stereo baseline whose fine-tuned version also improves on GAPartManip, supporting the dataset's depth benefit."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Rigid-object grasp pose baseline compared in simulation and real-world trials, and fine-tuned on the dataset in experiments."}],"review_version":1}