Pith. sign in

REVIEW 3 major objections 5 minor 53 references

Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Prompting a vision-language model with rendered futures from a physics-based digital twin yields better zero-shot manipulation than baselines that ask the model to reason about actions directly.

desk verdict A genuinely useful division of labor between VLM semantics and simulated physics, held back by thin statistics and unmeasured sim-to-real fidelity. read the letter →

arxiv 2506.13761 v1 pith:EO2GQW5E submitted 2025-06-16 cs.RO

Chuanruo Ning , Kuan Fang , Wei-Chiu Ma This is my paper
classification cs.RO
keywords open-worldmanipulationvision-languagemodelsdigitaltwinmodelpredictivecontrolGaussiansplattingsampling-basedplanningzero-shotlanguage-conditioned
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prompting with the Future (PWTF) tries to show that a vision-language model (VLM) can plan precise, contact-rich robot manipulation without ever being asked to reason about physics. The system scans a tabletop scene with a handheld video, builds an interactive digital twin whose meshes simulate dynamics and whose Gaussian splats render photorealistic futures, samples candidate gripper actions, simulates them, and prompts the VLM with images of the resulting future states. The VLM only evaluates which simulated outcome best matches a free-form language instruction, and a sampling-based optimizer selects the corresponding action. Across eight real-world tasks, PWTF reports 47 successes in 80 trials, versus 22 for the strongest baseline it compares against, with no robot demonstrations, fine-tuning, or in-context examples. If the claim holds, it points to a control architecture where physical prediction lives in a simulator and semantics live in the VLM.

What carries the argument

The interactive digital twin is the load-bearing object: a hybrid scene model in which a truncated signed-distance-field (TSDF) mesh, derived from 2D Gaussian splatting, provides geometry for a rigid-body simulator, while Gaussian splats anchored to movable meshes provide photorealistic rendering that follows object motion. Movable objects are identified by prompting a VLM to name them, another VLM to point at them, and a tracking segmenter to produce masks that are projected into 3D. Around the twin, a model-predictive loop samples actions, simulates their outcomes, renders multi-view futures, lets the VLM select the most informative viewpoint and the outcome closest to the current subtask, and refines the action distribution with the Cross-Entropy Method. The operative principle is that the VLM never sees the current frame with overlaid action arrows; it sees the future it would cause.

What would settle it

Take one contact-heavy task (for example, cleaning spilled tea with a sponge), run the full PWTF pipeline, and then rerun it with deliberately corrupted simulator physics—halved friction or doubled object mass—while keeping the rendered images visually realistic. If real-world success stays roughly constant, then physical grounding is not what is doing the work; if success drops, the paper's causal story is confirmed. A complementary measurement is to execute the same planned action in simulation and reality and compare the resulting object poses; large divergence without a success drop would also falsify the claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that decoupling semantic understanding from physical prediction is what lets a pre-trained VLM guide low-level manipulation in open worlds. Instead of requiring the VLM to output value maps, keypoints, or actions, PWTF generates candidate futures in a physics simulator, renders them from adaptively chosen viewpoints, and asks the VLM to choose the future most aligned with the instruction. The authors treat this as making manipulation an in-distribution visual-evaluation task for the VLM. Their ablations attribute the gains to the simulated futures: removing the cross-entropy optimization, fixing a single viewpoint, or skipping subgoal decomposition all reduce success, and contact-rich or precise-alignment tasks such as pressing a spacebar or hitting the lowest xylophone bar succeed only when the simulated futures are present.

Load-bearing premise

The load-bearing assumption is that the digital twin's physics matches the real world closely enough that a rendered future is the future the robot would actually cause; if the sponge slides or the grasp fails differently in simulation than reality, the VLM is evaluating outcomes that will not happen.

Editorial extensions

If this is right

  • If PWTF is right, a new manipulation setup can go from a video scan and a language instruction to closed-loop control without collecting robot data or updating model weights.
  • Physical realism of the twin becomes the bottleneck: better reconstruction and simulation should translate directly into higher task success, independent of VLM progress.
  • Tasks that resist parameterization by value maps or keypoints, such as 'play the lowest pitch with the drum stick,' become addressable because evaluation happens on images of outcomes.
  • The VLM's role as critic means future improvements in VLM perception should improve planning without any change to the control loop.
  • The reported 47/80 versus 22/80 aggregate implies the method roughly doubles the success rate of the best compared VLM-prompting baseline across the task suite.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: if simulator fidelity is the limiting factor, then deliberately corrupting twin physics (lowering friction, changing mass) while rendering looks unchanged should measurably reduce real-world success; the paper does not run this perturbation test.
  • My inference: the same digital-twin-plus-VLM-critic loop should extend to articulated or deformable objects once simulators support them, because the VLM only compares rendered outcomes and never needs a new action representation.
  • My inference: closed-loop replanning may compensate for some reconstruction or execution error by re-simulating from a fresh observation each step, so the system's robustness could be higher than a one-shot open-loop sim-to-real transfer; the paper does not isolate this effect.
  • My inference: aggregate success hides task-level variance, so practical deployment should focus on the failure categories the paper identifies (reconstruction, sampling, execution, planning) rather than the average.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Prompting with the Future (PWTF), a model-predictive control framework that constructs an interactive digital twin of a real-world tabletop scene from a handheld video scan, simulates candidate robot actions in the ManiSkill2 physics simulator, renders the predicted future states using a hybrid Gaussian-splatting/mesh representation from adaptively selected viewpoints, and uses GPT-4o as a critic to select the simulated outcome best aligned with a natural-language instruction. Action selection is performed with three CEM iterations over 90 sampled candidates per step. The method requires no robot demonstrations, fine-tuning, or in-context examples. Experiments on eight real-world tasks with 10 trials per task report higher success rates (47/80 aggregate) than VoxPoser*, MOKA, OpenVLA, π0, and their task-finetuned variants, with ablations and a failure breakdown.

Significance. The core idea—using a physics-based digital twin to generate action-conditioned future observations for VLM evaluation—is timely and addresses a real limitation of VLM-only control. The real-robot evaluation covers diverse contact-rich tasks and includes strong baselines, including VLA models with task-specific fine-tuning, which strengthens the comparison. The ablations and backbone-agnostic reconstruction experiments are also positive features. However, the headline claim of statistically significant improvement is not backed by appropriate tests, and the central assumption that simulated dynamics faithfully mirror real mechanics is never directly quantified. If these gaps are closed, the contribution would be substantial for open-world manipulation.

major comments (3)
  1. [IV-B and IV-C] The method's central premise is that the simulator transition T = S(M,a) faithfully predicts real-world outcomes, and all planning is optimized against this model. Yet no experiment quantifies sim-to-real fidelity. The failure taxonomy in Section V-E (Reconstruction 27.3%, Execution 15.2%, Sampling 45.4%, Planning 12.1%) does not include a simulation-mismatch category, and the paper does not compare simulated object states against real executed outcomes for the same actions. Without such a comparison, the reported gains in Table II could stem largely from the VLM evaluating rendered futures that are not physically faithful, weakening the claim that physics grounding is the source of improvement. The authors should provide quantitative sim-to-real fidelity metrics (e.g., object pose errors after executing planned actions in simulation and reality) or a dedicated analysis that separates simulation error from planning error.
  2. [Table II and Abstract] The abstract states that PWTF demonstrates 'significantly higher success rates' than baselines, but the experiments use 10 trials per task with no confidence intervals or significance tests. Many per-task gaps are small (e.g., Water plant: 5/10 vs 6/10 for VoxPoser*; Play drum: 6/10 vs 5/10). The aggregate 47/80 vs 22/80 is suggestive, but without appropriate statistical analysis (e.g., Wilson intervals, Fisher's exact test, or a paired test across scenes) the claim of significance is unsupported. The authors should either add such analyses or temper the wording to 'higher observed success rates' and discuss the uncertainty.
  3. [Section V-E] The failure categories are coarse and may conflate distinct causes. In particular, 'Sampling error' (45.4%) is attributed to 'inherent randomness of action sampling and errors in VLM reasoning' without considering that inaccurate simulated outcomes (a sim-to-real mismatch) would also cause the VLM to reject actions whose real-world effects differ from the renderings. The authors should clarify how each failure was classified (e.g., manual labeling criteria, inter-rater agreement) and whether any failures were attributed to simulation-reality mismatch; otherwise the taxonomy cannot support the conclusion that the digital twin is adequately accurate for planning.
minor comments (5)
  1. [References] References [9] and [10] both cite the Molmo paper, and [43] and [44] both cite 'Distilled Feature Fields'; please consolidate duplicate references.
  2. [Appendix B.1] There is a typo in the real-world settings paragraph: 'in dthe igital twin' should read 'in the digital twin'.
  3. [Table II] The table and text use inconsistent notation 'V oxposer*' instead of 'VoxPoser*'.
  4. [Abstract and Section I] The phrase 'significantly higher success rates' should be softened to match the statistical evidence (see Major Comment 2).
  5. [Section V-A] The tasks are described as requiring '6 DoF control', but the action space is defined as a 7-dimensional vector in Section III (6-DoF pose plus finger status); please reconcile the terminology.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the method is a self-contained pipeline evaluated by real-world task success, with no fitted parameters or load-bearing self-citations.

full rationale

The paper's derivation chain is: video scan -> hybrid Gaussian/mesh reconstruction -> VLM-based object segmentation -> ManiSkill2 physics simulation -> rendering of simulated futures -> VLM evaluation for CEM planning -> real-robot execution. Each stage has independent content, and none of the stages is defined in terms of the final success criteria. No parameter is fitted to the Table II outcomes or to the baseline results; the VLM is used inside the loop as an evaluator of rendered futures, which is the proposed method rather than a manipulation of the benchmark. The success criteria are human-defined real-world outcomes, and success is measured by physical task completion, not by the VLM's own ratings. The only prior work by the same authors cited in a comparative role is MOKA [14], which is used as a baseline to compare against, not as a load-bearing justification for PWTF; even if one counted this as a self-citation, it does not support the central claim. The unquantified sim-to-real fidelity of the digital twin is a genuine correctness and robustness concern (the paper's own failure breakdown in Figure 6 lists Reconstruction error 27.3% and Execution error 15.2%), but it is not circularity: the claim is that the overall pipeline achieves real-world success, and an inaccurate simulator would weaken that claim rather than being assumed true by definition. The ablations (w/o views, w/o subtasks, w/o CEM) further show that the components are individually removable and that the full system's advantage is not an artifact of a single fitted quantity. Therefore the derivation is self-contained against external task evaluation, and no step reduces to its own inputs by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The system rests on a chain of learned and hand-tuned components: reconstruction fidelity, physics simulator accuracy, VLM judgment, and segmentation reliability. No numbers are fitted to the success criteria, so circularity is low, but the dependencies are numerous.

free parameters (5)
  • CEM iterations = 3
    Chosen by hand, not fitted to outcome data; affects planning quality and latency (Section IV-C).
  • Candidates per CEM iteration = 90
    Chosen by hand; larger values improve coverage but increase simulation load (Section IV-C).
  • Candidate group size for VLM comparison = 5
    Chosen by hand; determines how many outcomes the VLM compares at once (Appendix A.2).
  • Number of rendered camera views = 4
    Fixed set (front, left, right, top-down); chosen by hand, not optimized (Appendix A.2).
  • Planning step budget = 30
    Defines trial timeout; chosen to bound experiment time at 5 minutes (Appendix B.2).
assumptions (5)
  • domain assumption ManiSkill2 physics simulation accurately reproduces real-world object dynamics for the tested objects.
    Used to generate future states from candidate actions (Section IV-A, IV-B). If sim-to-real gap is large, VLM evaluations of simulated futures may not transfer.
  • domain assumption 2D Gaussian Splatting with TSDF integration yields geometry accurate enough for physical simulation.
    Reconstruction pipeline (Section IV-A) creates meshes for simulation; reconstruction errors are acknowledged in failure analysis (Section V-E).
  • domain assumption GPT-4o's evaluation of rendered future states is a reliable proxy for physical task success.
    The VLM selects elite actions and subtasks; no calibration against real outcomes is reported (Section IV-C).
  • domain assumption Molmo and SAM2 correctly identify and segment movable objects from the task instruction.
    Object segmentation relies on these models (Section IV-A); failures would mislabel interactable objects.
  • domain assumption COLMAP camera pose estimation and chessboard calibration align the twin with the real robot frame.
    Planned trajectories are copied from twin to reality via joint angles (Appendix B.1); misalignment would cause execution errors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins." pith.science (2026). https://pith.science/paper/EO2GQW5E

@misc{pith2026250613761,
  author       = {Pith},
  title        = {Pith review of: Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EO2GQW5E}},
  note         = {Machine review of arXiv:2506.13761}
}
read the original abstract

Recent advancements in open-world robot manipulation have been largely driven by vision-language models (VLMs). While these models exhibit strong generalization ability in high-level planning, they struggle to predict low-level robot controls due to limited physical-world understanding. To address this issue, we propose a model predictive control framework for open-world manipulation that combines the semantic reasoning capabilities of VLMs with physically-grounded, interactive digital twins of the real-world environments. By constructing and simulating the digital twins, our approach generates feasible motion trajectories, simulates corresponding outcomes, and prompts the VLM with future observations to evaluate and select the most suitable outcome based on language instructions of the task. To further enhance the capability of pre-trained VLMs in understanding complex scenes for robotic control, we leverage the flexible rendering capabilities of the digital twin to synthesize the scene at various novel, unoccluded viewpoints. We validate our approach on a diverse set of complex manipulation tasks, demonstrating superior performance compared to baseline methods for language-conditioned robotic control using VLMs.

Figures

Figures reproduced from arXiv: 2506.13761 by the authors.

Figure 1
Figure 1. Prompting with the Future. We enable VLM-driven motion planning conditioned on free-form instructions by prompting with simulated future outcomes generated from an interactive digital twin. In contrast, previous methods [21, 14, 32] prompt VLMs with current observations and hand-defined action primitives, demanding implicit physical reasoning that VLMs often struggle to perform reliably. Please use Adobe Acrobat Rea… view at source ↗
Figure 2
Figure 2. Construction of interactive digital twins. Starting from a video scan of the environment, we construct an interactive digital twin that combines mesh-based simulation and Gaussian-based rendering. The resulting twin enables photorealistic rendering and accurate simulation of object dynamics conditioned on robot actions. by the digital twin, enabling adaptive observation and action optimization. Section IV-C. A. Cons… view at source ↗
Figure 3
Figure 3. Model Predictive Control through Simulation-Informed Prompting. Given a free-form instruction, our framework first performs high-level planning by generating structured subtasks from multi-view observations. At each step, the interactive digital twin simulates future states of candidate actions and renders the outcomes. The VLM adaptively selects the most informative view for rendering and evaluates the predicted ou… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Example on action optimization. We show the action optimization results of one planning step in subtask “wipe the spilled tea”. Our digital twin could simulate diverse results with accurate motion and collision of the sponge in initial sampling and VLM could effectivel…
Figure 5
Figure 5. Figure 5: Example trajectories. Example trajectories planned by our framework in both digital twin and real world (aligned). We highlight some key steps where VLM chooses to adaptively change the rendering view for better result evaluation (e.g., aligning the gripper from differ…
Figure 6
Figure 6. Figure 6: Failure analysis. Our main failure cases can be divided into four categories. We show the percentage and provide an example for each failure type. allows the VLM to focus on more concrete evaluation criteria for each state, which is beneficial for tasks involving multi…
Figure 7
Figure 7. Figure 7: Examples of task layouts. Random object layout and visual distractors for “press spacebar”, “pair up shoes”, and “cucumber in basket”. of which are divided into groups of 5 for evaluating the best outcome. C. Computational details Our entire pipeline runs on a single N…
Figure 8
Figure 8. Figure 8: Example trajectories. Example trajectories planned by our framework in both the digital twin and the real world. The camera views in the digital twin are chosen by VLM at each step, some planning steps are omitted for clear visualization [PITH_FULL_IMAGE:figures/full_…
Figure 9
Figure 9. Figure 9: Reconstruction quality under different configura￾tions and backbones. Our framework can easily be sped up by trading off quality and time or incorporating faster techniques. Methods Water plant Clean up Cucumber in basket SVRaster [45] 4/10 6/10 7/10 Gaussian Splatting…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 27 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Do as i can, not as i say: Grounding language in robotic affordances

    Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Haus- man, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv:2204.01691, 2022

    arXiv 2022

  3. [3]

    Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields

    Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021

    work page 2021

  4. [4]

    Mip-nerf 360: Unbounded anti-aliased neural radiance fields

    Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022

    2022

  5. [5]

    Sliceit!: Simulation-based reinforce- ment learning for compliant robotic food slicing

    Cristian Camilo Beltran-Hernandez, Nicolas Erbetti, and Masashi Hamaya. Sliceit!: Simulation-based reinforce- ment learning for compliant robotic food slicing. In ICRA Workshop, 2024

    work page 2024

  6. [6]

    π0: A vision-language-action flow model for general robot control

    Kevin Black et al. π0: A vision-language-action flow model for general robot control. arXiv:2410.24164, 2024

    arXiv 2024

  7. [7]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv:2307.15818, 2023

    arXiv 2023

  8. [8]

    Automated creation of digital cousins for robust policy learning

    Tianyuan Dai, Josiah Wong, Yunfan Jiang, Chen Wang, Cem Gokmen, Ruohan Zhang, Jiajun Wu, and Li Fei-Fei. Automated creation of digital cousins for robust policy learning. arXiv:2410.07408, 2024

    arXiv 2024

Show all 53 references
  1. [10]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv:2409.17146, 2024

    2024 arXiv

  2. [11]

    Palm-e: An embodied multimodal language model

    Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv:2303.03378, 2023

    2023 arXiv

  3. [12]

    Manipulate-anything: Automating real-world robots us- ing vision-language models

    Jiafei Duan, Wentao Yuan, Wilbert Pumacay, Yi Ru Wang, Kiana Ehsani, Dieter Fox, and Ranjay Krishna. Manipulate-anything: Automating real-world robots us- ing vision-language models. arXiv:2406.18915, 2024

    2024 arXiv

  4. [13]

    Visual foresight: Model-based deep reinforcement learning for vision- based robotic control

    Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision- based robotic control. arXiv:1812.00568, 2018

    2018 arXiv

  5. [14]

    Moka: Open-vocabulary robotic manipulation through mark-based visual prompting

    Kuan Fang, Fangchen Liu, Pieter Abbeel, and Sergey Levine. Moka: Open-vocabulary robotic manipulation through mark-based visual prompting. In RSS, 2024

    2024

  6. [15]

    Deep visual foresight for planning robot motion

    Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In ICRA, 2017

    2017

  7. [16]

    Physically grounded vision-language models for robotic manipulation

    Jensen Gao, Bidipta Sarkar, Fei Xia, Ted Xiao, Jiajun Wu, Brian Ichter, Anirudha Majumdar, and Dorsa Sadigh. Physically grounded vision-language models for robotic manipulation. In ICRA, 2024

    2024

  8. [17]

    Feedback mpc for torque-controlled legged robots

    Ruben Grandia, Farbod Farshidian, Ren ´e Ranftl, and Marco Hutter. Feedback mpc for torque-controlled legged robots. In IROS, 2019

    2019

  9. [18]

    Maniskill2: A unified benchmark for generalizable manipulation skills

    Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills. In ICLR, 2023

    2023

  10. [19]

    2d gaussian splatting for geometri- cally accurate radiance fields

    Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometri- cally accurate radiance fields. In SIGGRAPH, 2024

    2024

  11. [20]

    Diffusion-based generation, optimization, and planning in 3d scenes

    Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, and Song-Chun Zhu. Diffusion-based generation, optimization, and planning in 3d scenes. In CVPR, 2023

    2023

  12. [21]

    V oxposer: Composable 3d value maps for robotic manipulation with language models

    Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. In CoRL, 2023

    2023

  13. [22]

    Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation

    Wenlong Huang, Chen Wang, Yunzhu Li, Ruohan Zhang, and Li Fei-Fei. Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation. In CoRL, 2024

    2024

  14. [23]

    Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning

    Zhenyu Jiang, Yuqi Xie, Kevin Lin, Zhenjia Xu, Weikang Wan, Ajay Mandlekar, Linxi Fan, and Yuke Zhu. Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning. arXiv:2410.24185, 2024

    2024 arXiv

  15. [24]

    3d gaussian splatting for real-time radiance field rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG, 2023

    2023

  16. [25]

    Robot see robot do: Imitating articulated object manipu- lation with monocular 4d reconstruction

    Justin Kerr, Chung Min Kim, Mingxuan Wu, Brent Yi, Qianqian Wang, Ken Goldberg, and Angjoo Kanazawa. Robot see robot do: Imitating articulated object manipu- lation with monocular 4d reconstruction. In CoRL, 2024

    2024

  17. [26]

    Openvla: An open-source vision-language-action model

    Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv:2406.09246, 2024

    2024 arXiv

  18. [27]

    3d neural scene rep- resentations for visuomotor control

    Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene rep- resentations for visuomotor control. In CoRL, 2022

    2022

  19. [28]

    Code as policies: Language model programs for embod- ied control

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embod- ied control. In ICRA, 2023

    2023

  20. [29]

    Nerf: Representing scenes as neural radiance fields for view synthesis

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020

    2020

  21. [30]

    Model predictive robot-environment interaction control for mobile manip- ulation tasks

    Maria Vittoria Minniti, Ruben Grandia, Kevin F ¨ah, Far- bod Farshidian, and Marco Hutter. Model predictive robot-environment interaction control for mobile manip- ulation tasks. In ICRA, 2021

    2021

  22. [31]

    Embodiedgpt: Vision-language pre- training via embodied chain of thought

    Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre- training via embodied chain of thought. In NeurIPS, 2024

    2024

  23. [32]

    Pivot: Iterative visual prompting elicits actionable knowledge for vlms

    Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv:2402.07872, 2024

    2024 arXiv

  24. [33]

    Safe and fast tracking on a robot manipulator: Robust mpc and neural network control

    Julian Nubert, Johannes K ¨ohler, Vincent Berenz, Frank Allg¨ower, and Sebastian Trimpe. Safe and fast tracking on a robot manipulator: Robust mpc and neural network control. RA-L, 2020

    2020

  25. [34]

    Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction

    Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In ICCV, 2021

    2021

  26. [35]

    Open x-embodiment: Robotic learning datasets and rt-x models

    Abby O’Neill, Abdul Rehman, Abhinav Gupta, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abra- ham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv:2310.08864, 2023

    arXiv

  27. [36]

    A real-to-sim-to-real approach to robotic manipulation with vlm-generated iterative keypoint re- wards

    Shivansh Patel, Xinchen Yin, Wenlong Huang, Shubham Garg, Hooshang Nayyeri, Li Fei-Fei, Svetlana Lazebnik, and Yunzhu Li. A real-to-sim-to-real approach to robotic manipulation with vlm-generated iterative keypoint re- wards. In CoRL Workshop, 2024

    2024

  28. [37]

    Tiebot: Learning to knot a tie from visual demonstration through a real-to-sim-to-real approach

    Weikun Peng, Jun Lv, Yuwei Zeng, Haonan Chen, Siheng Zhao, Jichen Sun, Cewu Lu, and Lin Shao. Tiebot: Learning to knot a tie from visual demonstration through a real-to-sim-to-real approach. arXiv:2407.03245, 2024

    2024 arXiv

  29. [38]

    Splatsim: Zero-shot sim2real transfer of rgb manipulation policies using gaussian splatting

    Mohammad Nomaan Qureshi, Sparsh Garg, Fran- cisco Yandun, David Held, George Kantor, and Ab- hisesh Silwal. Splatsim: Zero-shot sim2real transfer of rgb manipulation policies using gaussian splatting. arXiv:2409.10161, 2024

    2024 arXiv

  30. [39]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Rong- hang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R ¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Doll ´ar, and Christoph Fei...

    2024 arXiv

  31. [40]

    The cross-entropy method for com- binatorial and continuous optimization

    Reuven Rubinstein. The cross-entropy method for com- binatorial and continuous optimization. Methodology and computing in applied probability , 1999

    1999

  32. [41]

    Structure-from-motion revisited

    Johannes Lutz Sch ¨onberger and Jan-Michael Frahm. Structure-from-motion revisited. In CVPR, 2016

    2016

  33. [42]

    Pixelwise view selection for unstructured multi-view stereo

    Johannes Lutz Sch ¨onberger, Enliang Zheng, Marc Polle- feys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In ECCV, 2016

    2016

  34. [44]

    Distilled feature fields enable few-shot language-guided manipulation

    William Shen, Ge Yang, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, and Phillip Isola. Distilled feature fields enable few-shot language-guided manipulation. In CoRL, 2023

    2023

  35. [45]

    Sparse voxels rasterization: Real-time high-fidelity radiance field rendering

    Cheng Sun et al. Sparse voxels rasterization: Real-time high-fidelity radiance field rendering. CVPR, 2025

    2025

  36. [46]

    Robot learning with super-linear scaling

    Marcel Torne, Arhan Jain, Jiayi Yuan, Vidaaranya Macha, Lars Ankile, Anthony Simeonov, Pulkit Agrawal, and Abhishek Gupta. Robot learning with super-linear scaling. arXiv:2412.01770, 2024

    2024

  37. [47]

    Rec- onciling reality through simulation: A real-to-sim-to-real approach for robust manipulation

    Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Rec- onciling reality through simulation: A real-to-sim-to-real approach for robust manipulation. arXiv:2403.03949, 2024

    2024 arXiv

  38. [48]

    Vlm see, robot do: Human demo video to robot action plan via vision language model

    Beichen Wang, Juexiao Zhang, Shuwen Dong, Irving Fang, and Chen Feng. Vlm see, robot do: Human demo video to robot action plan via vision language model. arXiv:2410.08792, 2024

    2024

  39. [49]

    Neus: Learning neural implicit surfaces by volume rendering for multi- view reconstruction

    Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi- view reconstruction. arXiv:2106.10689, 2021

    2021 arXiv

  40. [50]

    Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction

    Yiming Wang, Qin Han, Marc Habermann, Kostas Dani- ilidis, Christian Theobalt, and Lingjie Liu. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In ICCV, 2023

    2023

  41. [51]

    D 3 fields: Dynamic 3d de- scriptor fields for zero-shot generalizable rearrangement

    Yixuan Wang, Mingtong Zhang, Zhuoran Li, Tarik Ke- lestemur, Katherine Rose Driggs-Campbell, Jiajun Wu, Li Fei-Fei, and Yunzhu Li. D 3 fields: Dynamic 3d de- scriptor fields for zero-shot generalizable rearrangement. In CoRL, 2024

    2024

  42. [52]

    Rl-gsbridge: 3d gaus- sian splatting based real2sim2real method for robotic manipulation learning

    Yuxuan Wu, Lei Pan, Wenhua Wu, Guangming Wang, Yanzi Miao, and Hesheng Wang. Rl-gsbridge: 3d gaus- sian splatting based real2sim2real method for robotic manipulation learning. arXiv:2409.20291, 2024

    2024 arXiv

  43. [53]

    Robotic control via embodied chain-of-thought reasoning

    Michał Zawalski, William Chen, Karl Pertsch, Oier Mees, Chelsea Finn, and Sergey Levine. Robotic control via embodied chain-of-thought reasoning. arXiv:2407.08693, 2024

    2024 arXiv

  44. [54]

    Vlmpc: Vision- language model predictive control for robotic manipu- lation

    Wentao Zhao, Jiaming Chen, Ziyu Meng, Donghui Mao, Ran Song, and Wei Zhang. Vlmpc: Vision- language model predictive control for robotic manipu- lation. arXiv:2407.09829, 2024

    2024 arXiv

  45. [55]

    playing the drum

    Xinxin Zhao, Wenzhe Cai, Likun Tang, and Teng Wang. Imaginenav: Prompting vision-language mod- els as embodied navigator through scene imagination. arXiv:2410.09874, 2024. APPENDIX A IMPLEMENTATION DETAILS A. Reconstruction details For each scene, we use approximately 200 imag...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.