Pith. sign in

REVIEW 4 major objections 6 minor 74 references

Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Scan, Materialize, Simulate: a pipeline that lets robots plan contact-rich actions in novel scenes by reconstructing, assigning materials, and optimizing in a physics engine before execution.

desk verdict A competent integration of off-the-shelf components whose sim-to-sim evidence doesn't yet support the 'generalizable physical reasoning' claim. read the letter →

arxiv 2505.14938 v1 pith:4NZIPH4Z submitted 2025-05-20 cs.RO cs.CVcs.GRcs.LG

Amine Elhafsi , Daniel Morton , Marco Pavone This is my paper
classification cs.ROcs.CVcs.GRcs.LG
keywords 3DGaussianSplattingSceneSegmentationPhysicsSimulationModel-BasedPlanningSim-to-RealVision-LanguageMaterialInferenceQuadrotorLandingObject-Centric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Scan, Materialize, Simulate (SMS), a three-stage pipeline that lets a robot plan physically interactive actions in a scene it has never seen before. The robot first reconstructs the scene as 3D Gaussians from RGBD images, then segments objects with visual foundation models and converts them into simulation-ready meshes with material properties queried from a vision-language model, and finally optimizes the action in a physics engine before executing it. The authors claim this makes physics reasoning generalizable: the same pipeline handles a billiards-style striking task and a quadrotor landing on an unstable structure, in simulated transfer and on a real quadrotor. The contribution is showing that differentiable rendering, foundation models, and rigid-body simulation can be assembled into a planner that predicts contact dynamics well enough to deploy.

What carries the argument

The load-bearing mechanism is the surrogate optimal control loop: reconstruct a Gaussian splatting scene with depth, normal, and segmentation losses; densify Gaussians into surface point clouds and extract meshes via pruned Delaunay tetrahedralization, with RANSAC sphere fitting for balls; query a vision-language model for density, friction, restitution, Young's modulus, and Poisson's ratio; then instantiate meshes in a physics engine and minimize a task-specific objective with Nelder-Mead simplex optimization over a low-dimensional action parameterization. The core abstraction is the surrogate dynamics constraint $\hat{x}_{k+1} = \hat{f}(\hat{x}_k, u_k)$ supplied by the physics engine, which replaces any learned or analytic model of the world.

What would settle it

For a fixed scene, replace the vision-language model's material estimates with values measured in the lab and re-optimize the action; if the realized objective does not improve and the predicted-to-realized gap does not shrink, the material-inference step is not actually driving success. Alternatively, corrupt the estimated friction and restitution by swapping them between two objects and check whether the chosen action changes; if it does not, the planner is insensitive to the very information the framework claims to add.

Watch

Extended reading notes

Core claim

SMS's central claim is that physical reasoning for novel scenes does not require a learned dynamics model: a surrogate environment reconstructed with 3D Gaussian Splatting, segmented with visual foundation models, assigned material parameters by a vision-language model, and simulated in a physics engine is a sufficient substrate for optimizing actions that succeed when executed. The paper validates this in two domains. In 18 billiards scenes, the per-scene best action optimized in simulation reaches a mean realized distance of 5.5 cm from the goal, beating a geometric grid-search baseline (28.2 cm); in four quadrotor landing scenes, success rates are 80–100% versus 50–60% for a visual prompting baseline. A real quadrotor flight shows the optimized approach avoids toppling an overhanging landing platform while a direct approach fails.

Load-bearing premise

The framework assumes the reconstructed meshes and the material properties inserted into the simulator capture the physics that matters: if the geometry is wrong at contact points or the vision-language model's guesses for friction, restitution, and density are off, the action that looks good in simulation can fail in the real world.

Editorial extensions

If this is right

  • Actions optimized in the reconstructed virtual environment translate well to execution: in billiards, predicted and realized trajectories closely match across the featured scenes.
  • SMS discovers multiple distinct valid strategies for the same scene (e.g., rebounds off different obstacle materials in Scene D), showing the planner exploits geometry and material rather than a single memorized policy.
  • A purely visual foundation-model baseline fails on the propeller-wash-sensitive landing task, so physical simulation is necessary where a vision-language model cannot reason about dynamics.
  • The same pipeline succeeds on two very different control problems, suggesting it can be pointed at new contact-rich tasks by swapping the objective and action parameterization.
  • Per-scene best performance beats the geometric baseline, and the advantage grows in scenes with complex multi-collision interactions such as double rebounds.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the vision-language model's material estimates are the bottleneck, the framework's sensitivity could be tested by replacing them with measured values from a friction tester and drop test; if realized error drops sharply, the parameter guesses, not the geometry, currently bound performance.
  • The same scan-materialize-simulate loop could be closed with feedback: after each execution, compare predicted versus realized state and update mesh or material parameters, converting the open-loop planner into an adaptive system.
  • The per-scene-best evaluation (5.5 cm) is more informative than the overall mean (20.5 cm) because the simplex optimizer is initialization-sensitive; a practical deployment would run several initializations and pick the best predicted action, making 5.5 cm the realistic operating point.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents SMS, a three-stage framework: (1) Scan, which builds a 3D Gaussian splatting reconstruction from RGBD sequences together with object segmentation; (2) Materialize, which converts object Gaussians into physics-ready meshes and queries GPT-4o for density, friction, restitution, Young's modulus, and Poisson's ratio; and (3) Simulate, which optimizes robot actions in a physics surrogate using Nelder-Mead. The authors evaluate the framework on a simulated billiards manipulation task (18 scenes, comparing predicted and realized objectives) and a quadrotor landing task (four simulated scenes and one hardware demonstration), and claim that SMS achieves generalizable physical reasoning without retraining for each task.

Significance. Taken at face value, the framework is a timely integration of 3D Gaussian splatting, visual foundation models, and physics engines, and the modular design is sensible. The paper is honest about several limitations (open-loop control, hand-adjusted scenes, stacked-object geometry post-processing), provides detailed experimental appendices and prompt templates, and reports per-scene distributions rather than only averages. However, the evidence does not currently support the central generalization claim: the billiards results are sim-to-sim in author-constructed scenes, the only hardware data point is a single qualitative flight, and the material-property estimates are unvalidated. These gaps are load-bearing because the paper's title and abstract promise physically grounded planning that transfers to execution.

major comments (4)
  1. [§4.1, Table 1] The column labeled 'SMS Realized' reports execution in Isaac Sim, not on a physical robot; both the reconstruction source and the target environments are generated from the same Omniverse asset library, and Section 4.1 states that entities are manually assigned physical parameters. Consequently, Table 1 demonstrates simulation-to-simulation transfer between two simulated worlds and does not support the claim that actions 'translate well to execution' in a real target environment. An end-to-end physical experiment, or at minimum an explicit reframing of the results as sim-to-sim, is required.
  2. [§4.2, §5] The real-world quadrotor experiment is a single flight (Figure 5) with no trial count, no success rate, and no statement that the flight was produced end-to-end by scanning a physical scene, running Materialize, and optimizing the trajectory in the reconstructed model. Section 5 further admits that stacked-object interfaces 'necessitated post-processing to ensure the interface geometries were reasonably modeled.' This is not sufficient evidence for the abstract's claim of real-world validation of SMS.
  3. [§3.2] The VLM material estimates (density, friction, restitution) are fed directly into the physics optimizer, yet the paper reports no comparison against measured values and no sensitivity analysis. Since the planning objective is a deterministic function of these parameters, the contribution of Materialize is unvalidated: planning success could be insensitive to the estimates or could rely on favorable coincidences. I would like to see an error analysis on objects with known properties and an ablation that perturbs the material parameters in simulation to measure the effect on success.
  4. [§A.1] Scene generation included manual adjustment to 'ensure that the target ball could reasonably achieve the goal position.' This makes the 18 billiards scenes an author-curated test set rather than a random sample of tasks, and it should be reported in the main text with the scenes released. Without an automatic generator or a stated acceptance criterion, the per-scene success numbers are difficult to interpret as evidence of robustness.
minor comments (6)
  1. [§4.2] The word 'signficantly' should be 'significantly.'
  2. [Table 1] The table would be clearer if the caption defined 'Predicted' and 'Realized' and explained why the baseline has no 'Overall' entry; currently the footnote symbols are necessary but the definitions are incomplete.
  3. [Figure A.2] The mapping of Scenes A-D to Scenes 10, 12, 16, and 5 is given only in the caption; one might also state it in the main text for reader convenience.
  4. [§4.2, §B.3] The SPH propeller-wash model is calibrated with momentum theory, but the paper does not report any validation of the simulated wash force against a real propeller measurement; a comparison would increase confidence in the landing simulations.
  5. [General] The paper would benefit from a brief statement on code and data availability; the current version does not indicate whether the scene assets and reconstruction pipeline will be released.
  6. [§5] The open-loop limitation is acknowledged, but it is central enough to be mentioned in the abstract or introduction to prevent overgeneralizing the claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: SMS is an engineering pipeline whose predicted and realized outcomes are evaluated in independent environments, and the few self-citations are not load-bearing.

full rationale

SMS does not derive a theoretical result from an equivalent input. The 'predicted' billiards objective is the optimizer's value in the PyBullet reconstruction environment, while the 'realized' value is obtained by executing the same action in Isaac Sim scenes whose physical parameters were manually assigned independently: 'Entities in the scene, including the robot and floor, are manually assigned realistic physical parameters, such as friction, restitution, and mass, to ensure realistic behavior' (Sec. 4.1). No parameter is fitted to the realized outcomes; the GPT-4o material estimates are unvalidated inputs, not quantities recovered from the success metric. The paper's self-citations are not load-bearing: [71] supplies an operational-space torque controller used for execution, [18] appears in related work, and [65] appears only in future work. The acknowledged limitations (manual post-processing of stacked-object interfaces in Sec. 5, manual scene adjustment in Sec. A.1, and open-loop execution) weaken external-validity and robustness claims, but they do not make any prediction equal to its input by construction. No circular step can be exhibited from the paper's own equations or citation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework adds no new physical entity. Its central claim rests on the reliability of several external models and modeling choices: foundation-model segmentation, VLM material inference, and physics-engine fidelity. The free parameters are mostly unmeasured inputs to the pipeline rather than fitted constants.

free parameters (3)
  • GPT-4o material property estimates = Not reported (JSON outputs for density, friction, restitution, Young's modulus, Poisson's ratio)
    Every object's physical parameters in the simulation are taken from VLM outputs (Section 3.2, Appendix A.2). No comparison to measured values is given, and the planning outcome depends on these numbers.
  • Gaussian splatting loss weights = Not specified (lambda_photo, lambda_isotropic, lambda_normal, lambda_scale)
    Section 3.1 defines these regularization weights but does not report numerical values; they shape reconstruction quality and therefore downstream meshes.
  • SPH particle emitter parameters = Adaptively determined by the simulator (n_particles, ell_particle, d_particle)
    Appendix B.3 calibrates propeller wash with momentum disk theory, then lets the simulator choose particle counts and sizes; the resulting downwash forces are central to the landing results.
assumptions (4)
  • domain assumption Rigid-body simulation with Newtonian contact models is an accurate surrogate for the real dynamics of the billiards and landing scenes.
    The entire Simulate stage treats the physics engine output as the surrogate dynamics f (Section 3). The paper does not validate simulated forces or trajectories against measured real-world data except for one qualitative quadrotor flight.
  • domain assumption GPT-4o can infer density, friction, and restitution values from a single annotated image accurately enough for planning.
    Materialize (Section 3.2) relies on VLM material inference; the paper reports no comparison to measured material properties or ablation of material estimate errors.
  • domain assumption OWLv2 detections and SAM 2 masks are accurate enough to associate Gaussians with the correct objects.
    The segmentation labels supervise the affinity-feature classifier in Scan (Section 3.1); errors propagate into the meshes in Materialize.
  • domain assumption The mesh extraction pipeline (densified point cloud, Delaunay tetrahedralization, pruning, convex decomposition) preserves collision-relevant geometry.
    Section 3.2 and Appendix A.4; the paper itself notes that stacked-object interfaces had to be manually post-processed for the landing experiments (Section 5), indicating this assumption sometimes fails.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning." pith.science (2026). https://pith.science/paper/4NZIPH4Z

@misc{pith2026250514938,
  author       = {Pith},
  title        = {Pith review of: Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4NZIPH4Z}},
  note         = {Machine review of arXiv:2505.14938}
}
read the original abstract

Autonomous robots must reason about the physical consequences of their actions to operate effectively in unstructured, real-world environments. We present Scan, Materialize, Simulate (SMS), a unified framework that combines 3D Gaussian Splatting for accurate scene reconstruction, visual foundation models for semantic segmentation, vision-language models for material property inference, and physics simulation for reliable prediction of action outcomes. By integrating these components, SMS enables generalizable physical reasoning and object-centric planning without the need to re-learn foundational physical dynamics. We empirically validate SMS in a billiards-inspired manipulation task and a challenging quadrotor landing scenario, demonstrating robust performance on both simulated domain transfer and real-world experiments. Our results highlight the potential of bridging differentiable rendering for scene reconstruction, foundation models for semantic understanding, and physics-based simulation to achieve physically grounded robot planning across diverse settings.

Figures

Figures reproduced from arXiv: 2505.14938 by the authors.

Figure 1
Figure 1. Overview of the Scan, Materialize, Simulate (SMS) framework for physics-informed robot action planning. SMS consists of three steps: scanning to build a geometric environment model, materializing to convert this reconstruction into a simulation-ready representation, and sim￾ulating to optimize actions in a virtual environment prior to target-environment execution. 1 Introduction In recent years, autonomous robotic s… view at source ↗
Figure 2
Figure 2. Billiards scenes. Left: Motion composite images show the actions optimized in the re [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Distributions of SMS performance over 30 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Quadrotor approach comparison. Left: SMS optimizes [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Quadrotor hardware demonstration. Left: A direct ap [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

74 extracted references · 41 canonical work pages

  1. [1]

    Fazeli, M

    N. Fazeli, M. Oller, J. Wu, Z. Wu, J. B. Tenenbaum, and A. Rodriguez. See, feel, act: Hierar- chical learning for complex manipulation skills with multisensory fusion.Science Robotics, 4 (26):eaav3123, 2019. doi:10.1126/scirobotics.aav3123. URLhttps://www.science.org/ doi/abs/10.1126/scirobotics.aav3123

    doi:10.1126/scirobotics.aav3123 2019

  2. [2]

    Greenspan, J

    M. Greenspan, J. Lam, M. Godard, I. Zaidi, S. Jordan, W. Leckie, K. Anderson, and D. Dupuis. Toward a competitive pool-playing robot.Computer, 41(1):46–53, 2008

    work page 2008

  3. [3]

    R. E. Moutaouaffiq. Billiardbot: Physics-aware planning for robotic billiards.https:// rachad47.github.io/rwae/BilliardBot.html, 2025

    work page 2025

  4. [4]

    D. B. D’Ambrosio, N. Jaitly, V . Sindhwani, K. Oslund, P. Xu, N. Lazic, A. Shankar, T. Ding, J. Abelian, E. Coumans, et al. Robotic table tennis: A case study into a high speed learning system. InRobotics: Science and Systems, 2023

    work page 2023

  5. [5]

    Kerbl, G

    B. Kerbl, G. Kopanas, T. Leimk ¨uhler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering.ACM Transactions on Graphics, 42(4), July 2023. URLhttps: //repo-sam.inria.fr/fungraph/3d-gaussian-splatting/

    work page 2023

  6. [6]

    Minderer, A

    M. Minderer, A. Gritsenko, and N. Houlsby. Scaling open-vocabulary object detection.Ad- vances in Neural Information Processing Systems, 36:72983–73007, 2023

    work page 2023

  7. [7]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V . Alwala, N. Carion, C.-Y . Wu, R. Girshick, P. Doll ´ar, and C. Feichtenhofer. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024. URLhttps://arxiv.org/abs/2408.00714

    arXiv 2024

  8. [8]

    R. S. Sutton, A. G. Barto, et al. Reinforcement learning.Journal of Cognitive Neuroscience, 11(1):126–134, 1999

    work page 1999

Show all 74 references
  1. [9]

    Yarats, I

    D. Yarats, I. Kostrikov, and R. Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. InInternational Conference on Learning Representations,

  2. [10]

    S. Nair, A. Rajeswaran, V . Kumar, C. Finn, and A. Gupta. R3m: A universal visual repre- sentation for robot manipulation. InConference on Robot Learning, pages 892–909. PMLR, 2023

    2023

  3. [11]

    Hussein, M

    A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods.ACM Computing Surveys (CSUR), 50(2):1–35, 2017

    2017

  4. [12]

    Shridhar, L

    M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. InConference on Robot Learning, pages 785–799. PMLR, 2023

    2023

  5. [13]

    Florence, C

    P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mor- datch, and J. Tompson. Implicit behavioral cloning. InConference on robot learning, pages 158–168. PMLR, 2022

    2022

  6. [14]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 2024. 10

    2024

  7. [15]

    Tunyasuvunakool, A

    S. Tunyasuvunakool, A. Muldal, Y . Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lil- licrap, N. Heess, and Y . Tassa. dm control: Software and tasks for continuous con- trol.Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi:https://doi.org/10.1016/j. simpa.2020.100022. U...

    2020

  8. [16]

    Denil, P

    M. Denil, P. Agrawal, T. D. Kulkarni, T. Erez, P. Battaglia, and N. de Freitas. Learning to perform physics experiments via deep reinforcement learning. InInternational Conference on Learning Representations, 2017. URLhttps://openreview.net/forum?id=r1nTpv9eg

    2017

  9. [17]

    B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration.Robotics and autonomous systems, 57(5):469–483, 2009

    2009

  10. [18]

    Sinha, A

    R. Sinha, A. Sharma, S. Banerjee, T. Lew, R. Luo, S. M. Richards, Y . Sun, E. Schmerling, and M. Pavone. A system-level view on out-of-distribution data in robotics.arXiv preprint arXiv:2212.14020, 2022

    2022 arXiv

  11. [19]

    Brohan, N

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control.arXiv preprint arXiv:2307.15818, 2023

    2023 arXiv

  12. [20]

    O. X.-E. Collaboration, A. O’Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Her- zog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi, A. ...

    arXiv

  13. [21]

    M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. P. Foster, P. R. Sanketi, Q. Vuong, et al. Openvla: An open-source vision-language-action model. In8th Annual Conference on Robot Learning, 2024

    2024

  14. [22]

    Black, N

    K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Haus- man, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky.π 0: A vision...

    2024 arXiv

  15. [23]

    J. Wen, Y . Zhu, Z. Tang, J. Li, Y . Peng, C. Shen, and F. Feng. Dexvla: Vision-language model with plug-in diffusion expert for visuomotor policy learning.arXiv preprint arXiv:2502.05855, 2025

    2025 arXiv

  16. [24]

    Huang, F

    H. Huang, F. Liu, L. Fu, T. Wu, M. Mukadam, J. Malik, K. Goldberg, and P. Abbeel. Otter: A vision-language-action model with text-aware feature extraciton.arXiv preprint arXiv:2503.03734, 2025

    2025

  17. [25]

    C. Zhou, Q. Li, C. Li, J. Yu, Y . Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He, et al. A compre- hensive survey on pretrained foundation models: A history from bert to chatgpt.International Journal of Machine Learning and Cybernetics, pages 1–65, 2024

    2024

  18. [26]

    Mildenhall, P

    B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. InECCV, 2020

    2020

  19. [27]

    Fridovich-Keil, A

    S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radi- ance fields without neural networks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5501–5510, 2022

    2022

  20. [28]

    M ¨uller, A

    T. M ¨uller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding.ACM Trans. Graph., 41(4):102:1–102:15, July 2022. doi: 10.1145/3528223.3530127. URLhttps://doi.org/10.1145/3528223.3530127

    2022

  21. [29]

    T. Chen, P. Culbertson, and M. Schwager. Catnips: Collision avoidance through neural implicit probabilistic scenes.IEEE Transactions on Robotics, 2024

    2024

  22. [30]

    T. Chen, O. Shorinwa, J. Bruno, A. Swann, J. Yu, W. Zeng, K. Nagami, P. Dames, and M. Schwager. Splat-nav: Safe real-time robot navigation in gaussian splatting maps.arXiv preprint arXiv:2403.02751, 2024

    2024 arXiv

  23. [31]

    X. Lei, M. Wang, W. Zhou, and H. Li. Gaussnav: Gaussian splatting for visual navigation. IEEE Transactions on Pattern Analysis & Machine Intelligence, (01):1–14, 2025

    2025

  24. [32]

    Abou-Chakra, K

    J. Abou-Chakra, K. Rana, F. Dayoub, and N. Suenderhauf. Physically embodied gaussian splatting: A realtime correctable world model for robotics. In8th Annual Conference on Robot Learning, 2024. URLhttps://openreview.net/forum?id=AEq0onGrN2

    2024

  25. [33]

    Zhang, K

    M. Zhang, K. Zhang, and Y . Li. Dynamic 3d gaussian tracking for graph-based neural dynam- ics modeling. In8th Annual Conference on Robot Learning

  26. [34]

    K. M. Jatavallabhula, M. Macklin, F. Golemo, V . V oleti, L. Petrini, M. Weiss, B. Con- sidine, J. Parent-Levesque, K. Xie, K. Erleben, L. Paull, F. Shkurti, D. Nowrouzezahrai, and S. Fidler. gradsim: Differentiable simulation for system identification and visuomo- tor control...

    2021

  27. [35]

    R. Liu, A. Canberk, S. Song, and C. V ondrick. Differentiable robot rendering. In8th Annual Conference on Robot Learning. 12

  28. [36]

    J. Low, M. Adang, J. Yu, K. Nagami, and M. Schwager. Sous vide: Cooking visual drone navigation policies in a gaussian splatting vacuum.arXiv preprint arXiv:2412.16346, 2024

    2024 arXiv

  29. [37]

    X. Li, J. Li, Z. Zhang, R. Zhang, F. Jia, T. Wang, H. Fan, K.-K. Tseng, and R. Wang. Robogsim: A real2sim2real robotic gaussian splatting simulator.arXiv preprint arXiv:2411.11839, 2024

    2024 arXiv

  30. [38]

    M. N. Qureshi, S. Garg, F. Yandun, D. Held, G. Kantor, and A. Silwal. Splatsim: Zero- shot sim2real transfer of rgb manipulation policies using gaussian splatting.arXiv preprint arXiv:2409.10161, 2024

    2024 arXiv

  31. [39]

    Quach, M

    A. Quach, M. Chahine, A. Amini, R. Hasani, and D. Rus. Gaussian splatting to real world flight navigation transfer with liquid networks. In8th Annual Conference on Robot Learning

  32. [40]

    Torne, A

    M. Torne, A. Simeonov, Z. Li, A. Chan, T. Chen, A. Gupta, and P. Agrawal. Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation.arXiv preprint arXiv:2403.03949, 2024

    2024 arXiv

  33. [41]

    Barcellona, A

    L. Barcellona, A. Zadaianchuk, D. Allegro, S. Papa, S. Ghidoni, and E. Gavves. Dream to manipulate: Compositional world models empowering robot imitation learning with imagina- tion. InThe Thirteenth International Conference on Learning Representations, 2025. URL https://openr...

    2025

  34. [42]

    Meyer, F

    L. Meyer, F. Erich, Y . Yoshiyasu, M. Stamminger, N. Ando, and Y . Domae. Pegasus: Phys- ically enhanced gaussian splatting simulation system for 6dof object pose dataset generation. In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10710...

    2024

  35. [43]

    J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded ra- diance fields. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 19729–19739, 2023

    2023

  36. [44]

    S. Zhou, H. Chang, S. Jiang, Z. Fan, Z. Zhu, D. Xu, P. Chari, S. You, Z. Wang, and A. Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 216...

    2024

  37. [45]

    M. Qin, W. Li, J. Zhou, H. Wang, and H. Pfister. Langsplat: 3d language gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20051–20060, 2024

    2024

  38. [46]

    W. Shen, G. Yang, A. Yu, J. Wong, L. P. Kaelbling, and P. Isola. Distilled feature fields enable few-shot language-guided manipulation. InConference on Robot Learning, pages 405–424. PMLR, 2023

    2023

  39. [47]

    G. Lu, S. Zhang, Z. Wang, C. Liu, J. Lu, and Y . Tang. Manigaussian: Dynamic gaussian splatting for multi-task robotic manipulation. InEuropean Conference on Computer Vision, pages 349–366. Springer, 2024

    2024

  40. [48]

    T. Xie, Z. Zong, Y . Qiu, X. Li, Y . Feng, Y . Yang, and C. Jiang. Physgaussian: Physics- integrated 3d gaussians for generative dynamics. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4389–4398, 2024

    2024

  41. [49]

    R.-Z. Qiu, G. Yang, W. Zeng, and X. Wang. Feature splatting: Language-driven physics-based scene synthesis and editing. InEuropean Conference on Computer Vision (ECCV), 2024

    2024

  42. [50]

    H. Zhao, H. Wang, X. Zhao, H. Fei, H. Wang, C. Long, and H. Zou. Efficient physics simu- lation for 3d scenes via mllm-guided gaussian splatting, 2025. URLhttps://arxiv.org/ abs/2411.12789. 13

    2025 arXiv

  43. [51]

    Jiang, C

    C. Jiang, C. Schroeder, J. Teran, A. Stomakhin, and A. Selle. The material point method for simulating continuum materials. InAcm siggraph 2016 courses, pages 1–52. 2016

    2016

  44. [52]

    Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004

    2004

  45. [53]

    Yugay, Y

    V . Yugay, Y . Li, T. Gevers, and M. R. Oswald. Gaussian-slam: Photo-realistic dense slam with gaussian splatting.arXiv preprint arXiv:2312.10070, 2023

    2023 arXiv

  46. [54]

    Turkulainen, X

    M. Turkulainen, X. Ren, I. Melekhov, O. Seiskari, E. Rahtu, and J. Kannala. Dn-splatter: Depth and normal priors for gaussian splatting and meshing. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025

    2025

  47. [55]

    Q.-Y . Zhou, J. Park, and V . Koltun. Open3d: A modern library for 3d data processing

  48. [56]

    Todorov, T

    E. Todorov, T. Erez, and Y . Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–

    2012

  49. [57]

    Coumans and Y

    E. Coumans and Y . Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning.http://pybullet.org, 2016–2023

    2016

  50. [58]

    G. Authors. Genesis: A universal and generative physics engine for robotics and beyond, December 2024. URLhttps://github.com/Genesis-Embodied-AI/Genesis

    2024

  51. [59]

    Delaunay

    B. Delaunay. Sur la sph `ere vide. a la m´emoire de georges vorono¨ı.Bulletin de l’Acade ´emie des Sciences de l’URSS. Classe des sciences mathe ´ematiques et naturelles., (6):793–800, 1934

    1934

  52. [60]

    C. B. Barber, D. P. Dobkin, and H. Huhdanpaa. The quickhull algorithm for convex hulls. ACM Transactions on Mathematical Software (TOMS), 22(4):469–483, 1996

    1996

  53. [61]

    M. A. Fischler and R. C. Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.Communications of the ACM, 24 (6):381–395, 1981

    1981

  54. [62]

    J. A. Nelder and R. Mead. A simplex method for function minimization.The computer journal, 7(4):308–313, 1965

    1965

  55. [63]

    Koschier, J

    D. Koschier, J. Bender, B. Solenthaler, and M. Teschner. Smoothed particle hydrody- namics techniques for the physics based simulation of fluids and solids.arXiv preprint arXiv:2009.06944, 2020

    2009 arXiv

  56. [64]

    Nasiriany, F

    S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. InInternational Conference on Machine Learning, pages 37321–37341. PMLR, 2024

    2024

  57. [65]

    Z. Fan, K. Wen, W. Cong, K. Wang, J. Zhang, X. Ding, D. Xu, B. Ivanovic, M. Pavone, G. Pavlakos, Z. Wang, and Y . Wang. Instantsplat: Sparse-view gaussian splatting in seconds, 2024

    2024

  58. [66]

    J. Yang, J. Huang, B. Ivanovic, Y . Chen, Y . Wang, B. Li, Y . You, A. Sharma, M. Igl, P. Karkus, et al. Storm: Spatio-temporal reconstruction model for large-scale outdoor scenes. InThe Thirteenth International Conference on Learning Representations

  59. [67]

    Szymanowicz, J

    S. Szymanowicz, J. Y . Zhang, P. Srinivasan, R. Gao, A. Brussee, A. Holynski, R. Martin- Brualla, J. T. Barron, and P. Henzler. Bolt3D: Generating 3D Scenes in Seconds. arXiv:2503.14445, 2025. 14

    2025

  60. [68]

    Le Cleac’h, H.-X

    S. Le Cleac’h, H.-X. Yu, M. Guo, T. Howell, R. Gao, J. Wu, Z. Manchester, and M. Schwager. Differentiable physics simulation of dynamics-augmented neural objects.IEEE Robotics and Automation Letters, 8(5):2780–2787, 2023

    2023

  61. [69]

    M. Macklin. Warp: A high-performance python framework for gpu simulation and graphics. https://github.com/nvidia/warp, March 2022. NVIDIA GPU Technology Conference (GTC)

    2022

  62. [70]

    Huang, F

    Z. Huang, F. Chen, Y . Pu, C. Lin, H. Su, and C. Gan. Diffvl: Scaling up soft body manip- ulation using vision-language driven differentiable physics.Advances in Neural Information Processing Systems, 36:29875–29900, 2023

    2023

  63. [71]

    Morton and M

    D. Morton and M. Pavone. Safe, task-consistent manipulation with operational space control barrier functions. InIEEE/RSJ Int. Conf. on Intelligent Robots & Systems, 2025. URLhttps: //arxiv.org/pdf/2503.06736

    2025

  64. [72]

    Input." It will provide more specific requirements pertaining to the query under

    X. Wei, M. Liu, Z. Ling, and H. Su. Approximate convex decomposition for 3d meshes with collision-aware concavity and tree search.ACM Transactions on Graphics (TOG), 41(4):1–18, 2022. 15 A Additional Details for Billiards Scenario A.1 Scene Generation For our billiards experim...

    2022

  65. [2021]

    URLhttps://openreview.net/forum?id=GY6-6sTvGaf

  66. [5033]

    doi:10.1109/IROS.2012.6386109

    IEEE, 2012. doi:10.1109/IROS.2012.6386109

    2012

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.