gradSim: Differentiable simulation for system identification and visuomotor control
read the original abstract
We consider the problem of estimating an object's physical properties such as mass, friction, and elasticity directly from video sequences. Such a system identification problem is fundamentally ill-posed due to the loss of information during image formation. Current solutions require precise 3D labels which are labor-intensive to gather, and infeasible to create for many systems such as deformable solids or cloth. We present gradSim, a framework that overcomes the dependence on 3D supervision by leveraging differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation. This novel combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them. Moreover, our unified computation graph -- spanning from the dynamics and through the rendering process -- enables learning in challenging visuomotor control tasks, without relying on state-based (3D) supervision, while obtaining performance competitive to or better than techniques that rely on precise 3D labels.
This paper has not been read by Pith yet.
Forward citations
Cited by 6 Pith papers
-
RigPI: Dynamic Parameter Identification of Rigid Body via VLM-Seeded Differentiable Simulation
RigPI combines VLM semantic priors with two-stage gradient optimization in differentiable simulation to identify inertial and frictional parameters of rigid bodies from robot-object interactions.
-
$\Delta$ynamics: Language-Based Representation for Inferring Rigid-Body Dynamics From Videos
A vision-language framework generates text-based rigid-body scene configurations from videos using motion reasoning and optical flow, reporting 0.30 IoU on CLEVRER (7x over baselines) and transfer to 235 real videos.
-
Zero-Shot Sim-to-Real Robot Learning: A Dexterous Manipulation Study on Reactive Catching
DRIS improves zero-shot sim-to-real transfer for reactive catching by maintaining and acting on sets of randomized dynamics instances instead of single instances per episode.
-
RigPI: Dynamic Parameter Identification of Rigid Body via VLM-Seeded Differentiable Simulation
RigPI combines VLM initialization with two-stage gradient-based optimization in differentiable simulation to estimate dynamic parameters of rigid bodies from real robot interactions.
-
Physically Viable World Models: A Case for Query-Conditioned Embodied AI
Embodied AI requires query-conditioned world models that select the simplest physical abstraction sufficient to answer intervention queries.
-
Fast and Reliable Gradients for Deformables Across Frictional Contact Regimes
The authors develop a differentiable simulator enforcing Markovian dynamics on a position-velocity manifold and using a mass-aligned preconditioner with a soft Fischer-Burmeister operator to produce stable gradients f...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.