REVIEW 3 major objections 3 minor 34 references
GPU-batched robot simulators are not deterministic data sources: identical setups diverge across parallel environments and repeated runs, and scaling widens the gap.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 08:27 UTC pith:YRWLLH5R
load-bearing objection A solid, useful benchmark for GPU-simulator reproducibility that overclaims in the abstract: Type 1 'variability' is deterministic batching bias, not non-determinism, but the typology and metrics are worth keeping. the 3 major comments →
GPUSimBench: Towards Scalable and Reliable GPU-Accelerated Simulators in Embodied AI
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is that batched execution on a GPU injects non-determinism into physics simulation even when random seeds are fixed and task-level randomization is disabled. Across seven simulators under unified conditions, GPUSimBench records each simulator's throughput, memory footprint, distribution-level agreement with real-world data (measured as Earth Mover's Distance on the planar cube positions), and two variability metrics. Four regimes emerge: some simulators show drift between parallel environments but stable aggregate distributions across runs; others show both forms of drift; one shows neither; one shows only run-to-run drift. The drift is traced to operation-order-depende
What carries the argument
The load-bearing machinery is the benchmark's two metrics together with the real-world EMD. Parallel Variability is the average pairwise Earth Mover's Distance (a measure, in centimeters, of how much planar mass must be moved to make one environment's final cube positions match another's) between all pairs of environments in a single run; Run-to-Run Variability is the same distance between the aggregate distributions of independent runs. Crossing these two presence/absence axes yields the four stochasticity regimes, and comparing each simulator's aggregate distribution to the measured real-world distribution yields the physical-alignment ranking. These three numbers turn 'the simulator is no
Load-bearing premise
The physical-alignment ranking assumes that the single set of lab-measured material and contact parameters transfers faithfully to all seven simulators' different solvers without per-simulator calibration.
What would settle it
Re-run a simulator classified as Type 2 (both variabilities absent) on two different GPU models with identical seeds; if the run-to-run Earth Mover's Distance becomes nonzero, the claim that this regime is stable across runs is a configuration artifact rather than an inherent property.
If this is right
- If the central claim is right, reproducibility in GPU-batched simulation must be measured per simulator and per scale; assuming determinism from fixed seeds is unsafe.
- The four regimes imply that 'deterministic' is not a single property: a simulator can be internally consistent within a run yet unstable across runs, or vice versa, so reproducibility claims need to specify which axis.
- Sim-to-real contact benchmarks should expect large engine-to-engine spread in physical alignment; choices made for throughput can dominate the physical error.
- For repeated-run comparisons or parameter tuning, prefer simulators with neither variability; for single-run evaluation with plausible contact behavior, other simulators may suffice.
- Scaling to larger environment counts can push a simulator into a noisier regime, so scaling studies should report variability alongside throughput.
Where Pith is reading between the lines
- The variability metrics likely understate the effect on end-to-end robot learning, since policy gradients average over environments and runs; if environments drift, the effective sample size and gradient noise change in ways current RL pipelines do not track.
- The physical-alignment ranking mixes intrinsic solver fidelity with parameter transfer; a follow-up that calibrates friction and restitution per simulator would separate engine realism from parameter mismatch, and could reorder the ranking.
- A natural extension is to test whether variability grows with contact richness or simulation duration; the inclined-collision test is one contact-rich scene, while a simple cube-free-fall scene would be a lower-bound check.
- The four regimes suggest a design criterion for simulator builders: report which regime a release occupies at a given batch size, so users can decide whether to average across environments or across runs.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces GPUSimBench, a benchmark for GPU-accelerated robotic simulators, evaluating seven simulators (Isaac Lab, ManiSkill, Genesis, Madrona, MuJoCo Warp, MJX, MuJoCo Playground) along three axes: parallel scalability (throughput Eq. (1) and memory Eq. (2)), physical fidelity to a real-world inclined-plane collision experiment (EMD, Eq. (10)), and execution variability (parallel and run-to-run EMD, Eqs. (8) and (11)). The authors find large throughput and memory differences, a wide range of sim-to-real EMD values (2.52 cm to 211.60 cm), and four empirical determinism regimes based on the presence or absence of the two variability metrics. They conclude that GPU-batched execution introduces inherent non-determinism and provide simulator-selection guidelines.
Significance. GPUSimBench is a useful and well-scoped empirical contribution. The methodology is controlled in important ways: fixed seeds, disabled rendering, a unified timestep, matched scenes, warmup phases, and an external real-world apparatus. The variability metrics are clearly defined, and the four regimes give practitioners a practical framework for reasoning about reproducibility. The strongest and most defensible finding is that reproducibility is not guaranteed by setting a seed alone and must be measured per simulator and per scale. Explicit credit is due for the authors' care in controlling task-level randomization and for openly acknowledging the benchmark's scope limits in Section VII. If the results hold, the paper provides valuable guidance for choosing simulators in large-scale robot learning. However, the headline claim that GPU batching is 'inherently non-deterministic' goes beyond the data: only Type 3 simulators show run-to-run variability, and Type 1's parallel-only variability is more naturally interpreted as deterministic batching bias. The sim-to-real ranking also rests on a single parameter set transferred to heterogeneous contact models. Both issues are fixable by ref
major comments (3)
- [Abstract; Section VI.A; Table IV] The abstract and Section I claim that GPU-batched execution has 'inherent non-determinism' characterized by 'significant run-to-run and inter-environment variability.' Table IV does not support this universal claim. Isaac Lab and ManiSkill (Type 1) have Parallel Variability 4.21 and 4.76 cm, respectively, but Run-to-Run Variability 0.00±0.00; Genesis and MJX (Type 2) have both 0.00; Playground (Type 4) has run-to-run variability only. Type 1's pattern is expected from a deterministic mapping between environment index and numerical outcome (e.g., fixed thread ordering), not from stochastic execution. The robust conclusion is the weaker one: reproducibility must be measured per simulator and scale. Please revise the abstract and the 'inherent non-determinism' phrasing, and present the four regimes as observed patterns rather than evidence of universal stochasticity.
- [Section IV.B; Table II; Table IV] The physical-alignment ranking uses one lab-identified set of material/contact parameters (Table II) for all seven simulators. Since PhysX, XPBD, Taichi, and MuJoCo implement contact and friction quite differently, the same nominal friction/restitution values do not transfer with equal meaning. The EMD values in Table IV—e.g., ManiSkill 2.52 cm vs. Madrona 211.60 cm—therefore conflate simulator fidelity with parameter-transfer mismatch. The manuscript acknowledges 'best-effort parameter matching' (Section IV.B) but the guidelines in Section VI.C still rely on this ranking. Please add a per-simulator calibration experiment or, at minimum, explicitly reframe the metric as 'sim-to-real alignment under a common, uncalibrated parameter set' and temper the corresponding recommendations.
- [Section VI.B] The proposed mechanism for Parallel Variability is 'different thread schedules' and 'floating-point accumulation depends on operation order.' This is inconsistent with the measured Type 1 data: if thread schedules differed across runs, Run-to-Run Variability would not be 0.00±0.00. For Isaac Lab and ManiSkill, the effective execution order is stable across runs, so the nonzero Parallel Variability is a deterministic per-environment batching bias. The section should distinguish (i) stable intra-run heterogeneity (deterministic batching), (ii) unstable inter-run numerics (true non-determinism), and (iii) absence of both, and adjust the causal language accordingly.
minor comments (3)
- [Table III] The maximum-environment counts appear to be rendered as '2 11', '2 12', etc., which is likely exponent notation but is confusing as typeset. Please clarify with superscripts or a note such as '2^11'.
- [Fig. 2 caption] The caption mentions 'dice-drop joint pose distribution and Gaussian ball-drop distribution setups' that are 'omitted here due to space constraints.' Either include these experiments in the release or remove the dangling reference to avoid confusion about the benchmark's completeness.
- [Table IV caption] The checkmarks and ✗ markers are defined only in the text. Please state in the caption that a checkmark means the corresponding variability is nonzero after rounding at the reported precision, since values like 0.00±0.00 are precision-dependent.
Circularity Check
No significant circularity: GPUSimBench is an empirical benchmark whose claims rest on external measurements, not on self-referential fits.
full rationale
The paper is an empirical benchmark, not a derivation. Its central claims—scalability, physical alignment as measured by EMD against a real-world apparatus, and the four empirical regimes of variability—are computed directly from simulator outputs using the definitions in Eqs. (7)–(11). The variability metrics (Parallel Variability and Run-to-Run Variability) are descriptive aggregates of measured cube positions, and the four regimes are labels on whether those metrics are nonzero after rounding. Nothing in the classification is fitted to the claim it supports. The lab-identified material and contact parameters in Table II are experimental inputs, not outputs of the analysis, and the paper explicitly cautions that the EMD values are distribution-level agreement with a measured reference under best-effort parameter matching. The skeptical concern that Type 1 'Parallel Variability' combined with zero Run-to-Run Variability may reflect deterministic batching bias rather than true stochasticity is a question of physical interpretation and benchmark validity, not a circularity: the empirical observation does not reduce by construction to the paper's inputs. Self-citations in the introduction ([4], [5]) are background references to the authors' prior work and are not load-bearing for any benchmark conclusion. No uniqueness theorem, ansatz, or fitted parameter is imported to force a result. The limitation section is explicit about the benchmark's scope rather than masking any circular step. Therefore the appropriate score is 0.
Axiom & Free-Parameter Ledger
free parameters (1)
- Real-world material/contact parameters (friction, restitution, mass) =
Table II: s=0.65/0.35/0.35, k=0.45/0.25/0.25, e=0.40/0.40/0.55
axioms (4)
- domain assumption A single lab-identified parameter set transfers to all seven simulators' contact models without per-simulator calibration.
- domain assumption Fixing random seeds and disabling task-level randomization isolates numerical/execution nondeterminism from intentional stochasticity.
- domain assumption Aggregating Nenv=16 parallel environments and K=16 real trials yields comparable planar point sets for one-to-one EMD.
- domain assumption Disabling rendering does not alter physics stepping behavior.
read the original abstract
Data-driven embodied AI is rapidly transitioning into a paradigm that scales training through massively parallel simulation, where GPU-accelerated simulators serve as the foundational data infrastructure. However, as computational throughput scales, the underlying trade-offs between parallel efficiency, physical fidelity, and execution determinism remain largely unexamined, hindering the development of reliable robot learning. In this paper, we expose the hidden limits of mainstream GPU-based robotic simulators (e.g., Isaac Lab, Genesis) by introducing GPUSimBench, which focuses on scalability, physical consistency, and computational determinism. First, GPUSimBench establishes a physical grounding evaluation with a controlled inclined-plane task, quantifying the distributional alignment between simulated dynamics and their real-world counterparts. Second, we benchmark parallel scalability by measuring throughput and memory footprints across scaling environment counts. Crucially, beyond standard performance metrics, we unveil and quantify the inherent non-determinism introduced by GPU-batched execution, characterized by significant run-to-run and inter-environment variability even under identical initial conditions. Finally, we identify four empirical regimes of stochasticity within current simulator stacks, highlighting that unbounded scaling can compromise reproducibility without explicit constraints.
Figures
Reference graph
Works this paper leans on
-
[1]
Open x-embodiment: Robotic learning datasets and RT-X models : Open x-embodiment collaboration,
A. O’Neill, A. Rehmanet al., “Open x-embodiment: Robotic learning datasets and RT-X models : Open x-embodiment collaboration,” in IEEE International Conference on Robotics and Automation, ICRA 2024, Yokohama, Japan, May 13-17, 2024, 2024
2024
-
[2]
RT-2: vision-language-action models transfer web knowledge to robotic control,
B. Zitkovich, T. Yuet al., “RT-2: vision-language-action models transfer web knowledge to robotic control,” inCoRL, 2023
2023
-
[3]
RT-1: robotics transformer for real-world control at scale,
A. Brohan, N. Brownet al., “RT-1: robotics transformer for real-world control at scale,” inRSS, 2023
2023
-
[4]
Learning view-invariant world models for visual robotic manipulation,
J. Pang, N. Tanget al., “Learning view-invariant world models for visual robotic manipulation,” inICLR, 2025
2025
-
[5]
Object-oriented option framework for robotics manipulation in clutter,
J. Pang, S. Yanget al., “Object-oriented option framework for robotics manipulation in clutter,” inIROS, 2023
2023
-
[6]
Scaling robot learning with semantically imagined experience,
T. Yu, T. Xiaoet al., “Scaling robot learning with semantically imagined experience,” inRSS, 2023
2023
-
[7]
Data scaling laws in imitation learning for robotic manipulation,
F. Lin, Y . Huet al., “Data scaling laws in imitation learning for robotic manipulation,” inICLR, 2025
2025
-
[8]
Rebot: Scaling robot learning with real-to- sim-to-real robotic video synthesis,
Y . Fang, Y . Yanget al., “Rebot: Scaling robot learning with real-to- sim-to-real robotic video synthesis,” inIROS, 2025
2025
-
[9]
Isaac gym: High performance gpu-based physics simulation for robot learning,
V . Makoviychuk, L. Wawrzyniaket al., “Isaac gym: High performance gpu-based physics simulation for robot learning,” inNeurIPS, 2021
2021
-
[10]
Genesis: A generative and universal physics engine for robotics and beyond,
G. Authors, “Genesis: A generative and universal physics engine for robotics and beyond,” December 2024. [Online]. Available: https://github.com/Genesis-Embodied-AI/Genesis
2024
-
[11]
Residual MPC: blending reinforce- ment learning with gpu-parallelized model predictive control,
S. H. Jeon, H. J. Leeet al., “Residual MPC: blending reinforce- ment learning with gpu-parallelized model predictive control,”arXiv preprint arXiv:2510.12717
-
[12]
Maniskill3: GPU parallelized robotics sim- ulation and rendering for generalizable embodied AI,
S. Tao, F. Xianget al., “Maniskill3: GPU parallelized robotics sim- ulation and rendering for generalizable embodied AI,”arXiv preprint arXiv:2410.00425
-
[13]
Learning to walk in minutes using massively parallel deep reinforcement learning,
N. Rudin, D. Hoelleret al., “Learning to walk in minutes using massively parallel deep reinforcement learning,” inCoRL, 2021
2021
-
[14]
GEAR: A gpu-centric experience replay system for large reinforcement learning models,
H. Wang, M. Sitet al., “GEAR: A gpu-centric experience replay system for large reinforcement learning models,” inICML, 2023
2023
-
[15]
Faster training for robotic manipulation in GPU paral- lelized robotics simulation,
A. Pitkevich, “Faster training for robotic manipulation in GPU paral- lelized robotics simulation,”Int. J. Softw. Sci. Comput. Intell., vol. 17, no. 1, pp. 1–24, 2025
2025
-
[16]
Roboverse: Towards a unified platform, dataset and benchmark for scalable and generalizable robot learning,
H. Geng, F. Wanget al., “Roboverse: Towards a unified platform, dataset and benchmark for scalable and generalizable robot learning,” inRSS, 2025
2025
-
[17]
Realm: A real-to-sim validated benchmark for generalization in robotic manipulation,
M. Sedlacek, P. Yefanovet al., “Realm: A real-to-sim validated benchmark for generalization in robotic manipulation,”arXiv preprint arXiv:2512.19562, 2025
arXiv 2025
-
[18]
Performance comparison of typical physics engines using robot models with multiple joints,
Y . Liu, Q. Wanget al., “Performance comparison of typical physics engines using robot models with multiple joints,”IEEE Robot. Autom. Lett., 2023
2023
-
[19]
A comparative study on physics engines for robot simulation with mechanical interaction,
T. Yoon, J. Lee, and J. Park, “A comparative study on physics engines for robot simulation with mechanical interaction,”Sensors, 2023
2023
-
[20]
Benchmarking population-based reinforcement learning across robotic tasks with gpu-accelerated sim- ulation,
A. A. Shahid, Y . Naranget al., “Benchmarking population-based reinforcement learning across robotic tasks with gpu-accelerated sim- ulation,” inCASE, 2025
2025
-
[21]
Isaac lab: A GPU-accelerated simulation framework for multi-modal robot learning,
M. Mittal, P. Rothet al., “Isaac lab: A GPU-accelerated simulation framework for multi-modal robot learning,” 2025
2025
-
[22]
Maniskill3: GPU parallelized robotics simu- lation and rendering for generalizable embodied AI,
S. Tao, F. Xianget al., “Maniskill3: GPU parallelized robotics simu- lation and rendering for generalizable embodied AI,” inRSS, 2025
2025
-
[23]
An extensible, data-oriented ar- chitecture for high-performance, many-world simulation,
B. Shacklett, L. G. Rosenzweiget al., “An extensible, data-oriented ar- chitecture for high-performance, many-world simulation,”ACM Trans. Graph., vol. 42, no. 4, 2023
2023
-
[24]
Warp: A high-performance python framework for gpu simulation and graphics,
M. Macklin, “Warp: A high-performance python framework for gpu simulation and graphics,” https://github.com/nvidia/warp, March 2022, nVIDIA GPU Technology Conference (GTC)
2022
-
[25]
Mujoco: A physics engine for model-based control,
E. Todorov, T. Erez, and Y . Tassa, “Mujoco: A physics engine for model-based control,” inIROS. IEEE, 2012
2012
-
[26]
Mujoco playground: An open-source framework for gpu-accelerated robot learning and sim-to-real transfer
K. Zakka, B. Tabanpouret al., “Mujoco playground: An open-source framework for gpu-accelerated robot learning and sim-to-real transfer.” 2025
2025
-
[27]
Design and use paradigms for gazebo, an open-source multi-robot simulator,
N. Koenig and A. Howard, “Design and use paradigms for gazebo, an open-source multi-robot simulator,” inIROS, vol. 3. Ieee, 2004, pp. 2149–2154
2004
-
[28]
Pybullet, a python module for physics sim- ulation for games, robotics and machine learning,
E. Coumans and Y . Bai, “Pybullet, a python module for physics sim- ulation for games, robotics and machine learning,” http://pybullet.org, 2016–2021
2016
-
[29]
JAX: composable transformations of Python+NumPy programs,
J. Bradbury, R. Frostiget al., “JAX: composable transformations of Python+NumPy programs,” 2018. [Online]. Available: http: //github.com/jax-ml/jax
2018
-
[30]
Brax – a differentiable physics engine for large scale rigid body simulation,
C. D. Freeman, E. Freyet al., “Brax – a differentiable physics engine for large scale rigid body simulation,” inNeurIPS, 2021
2021
-
[31]
Taichi: a language for high-performance computation on spatially sparse data structures,
Y . Hu, T.-M. Liet al., “Taichi: a language for high-performance computation on spatially sparse data structures,”ACM Trans. Graph., 2019
2019
-
[32]
Taccel: Scaling up vision-based tactile robotics via high-performance gpu simulation,
Y . Li, W. Duet al., “Taccel: Scaling up vision-based tactile robotics via high-performance gpu simulation,” inNeurIPS, 2025
2025
-
[33]
SAPIEN: A simulated part-based interactive environment,
F. Xiang, Y . Qinet al., “SAPIEN: A simulated part-based interactive environment,” inCVPR, 2020
2020
-
[34]
Xpbd: Position-based simulation of compliant constrained dynamics,
M. Macklin, M. M”uller, and N. Chentanez, “Xpbd: Position-based simulation of compliant constrained dynamics,” inMIG, 2016
2016
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.