Pith. sign in

REVIEW 4 major objections 7 minor 6 cited by

Re$^3$Sim: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read RE3SIM's reconstruction-and-rendering pipeline lets policies trained only in simulation transfer zero-shot to a real Franka arm, averaging 58% success.

desk verdict Solid real-to-sim system, but the paper has not shown that its photorealism is what drives the transfer. read the letter →

arxiv 2502.08645 v3 pith:TLJDPW2K submitted 2025-02-12 cs.RO

Xiaoshen Han , Minghuan Liu , Yilun Chen , Junqiu Yu , Xiaoyang Lyu , Yang Tian , Bolun Wang , Weinan Zhang
show 1 more author
Jiangmiao Pang
This is my paper · ORCID
classification cs.RO
keywords real-to-simsim-to-realtransfer3DGaussiansplattingroboticmanipulationimitationlearningzero-shotreconstructionsyntheticdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RE3SIM claims that the usual sim-to-real failure modes—objects with wrong shapes and images that do not look like the real scene—can be engineered away by reconstructing the real environment inside a physics simulator rather than relying on pre-made CAD assets. Given a short video or photo set of a tabletop scene and its objects, the pipeline rebuilds collision meshes for the scene, renders the background with photorealistic 3D Gaussian splatting, aligns everything to the real robot coordinate frame, and fills the simulator with expert demonstrations from a privileged rule-based policy. Training an imitation-learning policy on only that simulated data yields zero-shot transfer to a real Franka arm, with an average success rate above 58% across three tabletop tasks, matching or slightly exceeding a policy trained on 50 real demonstrations. The broader point the authors want to establish is that high-fidelity real-to-sim can make simulation itself a scalable source of generalizable robot training data.

What carries the argument

The load-bearing mechanism is the hybrid real-to-sim reconstruction stack. Background meshes recovered by multi-view stereo give the physics engine collision geometry; the same scene is rendered as a 3D Gaussian splatting field for photorealistic pixels; foreground objects are rendered from textured meshes and composited with the background using Z-buffer depth; and ArUco-marker plus ICP alignment synchronizes the reconstructed scene with the real camera and robot frames. Together these choices separate the problem into where things collide (meshes) and what the camera sees (splatted Gaussians), and the paper argues this separation is why both the geometric and visual gaps stay small. The privileged rule-based policy with reject sampling then converts that aligned simulator into expert trajectories without human teleoperation.

What would settle it

Take a visually identical pair of real objects with very different mass or friction, reconstruct one of them into RE3SIM, train on simulated demonstrations with default physical parameters, and deploy zero-shot on both; if transfer succeeds only on the object whose dynamics match the defaults, the assumption that default physics parameters are adequate is what made the pipeline work.

Watch

Extended reading notes

Core claim

The central claim is a system-level result: closing the geometric and visual gap between a real workspace and its digital twin is sufficient for zero-shot sim-to-real transfer in rigid-object manipulation. RE3SIM reconstructs the background and objects separately, using structure-from-motion and multi-view stereo for collision meshes and Gaussian splatting for photorealistic background rendering, then aligns the reconstructed scene with the real robot via ArUco markers and ICP. Expert trajectories are generated by a privileged rule-based policy with domain randomization and rejection sampling, and an ACT-style policy with a DINOv2 encoder is trained on the rendered two-view images. On pick-and-place, block stacking, and vegetable-placement tasks, the simulated-data policy reaches 75%, 25%, and 75% real-world success respectively (58% average), outperforms the RialTo real-to-sim baseline, and its per-task success correlates with real results at Pearson r=0.924; a larger clear-the-table dataset additionally shows transfer to unseen objects and dark lighting, with success rising as the simulated dataset grows.

Load-bearing premise

The load-bearing premise is that the simulator's default mass and friction values are good enough for the reconstructed objects that the forces and contacts in simulated demonstrations match the real ones; the paper states these defaults are used because measurement is difficult, and no ablation checks whether they matter.

Editorial extensions

If this is right

  • A policy trained exclusively on 100 RE3SIM episodes can match or slightly beat one trained on 50 real teleoperated demonstrations on the same tasks.
  • Simulation success rate is a reliable predictor of real success (Pearson 0.924), so simulated evaluation can stand in for at least some real-world policy filtering.
  • Scaling the simulated dataset from hundreds to thousands of trajectories improves real-world success, suggesting synthetic data obeys a scaling benefit for manipulation policies.
  • Seen and unseen objects, plus changed lighting, were handled by the same policy after training on only five reconstructed objects, indicating generalization beyond the exact reconstructed scene.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its reconstruction cost (roughly minutes of capture and minutes of data generation on parallel GPUs) makes scene cloning cheap enough to repeat across many environments; a natural extension is to pre-train a generalist policy on dozens of reconstructed tables rather than one.
  • The Limitations section explicitly bounds the result to rigid objects, notes physics parameters are manual rather than identified, and expects rule-based data collection to struggle as tasks get harder; those bounds mean the 58% result is evidence for rigid tabletop manipulation, not for all manipulation.
  • A direct experiment the authors did not run is to vary only the real mass or friction of a visually identical object; if the policy tracks the simulated default rather than the real object, then physics identification, not rendering, is the actual bottleneck.
  • The 0.924 sim-real correlation suggests simulators built this way could serve as evaluation arenas for robot foundation models, letting researchers rank checkpoints without expensive real-world trials.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper presents RE3SIM, a real-to-sim-to-real pipeline that reconstructs a tabletop scene and objects using multi-view stereo and 3D Gaussian splatting, then generates expert demonstrations in Isaac Sim/PhysX using a privileged rule-based policy with reject sampling, and trains ACT/DINOv2 policies on simulated images for zero-shot transfer to a real Franka robot. The system is evaluated on three manipulation tasks (pick-and-drop a bottle, place a cucumber on a board, stack cubes) and a four-condition 'clear objects' task. On the three main tasks, policies trained on 100 simulated episodes achieve real-world success rates of 0.75, 0.25, and 0.75 (average 58%), comparable to or slightly better than policies trained on 50 real demonstrations (average 51.7%) and better than the RialTo baseline. The paper also reports rendering-quality comparisons (PSNR/SSIM), a sim-real success-rate correlation of 0.924, pipeline timing, and a data-scaling study. Appendix B includes an ablation of image augmentation that is central to assessing the causal role of the rendering quality.

Significance. If the findings hold, the paper contributes a practical real-to-sim pipeline that reduces both geometric and visual gaps, with a real-robot evaluation against a strong recent baseline (RialTo) and an honest limitations section acknowledging the reliance on default physics parameters and the restriction to rigid objects. The paper also reports a large-scale data generation experiment and a data-scaling curve (Fig. 5) that are useful to the community. The strength of the paper is its end-to-end validation on a real Franka robot, including zero-shot transfer results that, at face value, match or exceed those from 50 real demonstrations on two of three tasks. However, the lack of a matched attribution experiment for the rendering contribution and the absence of uncertainty quantification currently limit the strength of the causal claims.

major comments (4)
  1. [Sec. 4.4 / Appendix B, Table 7] The paper's central causal claim — that the 3D-photorealistic reconstruction/rendering pipeline is what enables zero-shot transfer — is not supported by the evidence as presented. In the single-item picking ablation, removing all image augmentation reduces real-world success from 0.80 to 0.25 (Table 7), while the simulated success remains 0.77; this shows that the learned policy relies on the augmentation recipe, not on the rendered images alone, to transfer. The full system in Table 2 uses the augmented training pipeline, so the headline 58% cannot be attributed to the rendering fidelity without a matched comparison. The paper should report no-augmentation and augmentation-matched results for the three main tasks in Sec. 4.3, and should compare RE3SIM's 3DGS rendering against a cheaper mesh-texture rendering baseline (e.g., OpenMVS, already available from Table 1) under the same augmentation and training protocol. Absent that, the specific contribution of 'high-fidelity 3D-photorealistic real-to-sim' over standard augmentation and a mesh renderer is undetermined.
  2. [Sec. 3.2 (Post-processing) and Limitations] The system sets mass and friction of reconstructed objects to default values, and the Limitations section acknowledges that physics parameters are 'manually defined' rather than system-identified. This assumption is load-bearing for the zero-shot transfer claim: if the default friction/mass are materially wrong for the plastic bottle, cucumber, or black cubes, the simulated demonstrations (including contact forces and grasp success) would not transfer to the real objects. No measurement, simulation-based sensitivity study, or ablation is provided to show that the default values are adequate for these objects. I request either (i) a brief calibration experiment (e.g., measuring friction/mass and re-running transfer), or (ii) a sensitivity ablation that varies friction and mass over plausible ranges and reports real success rates, to support the claimed small physical gap.
  3. [Sec. 4.3, Tables 2 and 4, Fig. 4] All key success rates are point estimates from 20 trials per task (10 in Fig. 4) with no confidence intervals, error bars, or significance tests. For example, the stack-cubes difference between RE3SIM (0.25) and Real+IL (0.15) is within binomial noise for 20 trials, as is the difference between RE3SIM (0.75) and Real+IL (0.8) on the bottle task. The paper should report the number of successes per trial count and binomial confidence intervals (or an equivalent), and should state the number of seeds used for the training runs in Sec. 4.3. This is needed to support any comparative claim, including the headline '58% average'.
  4. [Sec. 4.5, Fig. 1(d)] The Pearson correlation of 0.924 is interpreted as evidence of a 'minimal sim-to-real gap'. However, correlation does not measure calibration: a consistent offset or scale difference between sim and real success rates would still produce a high correlation. The paper should provide a scatter plot with the identity line and report the slope, intercept, and number of evaluation points, and should state how many real-world trials were used per point and over which checkpoints. Only if the points lie near the diagonal can the claim of a small gap be justified.
minor comments (7)
  1. [Sec. 2.2, Eq. (3)] The alpha-blending formula is written with the product over preceding alpha values; the conventional expression is C = sum_i alpha_i prod_{j<i}(1-alpha_j) c_i. As written, the formula is not the standard alpha-compositing equation.
  2. [Sec. 4.2] The sentence '3DGS outperforms Polycam in both RSNR and SSIM' should read 'PSNR' rather than 'RSNR'.
  3. [Appendix B] The list of augmentations includes 'Decocus', which should be 'Defocus'.
  4. [Sec. 3] The phrase 'Physic-based robot simulators' should be 'Physics-based robot simulators'.
  5. [Sec. 4.6, Table 3] The table is titled 'Human effort in reconstruction' but the text states that the total reconstruction time is the sum of background and object reconstruction times; clarify whether the reported values are human-only or wall-clock times, and define 'machine runtime' in the same section.
  6. [Sec. 4.3, Table 2] The text calls RE3SIM 'the best zero-shot sim-to-real performance', but AnyGrasp+Prim achieves 0.9 on the bottle task; since AnyGrasp uses additional primitives, the claim should be scoped to imitation-learned policies trained on simulated data.
  7. [Appendix B, Fig. 7] The x-axis label 'quality' is vague; specify that it is the JPEG quality parameter and give the actual success-rate numbers in the text.

Circularity Check

0 steps flagged · score 1.0 of 10

No load-bearing circularity; pipeline effects are measured against external real-world outcomes, and self-citations appear only as background.

full rationale

RE3SIM's central claim is assessed by zero-shot real-robot success rates, which are external to the pipeline's own construction: the reported 58% average in Table 2 is a measured outcome on a physical Franka robot, not a quantity derived from the reconstruction or rendering equations. The sim-real consistency result (Pearson correlation 0.924 in Fig. 1d) is likewise a measured regression between simulator and real-world rollouts of the same policy checkpoints. No fitted parameter is renamed as a prediction: masses and frictions are explicitly left at default values and acknowledged in the Limitations, and the data-scaling curve in Fig. 5 is an empirical measurement against real-world deployment. Self-citations such as Li et al. 2023 and Tian et al. 2024 appear only in the introduction as examples of prior robot-learning work and are not load-bearing for any central premise or uniqueness claim. The augmentation ablation in Table 7 (0.25 real success without augmentation vs 0.80 with Gaussian Noise on the single-item picking task) is a legitimate attribution concern about how much of the headline transfer is due to rendering fidelity versus standard augmentation, and it is relevant to the paper's causal interpretation; however, it is an experimental-design or correctness issue, not circularity, because the quoted numbers are external measurements and no reported prediction reduces to an input by construction. The limitation that physical parameters are manually set rather than identified is an acknowledged assumption, not a circular definition. Overall, the derivation chain is self-contained: the reconstruction, rendering, simulation-data generation, and real-world evaluation are separate stages verified against real outcomes rather than against the pipeline's own outputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four domain assumptions: reconstruction fidelity, default physics parameters, coordinate alignment accuracy, and the quality of scripted expert demonstrations. There are no mathematical derivations and no new physical entities. The main hand-chosen parameters are the simulator's default dynamics values, the JPEG quality level, and the image augmentation set; the augmentation choice is load-bearing for the zero-shot transfer claim (Table 7).

free parameters (3)
  • Default rigid-body physics parameters (mass, friction) = Isaac Sim and PhysX defaults
    Chosen by hand; the paper states 'acquiring physical parameters like mass and friction is challenging with only initial static visual data, we use default values' (Sec. 3.2). Every simulated expert trajectory depends on these defaults, and they are never measured for the reconstructed objects.
  • JPEG quality level for training images = 40
    Chosen in Appendix B as a storage versus quality trade-off; policies train on images compressed at quality 40 and deploy on uncompressed images, so this choice is part of the data pipeline.
  • Image augmentation recipe (blur, defocus, color jitter, Gaussian noise) = hand-selected combination, Table 7
    Table 7 shows real-world success without augmentation is 0.25 versus 0.8 with the selected augmentations, so this hand-tuned set is load-bearing for the zero-shot claim; it compensates for visual effects 3DGS cannot reproduce.
assumptions (4)
  • domain assumption The reconstructed meshes and 3DGS background faithfully represent the real scene geometry and appearance (COLMAP/OpenMVS for background, ARCode for objects).
    Invoked throughout Sections 3.2 to 3.3; the entire data-generation scheme presumes the reconstruction captures the real scene closely enough that physics simulation and rendered images match reality. Supported only by downstream task success and by Table 1 metrics, which are low in absolute terms.
  • domain assumption Default simulator physics parameters are adequate for the reconstructed rigid objects.
    Explicitly stated in Sec. 3.2 (post-processing): 'we use default values to simplify the process without sacrificing performance.' The zero-shot claim depends on these defaults being close enough to the real mass and friction of the bottle, cucumber, and cubes.
  • domain assumption ArUco marker plus ICP alignment synchronizes real and simulated world coordinates to pixel-level accuracy.
    Sec. 3.4 relies on this for both rendering similarity and task transfer; the paper concedes 'pixel-level deviations' remain, which Figure 3 and Table 1 (PSNR 13.29, SSIM 0.37) confirm.
  • domain assumption A scripted policy with privileged state (exact object poses) produces expert-level demonstrations that imitation learning can emulate.
    Sec. 3.5 generates all training data with a rule-based policy using ground-truth poses and RRTConnect motion planning; if the scripted trajectories are not representative of robust manipulation behavior, the trained policy inherits those limitations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Re$^3$Sim: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation." pith.science (2026). https://pith.science/paper/TLJDPW2K

@misc{pith2026250208645,
  author       = {Pith},
  title        = {Pith review of: Re$^3$Sim: Generating High-Fidelity Simulation Data via 3D-Photorealistic Real-to-Sim for Robotic Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TLJDPW2K}},
  note         = {Machine review of arXiv:2502.08645}
}
abstract

Real-world data collection for robotics is costly and resource-intensive, requiring skilled operators and expensive hardware. Simulations offer a scalable alternative but often fail to achieve sim-to-real generalization due to geometric and visual gaps. To address these challenges, we propose a 3D-photorealistic real-to-sim system, namely, RE$^3$SIM, addressing geometric and visual sim-to-real gaps. RE$^3$SIM employs advanced 3D reconstruction and neural rendering techniques to faithfully recreate real-world scenarios, enabling real-time rendering of simulated cross-view cameras within a physics-based simulator. By utilizing privileged information to collect expert demonstrations efficiently in simulation, and train robot policies with imitation learning, we validate the effectiveness of the real-to-sim-to-real pipeline across various manipulation task scenarios. Notably, with only simulated data, we can achieve zero-shot sim-to-real transfer with an average success rate exceeding 58%. To push the limit of real-to-sim, we further generate a large-scale simulation dataset, demonstrating how a robust policy can be built from simulation data that generalizes across various objects. Codes and demos are available at: http://xshenhan.github.io/Re3Sim/.

Figures

Figures reproduced from arXiv: 2502.08645 by the authors.

Figure 1
Figure 1. Illustration of RE3SIM. a) RE3SIM allows zero-shot policy transfer on various tasks. b) The system pipeline to generate high-quality data. c) High-fidelity rendering results. d) Consistency in success rates between real and simulated environments. Abstract Real-world data collection for robotics is costly and resource-intensive, requiring skilled opera￾tors and expensive hardware. Simulations offer a scalable altern… view at source ↗
Figure 2
Figure 2. Illustration of the proposed real-to-sim-to-real system, RE 3SIM. It leverages 3D reconstruction and a physics￾based simulator, providing small 3D gaps that enable large-scale simulation data generation for learning manipulation skills via sim-to-real transfer. 3.3. Hybrid Visual Rendering Color images, as a key type of perception signal, often yield a large visual gap between simulation and the real world. To close… view at source ↗
Figure 3
Figure 3. Visual comparison between real and simulation. Rendering results from our hybrid rendering method compared with photos captured by real-world cameras, highlighting the high fidelity and realism achieved by our approach [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Real-world evaluation and robustness test for large-scale sim-to-real. The success rate reflects the proportion of trials in which all objects were successfully grasped, while the grasp rate indicates the proportion of objects grasped relative to the total number on th…
Figure 5
Figure 5. Figure 5: Data scaling effects, tested on seen objects in the real world. maintain a relatively higher success rate, while a smaller dataset limits the policy’s ability to exhibit this capability. Furthermore, the policy demonstrates strong robustness to changes in lighting cond…
Figure 6
Figure 6. Figure 6: Visual comparison of rendering approaches. Rendering results of reconstruction outputs from PolyCam, OpenMVS, and 3DGS, compared with real-world photos [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Compression analysis. Qualitative and quantitative analysis using third-person view images to demonstrate the impact of different JPEG quality parameters on image quality, task success rate, and compression ratio in the place a vegetable on the board task. The policies…
Figure 8
Figure 8. Figure 8: Data distribution. We show the initial locations of objects in each task described in Sec. 4.3. There are 100 episodes of simulation data(Sim) and 50 episodes of real-world data(Real). We show each cube’s location in stack blocks tasks separately [PITH_FULL_IMAGE:figu…
Figure 9
Figure 9. Figure 9: Data quality. Analysis of differences between real-world and simulation data properties. The first row presents statistical data on the distribution of gripper angles per time step across episodes. The second row displays the Kernel Density Estimate (KDE) of episode le…
Figure 10
Figure 10. Figure 10: Features of co-training and fine-tuning models. (a) KDE of the Euclidean distance traveled by the robotic arm’s end effector between adjacent time steps. (b) The number of time steps taken by the robotic arm from the start of movement to the first closure of the gripp…
Figure 11
Figure 11. Figure 11: Visual comparison between RE 3SIM and OpenMVS. 6 [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: Rendering results of place a vegetable on the board task. 7 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Rendering results of stack blocks task. 8 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]
Figure 14
Figure 14. Figure 14: Rendering results of clear objects on the table task. 9 [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. B-spline Policy: Accelerating Manipulation Policies via B-spline Action Representations

    cs.RO 2026-07 conditional novelty 6.0 of 10

    Predicting continuous B-spline action curves instead of discrete chunks lets visuomotor policies finish real and simulated manipulation tasks substantially faster without large success-rate loss.

  2. SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

    cs.RO 2026-06 unverdicted novelty 6.0 of 10

    SimFoundry automates zero-shot real-to-sim scene generation from video, producing digital twins and cousins that enable policy training with 0.911 mean Pearson correlation to real-world results and 17-40% success gain...

  3. Learning in ImaginationLand: Omnidirectional Policies through 3D Generative Models (OP-Gen)

    cs.RO 2025-09 conditional novelty 6.0 of 10

    A robot policy trained on one real demonstration plus AI-generated 3D views succeeds from novel initial poses, including opposite-side starts, across six real manipulation tasks.

  4. DSG-World: Learning a 3D Gaussian World Model from Dual State Videos

    cs.CV 2025-06 conditional novelty 6.0 of 10

    DSG-World builds two segmented 3D Gaussian fields from two scene states and trains them with mutual consistency, enabling novel-state simulation without inpainting or dense capture.

  5. SimLauncher: Launching Sample-Efficient Real-world Robotic Reinforcement Learning via Simulation Pre-training

    cs.RO 2025-07 conditional novelty 5.0 of 10

    Simulation-pretrained policies, with digital-twin demos for critic bootstrapping and action proposals, cut real-world RL training time while reaching near-perfect success on three manipulation tasks.

  6. Bootstrapping Imitation Learning for Long-horizon Manipulation via Hierarchical Data Collection Space

    cs.RO 2025-05 conditional novelty 5.0 of 10

    Breaking long manipulation tasks into atomic subtasks and collecting demonstrations from varied starting poses improves imitation learning success using fewer demonstration frames.

Reference graph

Works this paper leans on

70 extracted references · 35 canonical work pages · cited by 6 Pith papers

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Aloha 2: An enhanced low-cost hardware for bimanual teleoperation

    Aldaco, J., Armstrong, T., Baruch, R., Bingham, J., Chan, S., Draper, K., Dwibedi, D., Finn, C., Florence, P., Goodrich, S., et al. Aloha 2: An enhanced low-cost hardware for bimanual teleoperation. arXiv preprint arXiv:2405.02292, 2024

    arXiv 2024

  3. [3]

    S., Pryor, M., and Thomaz, A

    Allevato, A., Short, E. S., Pryor, M., and Thomaz, A. Tunenet: One-shot residual tuning for system identification and sim-to-real robot task transfer. In Kaelbling, L. P., Kragic, D., and Sugiura, K. (eds.), Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp.\ 445--455. PMLR, 30 Oct--01 Nov 2020 a ....

    work page 2020

  4. [4]

    D., Schaertl Short, E., Pryor, M., and Thomaz, A

    Allevato, A. D., Schaertl Short, E., Pryor, M., and Thomaz, A. L. Iterative residual tuning for system identification and sim-to-real robot learning. Autonomous Robots, 44 0 (7): 0 1167--1182, 2020 b

    work page 2020

  5. [5]

    _0 : A vision-language-action flow model for general robot control

    Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. _0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024

    arXiv

  6. [6]

    Blinn, J. F. and Newell, M. E. Texture and reflection in computer generated images. Commun. ACM, 19 0 (10): 0 542–547, October 1976. ISSN 0001-0782. doi:10.1145/360349.360353. URL https://doi.org/10.1145/360349.360353

    arXiv 1976

  7. [7]

    Rt-1: Robotics transformer for real-world control at scale

    Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022

    arXiv 2022

  8. [8]

    Rt-2: Vision-language-action models transfer web knowledge to robotic control

    Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023

    arXiv 2023

Show all 70 references
  1. [9]

    I., Khvedchenya, E., Parinov, A., Druzhinin, M., and Kalinin, A

    Buslaev, A., Iglovikov, V. I., Khvedchenya, E., Parinov, A., Druzhinin, M., and Kalinin, A. A. Albumentations: Fast and flexible image augmentations. Information, 11 0 (2), 2020. ISSN 2078-2489. doi:10.3390/info11020125. URL https://www.mdpi.com/2078-2489/11/2/125

    2020 doi

  2. [10]

    An icp variant using a point-to-line metric

    Censi, A. An icp variant using a point-to-line metric. In 2008 IEEE international conference on robotics and automation, pp.\ 19--25. Ieee, 2008

    2008

  3. [11]

    OpenMVS : Multi-view stereo reconstruction library

    Cernea, D. OpenMVS : Multi-view stereo reconstruction library. 2020. URL https://cdcseacave.github.io/openMVS

    2020

  4. [12]

    Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation

    Cheang, C.-L., Chen, G., Jing, Y., Kong, T., Li, H., Li, Y., Liu, Y., Wu, H., Xu, J., Yang, Y., Zhang, H., and Zhu, M. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024

    arXiv

  5. [13]

    Understanding domain randomization for sim-to-real transfer

    Chen, X., Hu, J., Jin, C., Li, L., and Wang, L. Understanding domain randomization for sim-to-real transfer. arXiv preprint arXiv:2110.03239, 2021

    2021 arXiv

  6. [14]

    Urdformer: A pipeline for constructing articulated simulation environments from real-world images

    Chen, Z., Walsman, A., Memmel, M., Mo, K., Fang, A., Vemuri, K., Wu, A., Fox, D., and Gupta, A. Urdformer: A pipeline for constructing articulated simulation environments from real-world images. arXiv preprint arXiv:2405.11656, 2024

    2024 arXiv

  7. [15]

    Open-television: Teleoperation with immersive active visual feedback

    Cheng, X., Li, J., Yang, S., Yang, G., and Wang, X. Open-television: Teleoperation with immersive active visual feedback. arXiv preprint arXiv:2407.01512, 2024

    2024 arXiv

  8. [16]

    Code, A. Ar code. https://ar-code.com/, 2022

    2022

  9. [17]

    and Bai, Y

    Coumans, E. and Bai, Y. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pybullet.org, 2016--2021

    2016

  10. [18]

    Acdc: Automated creation of digital cousins for robust policy learning

    Dai, T., Wong, J., Jiang, Y., Wang, C., Gokmen, C., Zhang, R., Wu, J., and Fei-Fei, L. Acdc: Automated creation of digital cousins for robust policy learning. arXiv e-prints, pp.\ arXiv--2410, 2024

    2024

  11. [19]

    Anygrasp: Robust and efficient grasp perception in spatial and temporal domains

    Fang, H.-S., Wang, C., Fang, H., Gou, M., Liu, J., Yan, H., Liu, W., Xie, Y., and Lu, C. Anygrasp: Robust and efficient grasp perception in spatial and temporal domains. IEEE Transactions on Robotics, 2023

    2023

  12. [20]

    Z., and Finn, C

    Fu, Z., Zhao, T. Z., and Finn, C. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation. In Conference on Robot Learning (CoRL) , 2024

    2024

  13. [21]

    and Lepetit, V

    Gu \'e don, A. and Lepetit, V. Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 5354--5363, 2024

    2024

  14. [22]

    Graspsplats: Efficient manipulation with 3d feature splatting

    Ji, M., Qiu, R.-Z., Zou, X., and Wang, X. Graspsplats: Efficient manipulation with 3d feature splatting. arXiv preprint arXiv:2409.02084, 2024

    2024 arXiv

  15. [23]

    Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning

    Jiang, Z., Xie, Y., Lin, K., Xu, Z., Wan, W., Mandlekar, A., Fan, L., and Zhu, Y. Dexmimicgen: Automated data generation for bimanual dexterous manipulation via imitation learning. 2024

    2024

  16. [24]

    3d gaussian splatting for real-time radiance field rendering

    Kerbl, B., Kopanas, G., Leimk \"u hler, T., and Drettakis, G. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42 0 (4), July 2023. URL https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/

    2023

  17. [25]

    Evo-nerf: Evolving nerf for sequential robot grasping of transparent objects

    Kerr, J., Fu, L., Huang, H., Avigal, Y., Tancik, M., Ichnowski, J., Kanazawa, A., and Goldberg, K. Evo-nerf: Evolving nerf for sequential robot grasping of transparent objects. In Conference on Robot Learning, pp.\ 353--367. PMLR, 2023

    2023

  18. [26]

    K., Chen, L

    Khazatsky, A., Pertsch, K., Nair, S., Balakrishna, A., Dasari, S., Karamcheti, S., Nasiriany, S., Srirama, M. K., Chen, L. Y., Ellis, K., Fagan, P. D., Hejna, J., Itkina, M., Lepert, M., Ma, Y. J., Miller, P. T., Wu, J., Belkhale, S., Dass, S., Ha, H., Jain, A., Lee, A., Lee, ...

    2024 arXiv

  19. [27]

    J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al

    Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024

    2024 arXiv

  20. [28]

    and Howard, A

    Koenig, N. and Howard, A. Design and use paradigms for gazebo, an open-source multi-robot simulator. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566), volume 3, pp.\ 2149--2154 vol.3, 2004. doi:10.1109/IROS.2004.1389727

    2004 arXiv

  21. [29]

    and LaValle, S

    Kuffner, J. and LaValle, S. Rrt-connect: An efficient approach to single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065), volume 2, pp.\ 995--1001 vol.2, 20...

    2000

  22. [30]

    J., Vedder, K., Krishna, A., Jayaraman, D., and Eaton, E

    Le, L., Xie, J., Liang, W., Wang, H.-J., Yang, Y., Ma, Y. J., Vedder, K., Krishna, A., Jayaraman, D., and Eaton, E. Articulate-anything: Automatic modeling of articulated objects via a vision-language foundation model. arXiv preprint arXiv:2410.13882, 2024

    arXiv

  23. [31]

    Grey, M., Ha, S., Kunz, T., Jain, S., Ye, Y., S

    Lee, J., X. Grey, M., Ha, S., Kunz, T., Jain, S., Ye, Y., S. Srinivasa, S., Stilman, M., and Karen Liu, C. Dart: Dynamic animation and robotics toolkit. The Journal of Open Source Software, 3 0 (22): 0 500, 2018

    2018

  24. [32]

    Vision-language foundation models as effective robot imitators

    Li, X., Liu, M., Zhang, H., Yu, C., Xu, J., Wu, H., Cheang, C., Jing, Y., Zhang, W., Liu, H., et al. Vision-language foundation models as effective robot imitators. arXiv preprint arXiv:2311.01378, 2023

    2023 arXiv

  25. [33]

    Robogsim: A real2sim2real robotic gaussian splatting simulator

    Li, X., Li, J., Zhang, Z., Zhang, R., Jia, F., Wang, T., Fan, H., Tseng, K.-K., and Wang, R. Robogsim: A real2sim2real robotic gaussian splatting simulator. arXiv preprint arXiv:2411.11839, 2024 a

    2024 arXiv

  26. [34]

    Towards generalist robot policies: What matters in building vision-language-action models

    Li, X., Li, P., Liu, M., Wang, D., Liu, J., Kang, B., Ma, X., Kong, T., Zhang, H., and Liu, H. Towards generalist robot policies: What matters in building vision-language-action models. arXiv preprint arXiv:2412.14058, 2024 b

    2024 arXiv

  27. [35]

    Deep drone racing: From simulation to reality with domain randomization

    Loquercio, A., Kaufmann, E., Ranftl, R., Dosovitskiy, A., Koltun, V., and Scaramuzza, D. Deep drone racing: From simulation to reality with domain randomization. IEEE Transactions on Robotics, 36 0 (1): 0 1--14, 2019

    2019

  28. [36]

    Robo-gs: A physics consistent spatial-temporal model for robotic arm with hybrid representation

    Lou, H., Liu, Y., Pan, Y., Geng, Y., Chen, J., Ma, W., Li, C., Wang, L., Feng, H., Shi, L., et al. Robo-gs: A physics consistent spatial-temporal model for robotic arm with hybrid representation. arXiv preprint arXiv:2408.14873, 2024

    2024 arXiv

  29. [37]

    Linear least-squares optimization for point-to-plane icp surface registration

    Low, K.-L. Linear least-squares optimization for point-to-plane icp surface registration. Chapel Hill, University of North Carolina, 4 0 (10): 0 1--3, 2004

    2004

  30. [38]

    Isaac gym: High performance gpu-based physics simulation for robot learning, 2021

    Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., and State, G. Isaac gym: High performance gpu-based physics simulation for robot learning, 2021

    2021

  31. [39]

    Mimicgen: A data generation system for scalable robot learning using human demonstrations

    Mandlekar, A., Nasiriany, S., Wen, B., Akinola, I., Narang, Y., Fan, L., Zhu, Y., and Fox, D. Mimicgen: A data generation system for scalable robot learning using human demonstrations. In 7th Annual Conference on Robot Learning, 2023

    2023

  32. [40]

    J., and Paull, L

    Mehta, B., Diaz, M., Golemo, F., Pal, C. J., and Paull, L. Active domain randomization. In Conference on Robot Learning, pp.\ 1162--1176. PMLR, 2020

    2020

  33. [41]

    P., Tancik, M., Barron, J

    Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65 0 (1): 0 99--106, 2021

    2021

  34. [42]

    L., Singh, R., Guo, Y., Mazhar, H., Mandlekar, A., Babich, B., State, G., Hutter, M., and Garg, A

    Mittal, M., Yu, C., Yu, Q., Liu, J., Rudin, N., Hoeller, D., Yuan, J. L., Singh, R., Guo, Y., Mazhar, H., Mandlekar, A., Babich, B., State, G., Hutter, M., and Garg, A. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automat...

    2023

  35. [43]

    Robocasa: Large-scale simulation of everyday tasks for generalist robots

    Nasiriany, S., Maddukuri, A., Zhang, L., Parikh, A., Lo, A., Joshi, A., Mandlekar, A., and Zhu, Y. Robocasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS), 2024

    2024

  36. [44]

    Nvidia isaac sim, 2021 a

    NVIDIA. Nvidia isaac sim, 2021 a . URL https://developer.nvidia.com/isaac-sim

    2021

  37. [45]

    Nvidia physx, 2021 b

    NVIDIA. Nvidia physx, 2021 b . URL https://nvidia-omniverse.github.io/PhysX/

    2021

  38. [46]

    Open x-embodiment: Robotic learning datasets and rt-x models

    O'Neill, A., Rehman, A., Gupta, A., Maddukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.08864, 2023

    arXiv

  39. [47]

    Dinov2: Learning robust visual features without supervision

    Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  40. [48]

    Polycam. Polycam. https://poly.cam, 2020

    2020

  41. [49]

    N., Garg, S., Yandun, F., Held, D., Kantor, G., and Silwal, A

    Qureshi, M. N., Garg, S., Yandun, F., Held, D., Kantor, G., and Silwal, A. Splatsim: Zero-shot sim2real transfer of rgb manipulation policies using gaussian splatting. arXiv preprint arXiv:2409.10161, 2024

    2024 arXiv

  42. [50]

    and Tangirala, A

    Ramaswamy, J. and Tangirala, A. K. Enhancing system identification through transfer learning in gaussian process models: Bridging sim-to-real and cross-environment applications. In 2024 SICE Festival with Annual Conference (SICE FES), pp.\ 806--811, 2024

    2024

  43. [51]

    M., Kerr, J., Chen, L

    Rashid, A., Sharma, S., Kim, C. M., Kerr, J., Chen, L. Y., Kanazawa, A., and Goldberg, K. Language embedded radiance fields for zero-shot task-oriented grasping. In 7th Annual Conference on Robot Learning, 2023. URL https://openreview.net/forum?id=k-Fg8JDQmc

    2023

  44. [52]

    Schenker, P. S. Sensor fusion iv: Control paradigms and data structures; proceedings of the meeting, boston, ma, nov. 12-15, 1991. Number SPIE-1611. Society of Photo-Optical Instrumentation Engineers (SPIE Proceedings. Vol. 1611), 1992

    1991

  45. [53]

    Sch\" o nberger, J. L. and Frahm, J.-M. Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016

    2016

  46. [54]

    L., Zheng, E., Pollefeys, M., and Frahm, J.-M

    Sch\" o nberger, J. L., Zheng, E., Pollefeys, M., and Frahm, J.-M. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016

    2016

  47. [55]

    A., Seth, A., and Delp, S

    Sherman, M. A., Seth, A., and Delp, S. L. Simbody: multibody dynamics for biomedical research. Procedia Iutam, 2: 0 241--261, 2011

    2011

  48. [56]

    D., and Schwager, M

    Shorinwa, O., Tucker, J., Smith, A., Swann, A., Chen, T., Firoozi, R., Kennedy, M. D., and Schwager, M. Splat-mover: Multi-stage, open-vocabulary robotic manipulation via editable gaussian splatting. 2024

    2024

  49. [57]

    Smith., R. L. Open dynamics engine (ode). https://ode.org, 2001

    2001

  50. [58]

    Sim-to-real in unmanned surface vehicle control: A system identification-based approach for enhanced training environments

    Song, R., Gao, S., and Li, Y. Sim-to-real in unmanned surface vehicle control: A system identification-based approach for enhanced training environments. In 2024 9th International Conference on Electronic Technology and Information Science (ICETIS), pp.\ 563--570, 2024. doi:10...

    2024

  51. [59]

    Predictive inverse dynamics models are scalable learners for robotic manipulation

    Tian, Y., Yang, S., Zeng, J., Wang, P., Lin, D., Dong, H., and Pang, J. Predictive inverse dynamics models are scalable learners for robotic manipulation. arXiv preprint arXiv:2412.15109, 2024

    2024 arXiv

  52. [60]

    Domain randomization for transferring deep neural networks from simulation to the real world

    Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp.\ 23--30. IEEE, 2017

    2017

  53. [61]

    Domain randomization and generative models for robotic grasping

    Tobin, J., Biewald, L., Duan, R., Andrychowicz, M., Handa, A., Kumar, V., McGrew, B., Ray, A., Schneider, J., Welinder, P., et al. Domain randomization and generative models for robotic grasping. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS...

    2018

  54. [62]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109

    2012

  55. [63]

    Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation

    Torne, M., Simeonov, A., Li, Z., Chan, A., Chen, T., Gupta, A., and Agrawal, P. Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation. Arxiv, 2024

    2024

  56. [64]

    Training deep networks with synthetic data: Bridging the reality gap by domain randomization

    Tremblay, J., Prakash, A., Acuna, D., Brophy, M., Jampani, V., Anil, C., To, T., Cameracci, E., Boochoon, S., and Birchfield, S. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In Proceedings of the IEEE conference on computer visi...

    2018

  57. [65]

    X., Guibas, L

    Xiang, F., Qin, Y., Mo, K., Xia, Y., Zhu, H., Liu, F., Liu, M., Jiang, H., Yuan, Y., Wang, H., Yi, L., Chang, A. X., Guibas, L. J., and Su, H. SAPIEN : A simulated part-based interactive environment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020

    2020

  58. [66]

    Ace: A cross-platfrom visual-exoskeletons for low-cost dexterous teleoperation

    Yang, S., Liu, M., Qin, Y., Runyu, D., Jialong, L., Cheng, X., Yang, R., Yi, S., and Wang, X. Ace: A cross-platfrom visual-exoskeletons for low-cost dexterous teleoperation. arXiv preprint arXiv:240, 2024

    2024

  59. [67]

    Mvsnet: Depth inference for unstructured multi-view stereo

    Yao, Y., Luo, Z., Li, S., Fang, T., and Quan, L. Mvsnet: Depth inference for unstructured multi-view stereo. European Conference on Computer Vision (ECCV), 2018

    2018

  60. [68]

    Gsdf: 3dgs meets sdf for improved rendering and reconstruction

    Yu, M., Lu, T., Xu, L., Jiang, L., Xiangli, Y., and Dai, B. Gsdf: 3dgs meets sdf for improved rendering and reconstruction. arXiv preprint arXiv:2403.16964, 2024

    2024 arXiv

  61. [69]

    Z., Kumar, V., Levine, S., and Finn, C

    Zhao, T. Z., Kumar, V., Levine, S., and Finn, C. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023

    2023 arXiv

  62. [70]

    Gaussiangrasper: 3d language gaussian splatting for open-vocabulary robotic grasping

    Zheng, Y., Chen, X., Zheng, Y., Gu, S., Yang, R., Jin, B., Li, P., Zhong, C., Wang, Z., Liu, L., et al. Gaussiangrasper: 3d language gaussian splatting for open-vocabulary robotic grasping. arXiv preprint arXiv:2403.09637, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.