DSG-World: Learning a 3D Gaussian World Model from Dual State Videos

Gaoang Wang; Wenhao Hu; Xi Li; Xuexiang Wen

REVIEW 4 major objections 5 minor 46 references

DSG-World: Learning a 3D Gaussian World Model from Dual State Videos

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DSG-World builds a complete, editable 3D Gaussian world model from two scene states in different object configurations, using cross-state supervision instead of inpainting or dense capture.

desk verdict A practical dual-state Gaussian world model with a real contribution, but the pseudo-state construction is underspecified enough that the headline numbers need scrutiny before I'd trust the generalization claim. read the letter →

arxiv 2506.05217 v1 pith:GZSIKP3P submitted 2025-06-05 cs.CV

Wenhao Hu , Xuexiang Wen , Xi Li , Gaoang Wang This is my paper

classification cs.CV

keywords 3DGaussianSplattingworldmodeldual-statevideoreal2simscenereconstructionobjectmanipulationpseudo-statealignmentco-pruning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DSG-World claims that a complete, editable 3D Gaussian scene model can be built from just two static observations of the same scene in different object configurations, without inpainting or dense multi-view capture. The two states act as complementary views, revealing background regions that are occluded in the other. Two segmented Gaussian fields are optimized together, each transformed into the other state's configuration and supervised for photometric and semantic agreement, with a virtual intermediate state providing symmetric alignment. If the claim holds, minimal real-world capture is enough to build a high-fidelity simulator in which objects can be moved and rendered under arbitrary configurations.

What carries the argument

The central object is the dual segmented Gaussian field pair ($G_1$, $G_2$), where each Gaussian carries position, covariance, opacity, color, and a learnable segmentation identity. The key operator is an object-aware rigid transformation $T$ that moves each foreground Gaussian according to the object it belongs to while leaving background Gaussians fixed. Three mechanisms carry the argument: bidirectional alignment, which renders each transformed field into the other state's viewpoints and imposes photometric and segmentation-consistency losses; pseudo-state alignment, which maps both fields to a shared virtual intermediate configuration and matches their renders; and collaborative co-pruning, which discards transferred Gaussians whose nearest neighbor in the target field is farther than a threshold $\tau$. The reconstruction loss is inherited from Gaussian Grouping, and the two fields plus a shared segmentation classifier are optimized jointly.

What would settle it

Capture a scene twice with a deliberately unobserved region, such as a background patch hidden by a different object in each state, run DSG-World, and render a novel configuration: if the hidden patch is missing or shows artifacts, the claim that two complementary states suffice for complete reconstruction fails.

Watch

Extended reading notes

Core claim

The central claim is that two perturbed observations of a scene supply enough complementary information to reconstruct a complete, segmentation-aware Gaussian world model end-to-end. The paper builds two segmented Gaussian fields, transfers each into the other state's configuration using known rigid per-object transformations, and supervises them bidirectionally so that each field explains the other state's images and masks. A pseudo-intermediate Gaussian field, formed by transforming both fields toward a shared virtual configuration, adds a symmetric alignment signal, and collaborative co-pruning removes transferred Gaussians that disagree with the target state's geometry. The paper reports higher PSNR and SSIM for novel-state simulation than segmentation-only and segmentation-plus-inpainting baselines on its synthetic and real datasets.

Load-bearing premise

The method assumes the exact per-object rigid transformations between the two captured states are known in advance, and that the two states are complementary enough that every part of the scene is visible in at least one of them; if either assumption fails, the cross-state alignment and co-pruning cannot recover the missing geometry.

Editorial extensions

If this is right

A robot or simulation system could start from two short captures instead of dense per-object scans, because the complementary states fill each other's occluded regions during optimization.
Object-level editing and novel-state simulation reduce to applying a rigid transformation to the appropriate Gaussians, bypassing latent-space generation and image inpainting entirely.
Because both Gaussian fields are forced to agree through a shared pseudo-state, the reconstructed model behaves consistently no matter which observed state is used as the starting reference.
On the paper's datasets, the dual-state method reports higher simulation fidelity than segmentation-only and segmentation-plus-inpainting baselines, indicating that cross-state supervision removes the background-hole problem rather than patching it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension is to estimate the per-object rigid transformations jointly with the Gaussian parameters, since the bidirectional alignment loss already backpropagates through object positions.
The pseudo-state idea extends to several interpolated configurations along an object's trajectory; the paper's own ablation shows pseudo-state supervision is the largest contributor to its PSNR gain, so a denser set could improve alignment further.
Extending the method to non-rigid or articulated objects would require replacing the rigid operator $T$ with a per-part or deformation-aware mapping, a change the current co-pruning distance rule does not cover.
Because the output is an explicit Gaussian field tied to known object transformations, it could plug directly into a physics simulator that supplies per-timestep poses, enabling closed-loop control without retraining the model.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea is genuinely useful: instead of reconstructing one scene state and inpainting occluded background, take two states with complementary visibility and let them supervise each other through known rigid object transforms. That removes a whole class of multi-stage artifacts, and the reported gains over single-state baselines (Table 1, especially the synthetic PSNR jump to 38.37) are large enough to take seriously. The bidirectional alignment with photometric and semantic consistency is a clean formulation, and the ablation in Table 2 shows each ingredient helps, with pseudo-state giving the biggest boost. I believe the central claim is plausible and the direction is right for real-to-sim transfer.

The soft spots are real but not fatal. The pseudo-state term is the largest contributor but the construction is genuinely under-specified: Section 4.2 says it is synthesized via geometric constraints but never defines the transformation matrices T1→p and T2→p, and nothing rules out that the chosen pseudo-state overlaps the held-out test state. If the pseudo-state is just the test configuration, the 38.37 PSNR is inflated and the novelty-state generalization claim weakens. This is the main thing I would push on. Second, the evaluation is thin: 7 synthetic and 5 real scenes, no error bars, no code or data. The comparisons are against baselines that were adapted in non-trivial ways (inpainting variants), so I'd want the exact protocol described. Third, the known-transformation assumption (Eq. 6) is stated but never stress-tested; a sensitivity analysis on perturbed T(o) would make the method far more credible for practical use. The paper's own limitations section admits the complementary-visibility condition, which is honest.

For a serious referee: yes. The concept is solid, the experiments support the components in the ablation, and the limitations are stated rather than hidden. But it needs a substantial revision to specify the pseudo-state construction, isolate the test state, add error bars, and release data or code. My verdict is conditional: I would not accept it as is, but I would engage with a revised version and I'd send it to review rather than desk reject.

Referee Report

4 major / 5 minor

Summary. The paper proposes DSG-World, a 3D Gaussian world model trained from two observations ('dual states') of a scene under different object configurations. The method builds two segmentation-aware Gaussian fields and jointly optimizes them using bidirectional photometric and semantic consistency, a pseudo-intermediate state for symmetric alignment, and collaborative co-pruning/co-pasting for geometric completeness. Experiments on 7 synthetic and 5 real scenes report PSNR/SSIM improvements over single-state segmentation-plus-inpainting baselines, and an ablation attributes the largest gain to pseudo-state supervision.

Significance. If the reported results hold, DSG-World offers a practical alternative to multi-stage inpainting pipelines for real-to-sim transfer, requiring only two observations and no dense per-object scans. The explicit Gaussian representation and object-level rigid transformations enable direct scene manipulation and novel-state rendering. The paper is transparent about several limitations, including complementary-visibility requirements and lack of relighting. However, the evaluation is small, lacks error bars, and no code or data are released, and the pseudo-state component that drives the main reported improvement is not specified. The central contribution is therefore not yet independently verifiable, though the idea is promising and clearly presented in its overall structure.

major comments (4)

[Sec. 4.2, Eq. (8)] The pseudo-state Gp and the transformations T1→p and T2→p are not defined. The text says Gp is 'constructed by applying geometric constraints, such as collision and boundary regularization' and that T1→p and T2→p are 'computed', but no formula, algorithm, or parameterization is given for the pseudo-state poses. Because Table 2 shows that removing pseudo-state supervision drops PSNR from 38.37 to 36.57, this omission is load-bearing: without a precise construction the method cannot be reproduced, and the source of the largest reported gain cannot be audited.
[Sec. 5 and Sec. 4.2] The evaluation protocol may be circular. The test state is created by 'randomly repositioning objects', while the pseudo-state is 'synthesized' between the two observed states. The paper never states that the pseudo-state configuration is sampled independently of the held-out test configuration, nor that the same random process is avoided. If the pseudo-state is at or near the test configuration, Lpseudo supervises the two fields with the exact target, so the 38.37 PSNR in Table 1 would reflect fitting rather than novel-state generalization. Please specify the relationship between pseudo-state sampling and test-state sampling, and report results with pseudo-states sampled independently from the test distribution.
[Sec. 6.2, Table 1] The empirical claims rest on only 7 synthetic and 5 real scenes, with no per-scene numbers, no error bars, and no significance tests. While the reported margins over baselines are large (38.37 vs 30.88 PSNR on synthetic data), the absence of variance information makes it impossible to judge stability across scenes. Adding error bars or per-scene tables is essential for the claimed 'strong generalization' to novel views and states.
[Sec. 3.4-3.5 and Sec. 7] The method assumes exact known per-object rigid transformations T(o), but the paper provides no sensitivity analysis to pose errors and no statement of how T(o) is obtained for the real-world captures. Section 7 acknowledges the complementary-visibility assumption but not pose uncertainty. Since all cross-state losses (Eqs. 7-8) and the final composition (Eq. 6) depend on T(o), a small perturbation analysis or discussion of pose estimation error is needed to establish robustness in the real2sim setting.

minor comments (5)

[Abstract, Sec. 6.2, Figure 2] The manuscript contains several typos: 'trategies' should be 'strategies' in the abstract, 'datasts' should be 'datasets' in Sec. 6.1, 'Guassian' should be 'Gaussian' in Sec. 6.2, and 'Aera after removal' in Figure 2 should be 'Area after removal'.
[Sec. 6.3, Table 3] 'Sim1' and 'Sim2' are never defined. Please state which scenes these are and why only two scenes are used for the convergence experiment.
[Sec. 1] The sentence 'We argue that remains unresolved:' is grammatically incomplete; it should read 'We argue that the following question remains unresolved:'.
[Sec. 6.1] In the implementation details, 'we set λa = 1.0 and λ=1.0' uses a bare λ; it should be λp = 1.0 for consistency with Eq. (10).
[Figure 3] The spelling 'Gaussiancut' in the figure caption is inconsistent with the reference 'GaussianCut [15]' in the bibliography.

Circularity Check

0 steps flagged · score 0.0 of 10

No demonstrated circularity; the pseudo-state construction is underspecified but no equation reduces the reported novel-state results to a fitted target.

full rationale

The derivation is self-contained and non-circular. The two Gaussian fields are optimized by Lr against the observed images and masks of each state (Eq. 10); Lalign (Eq. 7) supervises each transformed field against the actual other-state observations; and the co-pruning rule (Eq. 9) is a geometric filter, not a fitted target. The novel-state evaluation in Sec. 5 holds out a test state created by random object repositioning and compares rendered images to ground-truth images, so the numbers in Tables 1 and 2 are not used as training targets. The only point that could raise a concern is the unspecified construction of the pseudo-state in Sec. 4.2: no formula for T1→p or T2→p is given, and therefore the text does not allow one to verify that pseudo-state configurations are disjoint from the held-out test states. However, the paper never states that the pseudo-state equals a test configuration, and the main reconstruction terms are independently supervised. Under the hard rule that circularity requires a demonstrated Eq.-to-Eq. or fitted-parameter reduction, this is an evaluation-validity limitation rather than a circular step. There is also no load-bearing self-citation: reference [12] appears only as related work, and the central claim does not depend on it. No prediction in the paper reduces, by construction, to its own input.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The method's novelty is a training scheme, not a derived law. The load-bearing inputs are known rigid transformations, complementary states, and 2D masks, plus an unspecified pseudo-state construction. The hand-chosen hyperparameters tau and lambda are minor but not free of tuning.

free parameters (2)

Pruning threshold tau = 0.5
Selected by hand in Section 6.1; controls which transferred Gaussians are discarded in Eq. 9. The ablation shows co-pruning contributes modestly, but the final method depends on this choice.
Loss weights lambda_a and lambda_p = 1.0 and 1.0
Chosen by hand in Section 6.1; balance reconstruction, bidirectional alignment, and pseudo-state losses in Eq. 10. No sensitivity study is reported.

assumptions (5)

standard math 3D Gaussian Splatting rendering: projected alpha-blending produces differentiable RGB and segmentation images (Eq. 1).
Taken from prior work [16,38] and used as the renderer for all losses.
domain assumption Foreground objects move as rigid bodies with known per-object transformations T(o) between the two states.
Invoked in Section 3.4-3.5 (Eqs. 4-6); no pose estimation or uncertainty handling is presented, and evaluation assumes T is given.
domain assumption The pair of states is complementary: every background region is visible in at least one state and lighting is constant.
Stated as dataset design in Section 5 and acknowledged as a limitation in Section 7; if violated, occluded areas cannot be recovered from the alternate state.
domain assumption 2D instance masks S1 and S2 are available and accurate enough to initialize segmentation features.
Section 3.4 treats masks as input; the method corrects boundary errors only partially via co-pruning, and there is no sensitivity analysis to mask quality.
ad hoc to paper A pseudo-state Gp can be constructed by geometric constraints such as collision and boundary regularization, with the explicit formula supplied in an appendix.
Section 4.2 defines Gp and the associated losses but defers the construction of T1->p and T2->p; the main-text method depends on this auxiliary state existing as described.

invented entities (1)

Pseudo-state Gaussian field Gp
purpose: A virtual intermediate configuration used as a shared reference to align G1 and G2 (Section 4.2, Eq. 8).
It is not observed in data and has no external falsifiable handle; it is an auxiliary supervision target. This is a modeling choice, not a physical prediction, but its construction must be specified for the method to be reproducible.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DSG-World: Learning a 3D Gaussian World Model from Dual State Videos." pith.science (2026). https://pith.science/paper/GZSIKP3P

@misc{pith2026250605217,
  author       = {Pith},
  title        = {Pith review of: DSG-World: Learning a 3D Gaussian World Model from Dual State Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GZSIKP3P}},
  note         = {Machine review of arXiv:2506.05217}
}

read the original abstract

Building an efficient and physically consistent world model from limited observations is a long standing challenge in vision and robotics. Many existing world modeling pipelines are based on implicit generative models, which are hard to train and often lack 3D or physical consistency. On the other hand, explicit 3D methods built from a single state often require multi-stage processing-such as segmentation, background completion, and inpainting-due to occlusions. To address this, we leverage two perturbed observations of the same scene under different object configurations. These dual states offer complementary visibility, alleviating occlusion issues during state transitions and enabling more stable and complete reconstruction. In this paper, we present DSG-World, a novel end-to-end framework that explicitly constructs a 3D Gaussian World model from Dual State observations. Our approach builds dual segmentation-aware Gaussian fields and enforces bidirectional photometric and semantic consistency. We further introduce a pseudo intermediate state for symmetric alignment and design collaborative co-pruning trategies to refine geometric completeness. DSG-World enables efficient real-to-simulation transfer purely in the explicit Gaussian representation space, supporting high-fidelity rendering and object-level scene manipulation without relying on dense observations or multi-stage pipelines. Extensive experiments demonstrate strong generalization to novel views and scene states, highlighting the effectiveness of our approach for real-world 3D reconstruction and simulation.

Figures

Figures reproduced from arXiv: 2506.05217 by the authors.

**Figure 2.** Overview of our dual-state Gaussian optimization pipeline. Given two complementary scene [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Qualitative comparison of simulated scene reconstruction under different segmentation [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Visual comparison of synthetic (top) and real-world (bottom) scenes simulation across [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 14 canonical work pages

[1]

Navigation world models

Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. arXiv preprint arXiv:2412.03572, 2024

arXiv 2024
[2]

Dream to manipulate: Compositional world models empowering robot imitation learning with imagination

Leonardo Barcellona, Andrii Zadaianchuk, Davide Allegro, Samuele Papa, Stefano Ghidoni, and Efstratios Gavves. Dream to manipulate: Compositional world models empowering robot imitation learning with imagination. arXiv preprint arXiv:2412.14957, 2024

arXiv 2024
[3]

Unifying (machine) vision via counterfactual world modeling

Daniel M Bear, Kevin Feigelis, Honglin Chen, Wanhee Lee, Rahul Venkatesh, Klemen Kotar, Alex Durango, and Daniel LK Yamins. Unifying (machine) vision via counterfactual world modeling. arXiv preprint arXiv:2306.01828, 2023

arXiv 2023
[4]

Blender - a 3d modelling and rendering package

Blender Online Community. Blender - a 3d modelling and rendering package. https://www. blender.org, 2023

work page 2023
[5]

Blenderkit: 3d asset library for blender

BlenderKit. Blenderkit: 3d asset library for blender. https://www.blenderkit.com, 2023. Accessed in 2023

work page 2023
[6]

Mvinpainter: Learning multi-view consistent inpainting to bridge 2d and 3d editing

Chenjie Cao, Chaohui Yu, Fan Wang, Xiangyang Xue, and Yanwei Fu. Mvinpainter: Learning multi-view consistent inpainting to bridge 2d and 3d editing. arXiv preprint arXiv:2408.08000, 2024

arXiv 2024
[7]

Neuma: Neural material adaptor for visual grounding of intrinsic dynamics

Junyi Cao, Shanyan Guan, Yanhao Ge, Wei Li, Xiaokang Yang, and Chao Ma. Neuma: Neural material adaptor for visual grounding of intrinsic dynamics. Advances in Neural Information Processing Systems, 37:65643–65669, 2024

work page 2024
[8]

Ovgaussian: Generalizable 3d gaussian segmentation with open vocabularies

Runnan Chen, Xiangyu Sun, Zhaoqing Wang, Youquan Liu, Jiepeng Wang, Lingdong Kong, Jiankang Deng, Mingming Gong, Liang Pan, Wenping Wang, et al. Ovgaussian: Generalizable 3d gaussian segmentation with open vocabularies. arXiv preprint arXiv:2501.00326, 2024

arXiv 2024

Show all 46 references

[9]

Gaussianeditor: Swift and controllable 3d editing with gaussian splatting

Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang, Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In Proceedings of the IEEE/CVF conference on computer vision and pattern ...

2024
[10]

Tracking anything with decoupled video segmentation

Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon-Young Lee. Tracking anything with decoupled video segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1316–1326, 2023

2023
[11]

Re ˆ 3 sim: Generating high-fidelity simulation data via 3d-photorealistic real-to-sim for robotic manipulation

Xiaoshen Han, Minghuan Liu, Yilun Chen, Junqiu Yu, Xiaoyang Lyu, Yang Tian, Bolun Wang, Weinan Zhang, and Jiangmiao Pang. Re ˆ 3 sim: Generating high-fidelity simulation data via 3d-photorealistic real-to-sim for robotic manipulation. arXiv preprint arXiv:2502.08645, 2025

2025 arXiv
[12]

Pointmap association and piecewise-plane constraint for consistent and compact 3d gaussian segmentation field

Wenhao Hu, Wenhao Chai, Shengyu Hao, Xiaotong Cui, Xuexiang Wen, Jenq-Neng Hwang, and Gaoang Wang. Pointmap association and piecewise-plane constraint for consistent and compact 3d gaussian segmentation field. arXiv preprint arXiv:2502.16303, 2025

2025 arXiv
[13]

Semantic anything in 3d gaussians

Xu Hu, Yuxi Wang, Lue Fan, Junsong Fan, Junran Peng, Zhen Lei, Qing Li, and Zhaoxiang Zhang. Semantic anything in 3d gaussians. arXiv preprint arXiv:2401.17857, 2024

2024 arXiv
[14]

3d gaussian inpainting with depth-guided cross-view consistency

Sheng-Yu Huang, Zi-Ting Chou, and Yu-Chiang Frank Wang. 3d gaussian inpainting with depth-guided cross-view consistency. arXiv preprint arXiv:2502.11801, 2025. 10

2025 arXiv
[15]

Gaussiancut: Interactive segmentation via graph cut for 3d gaussian splatting

Umangi Jain, Ashkan Mirzaei, and Igor Gilitschenski. Gaussiancut: Interactive segmentation via graph cut for 3d gaussian splatting. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

2024
[16]

3d gaussian splatting for real-time radiance field rendering

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1–14, 2023

2023
[17]

Instancegaussian: Appearance-semantic joint gaussian representation for 3d instance- level perception

Haijie Li, Yanmin Wu, Jiarui Meng, Qiankun Gao, Zhiyao Zhang, Ronggang Wang, and Jian Zhang. Instancegaussian: Appearance-semantic joint gaussian representation for 3d instance- level perception. arXiv preprint arXiv:2411.19235, 2024

2024 arXiv
[18]

Robogsim: A real2sim2real robotic gaussian splatting simulator

Xinhai Li, Jialin Li, Ziheng Zhang, Rui Zhang, Fan Jia, Tiancai Wang, Haoqiang Fan, Kuo-Kun Tseng, and Ruiping Wang. Robogsim: A real2sim2real robotic gaussian splatting simulator. arXiv preprint arXiv:2411.11839, 2024

2024 arXiv
[19]

Infusion: Inpainting 3d gaussians via learning depth completion from diffusion prior

Zhiheng Liu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Jie Xiao, Kai Zhu, Nan Xue, Yu Liu, Yujun Shen, and Yang Cao. Infusion: Inpainting 3d gaussians via learning depth completion from diffusion prior. arXiv preprint arXiv:2404.11613, 2024

2024 arXiv
[20]

Robo-gs: A physics consistent spatial-temporal model for robotic arm with hybrid representation

Haozhe Lou, Yurong Liu, Yike Pan, Yiran Geng, Jianteng Chen, Wenlong Ma, Chenglong Li, Lin Wang, Hengzhen Feng, Lu Shi, et al. Robo-gs: A physics consistent spatial-temporal model for robotic arm with hybrid representation. arXiv preprint arXiv:2408.14873, 2024

2024 arXiv
[21]

Gaga: Group any gaussians via 3d-aware memory bank

Weijie Lyu, Xueting Li, Abhijit Kundu, Yi-Hsuan Tsai, and Ming-Hsuan Yang. Gaga: Group any gaussians via 3d-aware memory bank. arXiv preprint arXiv:2404.07977, 2024

2024 arXiv
[22]

Structured world models from human videos

Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. arXiv preprint arXiv:2308.10901, 2023

2023 arXiv
[23]

Learning view-invariant world models for visual robotic manipulation

Jing-Cheng Pang, Nan Tang, Kaiyuan Li, Yuting Tang, Xin-Qiang Cai, Zhen-Yu Zhang, Gang Niu, Masashi Sugiyama, and Yang Yu. Learning view-invariant world models for visual robotic manipulation. In The Thirteenth International Conference on Learning Representations
[24]

Langsplat: 3d language gaussian splatting

Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20051–20060, 2024

2024
[25]

Gls: Geometry-aware 3d language gaussian splatting

Jiaxiong Qiu, Liu Liu, Zhizhong Su, and Tianwei Lin. Gls: Geometry-aware 3d language gaussian splatting. arXiv preprint arXiv:2411.18066, 2024

2024 arXiv
[26]

Splatsim: Zero-shot sim2real transfer of rgb manipulation policies using gaussian splatting

Mohammad Nomaan Qureshi, Sparsh Garg, Francisco Yandun, David Held, George Kantor, and Abhisesh Silwal. Splatsim: Zero-shot sim2real transfer of rgb manipulation policies using gaussian splatting. arXiv preprint arXiv:2409.10161, 2024

2024 arXiv
[27]

Flashsplat: 2d to 3d gaussian splatting segmentation solved optimally

Qiuhong Shen, Xingyi Yang, and Xinchao Wang. Flashsplat: 2d to 3d gaussian splatting segmentation solved optimally. In European Conference on Computer Vision, pages 456–472. Springer, 2024

2024
[28]

Language embedded 3d gaus- sians for open-vocabulary scene understanding

Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaus- sians for open-vocabulary scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5333–5343, 2024

2024
[29]

Resolution-robust large mask inpainting with fourier convolutions

Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winte...

2022
[30]

Nerf in robotics: A survey

Guangming Wang, Lei Pan, Songyou Peng, Shaohui Liu, Chenfeng Xu, Yanzi Miao, Wei Zhan, Masayoshi Tomizuka, Marc Pollefeys, and Hesheng Wang. Nerf in robotics: A survey. arXiv preprint arXiv:2405.01333, 2024

2024 arXiv
[31]

De- coupledgaussian: Object-scene decoupling for physics-based interaction

Miaowei Wang, Yibo Zhang, Rui Ma, Weiwei Xu, Changqing Zou, and Daniel Morris. De- coupledgaussian: Object-scene decoupling for physics-based interaction. arXiv preprint arXiv:2503.05484, 2025. 11

2025 arXiv
[32]

Vid- man: Exploiting implicit dynamics from video diffusion model for effective robot manipulation

Youpeng Wen, Junfan Lin, Yi Zhu, Jianhua Han, Hang Xu, Shen Zhao, and Xiaodan Liang. Vid- man: Exploiting implicit dynamics from video diffusion model for effective robot manipulation. Advances in Neural Information Processing Systems , 37:41051–41075, 2024

2024
[33]

ivideogpt: Interactive videogpts are scalable world models

Jialong Wu, Shaofeng Yin, Ningya Feng, Xu He, Dong Li, Jianye Hao, and Mingsheng Long. ivideogpt: Interactive videogpts are scalable world models. Advances in Neural Information Processing Systems, 37:68082–68119, 2024

2024
[34]

Opengaussian: Towards point-level 3d gaussian-based open vocabulary understanding

Yanmin Wu, Jiarui Meng, Haijie Li, Chenming Wu, Yahao Shi, Xinhua Cheng, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, et al. Opengaussian: Towards point-level 3d gaussian-based open vocabulary understanding. arXiv preprint arXiv:2406.02058, 2024

2024 arXiv
[35]

Physgaussian: Physics-integrated 3d gaussians for generative dynamics

Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4389–4398, 2024

2024
[36]

Learning interactive real-world simulators

Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114, 1(2):6, 2023

2023 arXiv
[37]

Novel demonstration generation with gaussian splatting enables robust one-shot manipulation

Sizhe Yang, Wenye Yu, Jia Zeng, Jun Lv, Kerui Ren, Cewu Lu, Dahua Lin, and Jiangmiao Pang. Novel demonstration generation with gaussian splatting enables robust one-shot manipulation. arXiv preprint arXiv:2504.13175, 2025

2025 arXiv
[38]

Gaussian grouping: Segment and edit anything in 3d scenes

Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. arXiv preprint arXiv:2312.00732, 2023

2023 arXiv
[39]

Real2render2real: Scaling robot data without dynamics simulation or robot hardware

Justin Yu, Letian Fu, Huang Huang, Karim El-Refai, Rares Andrei Ambrus, Richard Cheng, Muhammad Zubair Irshad, and Ken Goldberg. Real2render2real: Scaling robot data without dynamics simulation or robot hardware. arXiv preprint arXiv:2505.09601, 2025

2025 arXiv
[40]

Cob-gs: Clear object boundaries in 3dgs segmentation based on boundary-adaptive gaussian splitting

Jiaxin Zhang, Junjun Jiang, Youyu Chen, Kui Jiang, and Xianming Liu. Cob-gs: Clear object boundaries in 3dgs segmentation based on boundary-adaptive gaussian splitting. arXiv preprint arXiv:2503.19443, 2025

2025 arXiv
[41]

3d-vla: A 3d vision-language-action generative world model

Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, and Chuang Gan. 3d-vla: A 3d vision-language-action generative world model. arXiv preprint arXiv:2403.09631, 2024

2024 arXiv
[42]

Reconstruction and simulation of elastic objects with spring-mass 3d gaussians

Licheng Zhong, Hong-Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. In European Conference on Computer Vision, pages 407–423. Springer, 2024

2024
[43]

Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets

Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. arXiv preprint arXiv:2504.02792, 2025

2025 arXiv
[44]

Rethinking end-to-end 2d to 3d scene segmentation in gaussian splatting

Runsong Zhu, Shi Qiu, Zhengzhe Liu, Ka-Hei Hui, Qianyi Wu, Pheng-Ann Heng, and Chi-Wing Fu. Rethinking end-to-end 2d to 3d scene segmentation in gaussian splatting. arXiv preprint arXiv:2503.14029, 2025

2025 arXiv
[45]

Vr-robo: A real-to-sim-to-real framework for visual robot navigation and locomotion

Shaoting Zhu, Linzhan Mou, Derun Li, Baijun Ye, Runhan Huang, and Hang Zhao. Vr-robo: A real-to-sim-to-real framework for visual robot navigation and locomotion. arXiv preprint arXiv:2502.01536, 2025

2025 arXiv
[46]

3d gaussian splatting in robotics: A survey

Siting Zhu, Guangming Wang, Xin Kong, Dezhi Kong, and Hesheng Wang. 3d gaussian splatting in robotics: A survey. arXiv preprint arXiv:2410.12262, 2024. 12

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Navigation world models

Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. arXiv preprint arXiv:2412.03572, 2024

arXiv 2024

[2] [2]

Dream to manipulate: Compositional world models empowering robot imitation learning with imagination

Leonardo Barcellona, Andrii Zadaianchuk, Davide Allegro, Samuele Papa, Stefano Ghidoni, and Efstratios Gavves. Dream to manipulate: Compositional world models empowering robot imitation learning with imagination. arXiv preprint arXiv:2412.14957, 2024

arXiv 2024

[3] [3]

Unifying (machine) vision via counterfactual world modeling

Daniel M Bear, Kevin Feigelis, Honglin Chen, Wanhee Lee, Rahul Venkatesh, Klemen Kotar, Alex Durango, and Daniel LK Yamins. Unifying (machine) vision via counterfactual world modeling. arXiv preprint arXiv:2306.01828, 2023

arXiv 2023

[4] [4]

Blender - a 3d modelling and rendering package

Blender Online Community. Blender - a 3d modelling and rendering package. https://www. blender.org, 2023

work page 2023

[5] [5]

Blenderkit: 3d asset library for blender

BlenderKit. Blenderkit: 3d asset library for blender. https://www.blenderkit.com, 2023. Accessed in 2023

work page 2023

[6] [6]

Mvinpainter: Learning multi-view consistent inpainting to bridge 2d and 3d editing

Chenjie Cao, Chaohui Yu, Fan Wang, Xiangyang Xue, and Yanwei Fu. Mvinpainter: Learning multi-view consistent inpainting to bridge 2d and 3d editing. arXiv preprint arXiv:2408.08000, 2024

arXiv 2024

[7] [7]

Neuma: Neural material adaptor for visual grounding of intrinsic dynamics

Junyi Cao, Shanyan Guan, Yanhao Ge, Wei Li, Xiaokang Yang, and Chao Ma. Neuma: Neural material adaptor for visual grounding of intrinsic dynamics. Advances in Neural Information Processing Systems, 37:65643–65669, 2024

work page 2024

[8] [8]

Ovgaussian: Generalizable 3d gaussian segmentation with open vocabularies

Runnan Chen, Xiangyu Sun, Zhaoqing Wang, Youquan Liu, Jiepeng Wang, Lingdong Kong, Jiankang Deng, Mingming Gong, Liang Pan, Wenping Wang, et al. Ovgaussian: Generalizable 3d gaussian segmentation with open vocabularies. arXiv preprint arXiv:2501.00326, 2024

arXiv 2024