Pith. sign in

REVIEW 3 major objections 4 minor 84 references

Language-model agents can build physically plausible 4D worlds from text, by coding a physics engine and self-critiquing renders.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 07:10 UTC pith:MI4YW3LE

load-bearing objection A genuinely useful multi-agent system for building 4D simulations from text, but the headline numbers compare two different metrics and the stopping rule leans on an unvalidated self-critic. the 3 major comments →

arxiv 2607.21522 v1 pith:MI4YW3LE submitted 2026-07-23 cs.RO cs.AIcs.CLcs.CV

GS-Agent: Creating 4D Physical Worlds With Generative Simulation

Hongxin Zhang , Chunru Lin , Junyan Li , Zhou Xian , Tsun-Hsuan Wang , Chuang Gan This is my paper
classification cs.RO cs.AIcs.CLcs.CV
keywords 4D world generationphysics enginemulti-agent LLM systemgenerative simulationphysical plausibilitycamera and lighting controlembodied AI datanatural language scene creation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to establish that a multi-agent language-model system can replace manual 4D world-building: given a text prompt, three specialist agents—a manager, an entity agent, and a render agent—write executable code for a physics engine, watch intermediary images and videos, and iterate until the simulation matches the description. The central claim is that this physics-in-the-loop approach produces scenes where liquids, deformables, and rigid bodies interact plausibly, and where camera and lighting can be controlled cinematically, without human parameter tuning. A sympathetic reader would care because the method addresses the physical-plausibility and controllability failures of text-to-video generators, and yields not just pixels but an executable world with ground-truth state. If correct, the practical consequence is that describing a scene in natural language is enough to get back a running simulation and a rendered video that obeys physical laws.

Core claim

GS-Agent is an end-to-end multi-agent framework that decomposes world creation into entity management and rendering configuration. The Manager Agent plans, verifies, and advances the scene; the Entity Agent retrieves or generates 3D assets, tunes material parameters, places objects, and programs motion; the Render Agent sets cameras, lighting, and recording schedules. All agents communicate through a message tool and interact with the physics engine through a structured code-writing interface, so every proposed change is executed deterministically and then inspected through multimodal feedback. Across 24 physics-law scenes and 30 complex scenes, GS-Agent reports the best physical-invariance

What carries the argument

The load-bearing object is the agent-simulation interface: a curated toolbox (AddEntity, UpdateEntity, ConfigureEntitiesControl, ConfigureCameraControl, AdvanceScene, RenderCurrentFrame, and friends) that gives each agent a semantically structured way to write simulation code. It converts natural-language goals into deterministic simulator commands, constrains agents away from low-level API mistakes, and returns multimodal feedback—runtime information, rendered frames, videos, and entity states—that the agents use to verify and refine. The mechanism's real work is the perception-action loop: write code, execute, render, watch, diagnose, and rewrite, with the physics solver supplying guarante

Load-bearing premise

The whole pipeline rests on the language model's ability to watch its own rendered videos and honestly judge whether they match the prompt and where the physics went wrong—yet the paper's failure analysis admits the agent sometimes convinces itself the plan succeeded and sometimes tunes the wrong parameter, so if self-critique is unreliable the end-to-end claim collapses even when the physics engine itself is sound.

What would settle it

Present the Manager Agent with rendered videos that have been silently altered so the motion or framing no longer matches its own plan, and measure whether it flags the discrepancy. If it accepts these as successful at a high rate—or instead keeps adjusting irrelevant parameters until it hits a tool-call limit—then the self-critique loop is not verifying what it claims, and the end-to-end autonomy result is falsified. A quantitative version: compare the agent's acceptance verdicts against human labels on a set of failure-injected renders.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A user can describe a scene in plain language and receive an executable 4D simulation and a rendered video, with no manual material or parameter tuning.
  • The output is more than pixels: metric depth, segmentation masks, surface normals, and particle-level dynamics are available as aligned ground-truth data.
  • Users can interact after generation, asking for slow motion, additional objects, or different velocities, and the agent reconfigures the simulation accordingly.
  • The agent can autonomously spot and fix unforeseen simulation failures, such as a leaking mesh, by editing the simulation code itself.
  • Because dynamics come from a deterministic physics solver, long-horizon spatial and temporal consistency is guaranteed by construction rather than approximated by a generative model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's stated bottleneck is the foundation model's self-critique; a separate, more reliable motion/cinematic judge would likely raise the ceiling more than any change to the physics engine.
  • The same tool-mediated agent loop could be ported to other simulators and task domains (for example, robotic manipulation or autonomous driving) by swapping the interface, but the paper does not test that portability.
  • The ground-truth state-based physical-invariance metric is a genuinely new evaluation opportunity that only executable simulators unlock; future benchmarks could adopt it, though it measures the simulator's physics rather than the language-to-scene mapping directly.
  • The approach's performance is bounded by the reasoning and video-comprehension ability of the language-model backbone; smaller open-weight models already work but degrade, suggesting the framework is a wrapper that inherits backbone strengths and limits.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes GS-Agent, a multi-agent LLM framework that translates natural-language prompts into executable physics-simulation code and rendered videos. A Manager Agent plans and delegates to Entity and Render Agents, which interact with the Genesis physics engine through structured tool interfaces and iteratively refine the scene based on rendered visual feedback. The authors evaluate on the NewtonGen benchmark and a custom 30-scene suite, comparing against Sora2, Wan2.2, SWE-Agent, and a visually augmented SWE-Agent, using state-based PIS, video-based PIS, alignment, aesthetics, and a user study. The main claims are that GS-Agent produces more physically plausible, controllable, and instruction-aligned 4D worlds than the baselines.

Significance. The direction is timely and valuable: combining a physics engine with LLM agents for procedural 4D world generation is a promising alternative to learned video generation, and the paper provides a complete system description, including full prompts and tool interfaces, which supports reproducibility. The qualitative demonstrations of autonomous error recovery, fine-grained user control, and cross-backbone operation are strengths, as is the inclusion of a user study. However, the headline quantitative evidence is weakened by a load-bearing metric-circularity issue and an unvalidated self-critique loop. These are addressable with additional experiments and re-analysis, so the paper merits revision rather than rejection.

major comments (3)
  1. [§4.1, Table 1] The headline State-PIS (0.83) is computed from exact center-of-mass kinematics extracted inside the same Genesis engine that executes the generated code. Since the solver enforces the conserved quantities used in PIS, State-PIS primarily measures whether the code executes and the solver remains stable, not whether the agent produced a world that faithfully matches the text. Moreover, Table 1 lists State-PIS as "-" for Sora2 and Wan2.2, so the "best State-PIS" claim is not a comparison against the main video baselines; it is only a comparison among simulator-based methods that share the same engine. The paper should base cross-family claims on Video-PIS (where all methods are measured through the same pixel-level pipeline) or add an independent physics-plausibility metric applied to rendered videos, and treat State-PIS as a solver-stability diagnostic.
  2. [§5.1, Appendix C.3] The Manager Agent's self-critique is the stopping rule for the entire generation loop: it inspects rendered videos, decides whether the world matches the prompt, and only then calls FinalizeOutcome. Appendix C.3 documents that this critic suffers from context contamination (the agent "convincing itself that the final rendered output successfully executed the plan"), misattribution of failure, and hallucinated APIs, and §5.1 concedes foundation models "remain far from perfect" at judging motion and cinematic expression. The paper reports no quantitative measurement of critic accuracy, no false-acceptance rate, and no ablation comparing the self-critique loop against a blind or external critic. Since the end-to-end autonomy claim depends on this loop, the reported scores are conditional on an unvalidated component. Please add an experiment that removes or blinds the critic and report agree
  3. [Tables 1, 3, 4] The Alignment Score for GS-Agent is 32.2 in Table 1 but 29.9 in both Table 3 and Table 4. The captions of Tables 3 and 4 do not specify which evaluation subset is used, so it is unclear whether the backbone and ablation comparisons are run on the same 30 scenes as Table 1. As written, the reader cannot tell whether the alignment drop in the ablations is an effect of the ablated component or just a different evaluation set. Please state the subset used in each table and correct the inconsistency.
minor comments (4)
  1. [§C.2] The user study reports 15 participants and 270 effective responses. With five methods and four criteria, the relationship between participants, scene comparisons, and responses is unclear; please report the number of prompt-scene comparisons per participant and, ideally, inter-rater agreement.
  2. [§2.3] Typo: "code generatoin" should be "code generation".
  3. [References] Reference [3] is cited as "G. Authors" with a project-style title; please provide a proper citation with the full author list or a stable technical report identifier.
  4. [§4.1] The Alignment Score uses Perception Encoder feature similarity, but the text-video aggregation procedure is not described. A brief explanation of how prompt-video similarity is computed and averaged would help reproducibility.

Circularity Check

1 steps flagged

State-PIS is partially circular: physical plausibility is read out from the same simulator that generates the world; the other metrics are external.

specific steps
  1. self definitional [Section 4.1 (Metrics), Table 1]
    "We directly extract exact 3D center-of-mass kinematics from the physics engine at every timestep to compute State-PIS. This ground-truth measurement is inherently inaccessible to standard text-to-video models that only output pixel data."

    The headline physical-plausibility metric is defined from the same Genesis simulator that produces GS-Agent's outputs. The solver already enforces the conserved/invariant quantities (e.g., constant gravitational acceleration during free flight), so any executable Genesis scene inherits high State-PIS by construction. The agent's contribution is setting up the scene, not guaranteeing the physical law, and the metric is asymmetric: text-to-video baselines are evaluated by pixel-based Video-PIS while GS-Agent is granted exact internal simulator states. This makes State-PIS a partially self-referential evaluation rather than independent confirmation of physical plausibility.

full rationale

The central claim is that GS-Agent turns natural language into physically plausible 4D worlds via a multi-agent loop over a physics engine. The derivation is not a fitted-parameter or equation-identity argument, and there is no load-bearing self-citation chain: the engine citation [3] is used as a tool, not as a uniqueness theorem. The partially circular element is evaluative: State-PIS is extracted from the same Genesis states that the method itself produces, so the top quantitative physical-plausibility result partly reflects solver guarantees rather than agent competence. This is not total circularity, because the ablations show monolithic agents receive much lower State-PIS, and the alignment score and 270-response user study are external to the engine. The self-critique weakness is explicitly documented in Section 5.1 and Appendix C.3 (context contamination, misattribution, hallucinated APIs); that is a correctness risk that undermines the autonomy claim but is not itself a circular derivation. Overall, the main quantitative physical-plausibility evidence is partially by construction, but the central system claim still has independent content, warranting a score of 5 rather than 8-10.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

No numerical free parameters were fitted in the system; instead the dependencies are hand-authored prompts/tool schemas and external model/engine quality assumptions. The central result inherits physical correctness from Genesis, so the key assumptions are about engine fidelity and LLM self-critique reliability.

axioms (5)
  • domain assumption Genesis physics engine's rigid/MPM/SPH solvers produce physically plausible dynamics at the fidelity claimed.
    The entire physical-plausibility guarantee rests on Genesis (ref [3]); the paper does not validate solver outputs against real-world measurements.
  • domain assumption The LLM backbone can write correct Genesis tool code and can accurately self-critique rendered frames/videos.
    Central to end-to-end automation; Appendix C.3 itself documents context contamination, misattribution, and hallucinated APIs, so this assumption is explicitly fragile.
  • domain assumption Perception Encoder similarity (Alignment Score) and VBench Aesthetic are valid automatic proxies for instruction fidelity and visual quality.
    Used in Tables 1/3/4 without task-specific validation; the user study is the only human check but covers only a subset of conditions.
  • domain assumption State-PIS values extracted from engine state are an acceptable evaluation of the agent's output rather than of the engine.
    The paper computes PIS from exact engine center-of-mass trajectories, which are physically invariant by construction; this measures the engine/setup more than the language-to-world mapping.
  • domain assumption 3D asset libraries (BlenderKit, PolyHaven) and Meshy text-to-3D generation contain sufficient assets for the tested prompts.
    Entity Agent's curation pipeline depends on these; fallback to primitives limits visual fidelity.

pith-pipeline@v1.3.0-alltime-deepseek · 19749 in / 12694 out tokens · 125025 ms · 2026-08-01T07:10:08.333980+00:00 · methodology

0 comments
read the original abstract

Creating dynamic and physically realistic 4D worlds from natural language descriptions is both fascinating and challenging. Traditional computer graphics methods rely on manual creation, requiring extensive human effort to fine-tune materials, motions, and visual fidelity. Recent advances in generative foundation models have sparked interest in learning to generate such 4D worlds from large-scale data; however, existing methods still struggle to ensure physical plausibility and controllability. In this work, we take a different path by leveraging foundation models to construct an agentic system that emulates how humans traditionally create 4D worlds, yet automates the entire process. We present GS-Agent, an end-to-end multi-agent framework that integrates physics engines in the loop to generate realistic, dynamic, and controllable 4D physical worlds from natural language. Inspired by how humans build 4D worlds, GS-Agent decomposes the task into entity management, covering 3D asset curation, material tuning, placement, and motion control, and rendering configuration, including camera and lighting manipulation. Multiple agents with distinct expertise interact with the physics engine via code, seek multimodal feedback, and collaborate to iteratively construct 4D worlds that align with the given descriptions. Experimental results show that GS-Agent effectively converts natural language into diverse and physically plausible 4D worlds exhibiting rich interactions among liquids, deformable objects, and rigid bodies, while achieving cinematic camera and lighting control. We envision GS-Agent as a foundation for a new paradigm in 4D world generation, empowering creative content creation and physical AI. Project page at https://umass-embodied-agi.github.io/gs-agent/

Figures

Figures reproduced from arXiv: 2607.21522 by Chuang Gan, Chunru Lin, Hongxin Zhang, Junyan Li, Tsun-Hsuan Wang, Zhou Xian.

Figure 1
Figure 1. Figure 1: GS-Agent creates 4D worlds from natural language through generative simulations, producing physically plausible interactions among liquids, deformable objects, and rigid bodies, together with cinematic camera and lighting controls. GS-Agent generates more than pixels; data modalities including fine-grained dynamics, depth and surface normals enable richer applications. Abstract Creating dynamic and physica… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of GS-Agent. The Manager Agent interprets the natural language descrip￾tion, decomposes it into actionable sub-tasks, configures global simulation options, controls scene advancement, and delegates tasks to the appropriate agents. The Entity Agent curates assets for morphology creation, places entities according to spatial constraints, tunes material parameters for physical plausibility, and contr… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparisons between GS-Agent and the baseline methods. More cases are presented in the Appendix C. 4 Experiments 4.1 Experimental Setup To systematically evaluate GS-Agent’s ability to construct 4D physical worlds, we conduct exper￾iments across two evaluation suites. The first comprises 24 scenes from the NewtonGen bench￾mark [73], specifically designed to test 12 distinct physical laws (e.g.,… view at source ↗
Figure 4
Figure 4. Figure 4: Autonomous error recovery. GS-Agent automatically detects a leaking 3D asset and applies rigid material patches to the corners without human intervention. A white ball hitting a blue ball on the floor. User Good! Can GS-Agent you make a video of slow motion so that I could have a closer look at the motion? Add more billiard balls to the scene. Larger velocity to make a break. 00:00.000 00:00.333 00:01.000 … view at source ↗
Figure 6
Figure 6. Figure 6: Ablation of framework components. Removing the multi-agent design leads to in￾complete scenes, while removing the specialized agent-simulator interface causes the model to struggle with generating stable simulation code. water leakage, GS-Agent recognized the physics failure, proposed a programmatic fix by applying rigid material patches to the geometry’s corners, and automatically verified the corrected o… view at source ↗
Figure 7
Figure 7. Figure 7: More qualitative comparisons between GS-Agent (Ours) and the baseline methods. the received results (rendered images/videos or numerical states or messages from other agents) in-context, and continue until the agent is satisfied with the result or hits the tool call limits. C Additional Experiment Details C.1 More Qualitative Comparisons We show more qualitative comparisons between our method and other bas… view at source ↗
Figure 8
Figure 8. Figure 8: User study interface. C.2 User Study To provide more reliable comparisons, we conducted a user study with 15 subjects. Subjects are instructed to rate five randomly ordered videos generated by our method and baselines in a 5-point Likert Scale, for the aspects of physical plausibility, camera controllability, Content Alignment, and aesthetics. A total of 270 effective responses is collected. The user study… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

84 extracted references · 27 linked inside Pith

  1. [1]

    Agashe, K

    S. Agashe, K. Wong, V . Tu, J. Yang, A. Li, and X. E. Wang. Agent s2: A compositional generalist-specialist framework for computer use agents.arXiv preprint arXiv:2504.00906, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    M. Ahn, A. Brohan, N. Brown, Y . Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as i can, not as i say: Grounding language in robotic affordances.arXiv preprint arXiv:2204.01691, 2022

    Pith/arXiv arXiv 2022

  3. [3]

    G. Authors. Genesis: A generative and universal physics engine for robotics and beyond, December 2024

    2024

  4. [4]

    Bahmani, I

    S. Bahmani, I. Skorokhodov, V . Rong, G. Wetzstein, L. Guibas, P. Wonka, S. Tulyakov, J. J. Park, A. Tagliasacchi, and D. B. Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling.IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  5. [5]

    Blattmann, T

    A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y . Levi, Z. English, V . V oleti, A. Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv preprint arXiv:2311.15127, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Boeing and T

    A. Boeing and T. Br¨aunl. Evaluation of real-time physics simulation systems. InProceedings of the 5th international conference on Computer graphics and interactive techniques in Australia and Southeast Asia, pages 281–288, 2007

    2007

  7. [7]

    Bolya, P.-Y

    D. Bolya, P.-Y . Huang, P. Sun, J. H. Cho, A. Madotto, C. Wei, T. Ma, J. Zhi, J. Rajasegaran, H. Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network.arXiv preprint arXiv:2504.13181, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Bruce, M

    J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y . Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, et al. Genie: Generative interactive environments. InForty-first International Conference on Machine Learning, 2024

    2024

  9. [9]

    Chang, W

    A. Chang, W. Monroe, M. Savva, C. Potts, and C. D. Manning. Text to 3d scene generation with rich lexical grounding.arXiv preprint arXiv:1505.06289, 2015

    Pith/arXiv arXiv 2015

  10. [10]

    Chang, M

    A. Chang, M. Savva, and C. D. Manning. Learning spatial knowledge for text to 3d scene generation. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 2028–2038, 2014

    2014

  11. [11]

    A. H. Chen, J. Hsu, Z. Liu, M. Macklin, Y . Yang, and C. Yuksel. Offset geometric contact. ACM Transactions on Graphics (TOG), 44(4):1–21, 2025

    2025

  12. [12]

    B. Chen, H. Jiang, S. Liu, S. Gupta, Y . Li, H. Zhao, and S. Wang. Physgen3d: Crafting a miniature interactive world from a single image. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 6178–6189, 2025

    2025

  13. [13]

    H. Chen, Y . Zhang, X. Cun, M. Xia, X. Wang, C. Weng, and Y . Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7310–7320, 2024

    2024

  14. [14]

    Chung, S

    J. Chung, S. Lee, H. Nam, J. Lee, and K. M. Lee. Luciddreamer: Domain-free generation of 3d gaussian splatting scenes.arXiv preprint arXiv:2311.13384, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Y . Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch. Improving factuality and reasoning in language models through multiagent debate. InForty-first International Conference on Machine Learning, 2023

    2023

  16. [16]

    Y . Du, M. Yang, P. Florence, F. Xia, A. Wahid, B. Ichter, P. Sermanet, T. Yu, P. Abbeel, J. B. Tenenbaum, et al. Video language planning.arXiv preprint arXiv:2310.10625, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Gillman, C

    N. Gillman, C. Herrmann, M. Freeman, D. Aggarwal, E. Luo, D. Sun, and C. Sun. Force prompting: Video generation models can learn and generalize physics-based control signals. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. 10

  18. [18]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    I. Gur, H. Furuta, A. Huang, M. Safdari, Y . Matsuo, D. Eck, and A. Faust. A real-world webagent with planning, long context understanding, and program synthesis.arXiv preprint arXiv:2307.12856, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    H. He, Y . Xu, Y . Guo, G. Wetzstein, B. Dai, H. Li, and C. Yang. Cameractrl: Enabling camera control for text-to-video generation.arXiv preprint arXiv:2404.02101, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, et al. Imagen video: High definition video generation with diffusion models.arXiv preprint arXiv:2210.02303, 2022

    Pith/arXiv arXiv 2022

  22. [22]

    J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

    2020

  23. [23]

    W. Hong, W. Wang, Q. Lv, J. Xu, W. Yu, J. Ji, Y . Wang, Z. Wang, Y . Dong, M. Ding, et al. Cogagent: A visual language model for gui agents. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024

    2024

  24. [24]

    M. Hu, Y . Zhou, W. Fan, Y . Nie, B. Xia, T. Sun, Z. Ye, Z. Jin, Y . Li, Q. Chen, et al. Owl: Optimized workforce learning for general multi-agent assistance in real-world task automation. arXiv preprint arXiv:2505.23885, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Z. Hu, A. Iscen, A. Jain, T. Kipf, Y . Yue, D. A. Ross, C. Schmid, and A. Fathi. Scenecraft: An llm agent for synthesizing 3d scenes as blender code. InForty-first International Conference on Machine Learning, 2024

    2024

  26. [26]

    Huang, C

    W. Huang, C. Wang, R. Zhang, Y . Li, J. Wu, and L. Fei-Fei. V oxposer: Composable 3d value maps for robotic manipulation with language models. InConference on Robot Learning, pages 540–562. PMLR, 2023

    2023

  27. [27]

    Huang, Y

    Z. Huang, Y . He, J. Yu, F. Zhang, C. Si, Y . Jiang, Y . Zhang, T. Wu, Q. Jin, N. Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21807–21818, 2024

    2024

  28. [28]

    Jiang, C

    C. Jiang, C. Schroeder, J. Teran, A. Stomakhin, and A. Selle. The material point method for simulating continuum materials. InAcm siggraph 2016 courses, pages 1–52. 2016

    2016

  29. [29]

    B. Kang, Y . Yue, R. Lu, Z. Lin, Y . Zhao, K. Wang, G. Huang, and J. Feng. How far is video generation from world model: A physical law perspective.arXiv preprint arXiv:2411.02385, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Kondratyuk, L

    D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, G. Schindler, R. Hornung, V . Birodkar, J. Yan, M.-C. Chiu, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem. Camel: Communicative agents for” mind” exploration of large language model society.Advances in Neural Information Processing Systems, 36:51991–52008, 2023

    2023

  32. [32]

    Li, H.-X

    Z. Li, H.-X. Yu, W. Liu, Y . Yang, C. Herrmann, G. Wetzstein, and J. Wu. Wonderplay: Dynamic 3d scene generation from a single image and actions.arXiv preprint arXiv:2505.18151, 2025

    arXiv 2025

  33. [33]

    Z. Lin, S. Cen, D. Jiang, J. Karhade, H. Wang, C. Mitra, T. Ling, Y . Huang, S. Liu, M. Chen, et al. Towards understanding camera motions in any video.arXiv preprint arXiv:2504.15376, 2025

    Pith/arXiv arXiv 2025

  34. [34]

    H. Liu, J. He, Y . Jin, D. Zheng, Y . Dong, F. Zhang, Z. Huang, Y . He, Y . Li, W. Chen, et al. Shotbench: Expert-level cinematic understanding in vision-language models.arXiv preprint arXiv:2506.21356, 2025. 11

    arXiv 2025

  35. [35]

    H. Liu, C. Li, Z. Li, Y . Wu, W. Li, Z. Yang, Z. Zhang, Y . Lin, S. Han, and B. Y . Feng. Ir3d- bench: Evaluating vision-language model scene understanding as agentic inverse rendering. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track

  36. [36]

    H. Liu, C. Li, Q. Wu, and Y . J. Lee. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Liu, X.-W

    N. Liu, X.-W. Hua, R. Zhang, and L. Wang. Physical material editing with structure embedding for realistic deformable objects. InProceedings of the ACM Symposium on Interactive 3D Graphics and Games (I3D), 2012

    2012

  38. [38]

    S. Liu, Z. Ren, S. Gupta, and S. Wang. Physgen: Rigid-body physics-grounded image-to-video generation. InEuropean Conference on Computer Vision, pages 360–378. Springer, 2024

    2024

  39. [39]

    Z. Liu, Y . Zhang, P. Li, Y . Liu, and D. Yang. A dynamic LLM-powered agent network for task-oriented agent collaboration. InFirst Conference on Language Modeling, 2024

    2024

  40. [40]

    S. Lu, G. Chen, N. A. Dinh, I. Lang, A. Holtzman, and R. Hanocka. Ll3m: Large language 3d modelers.arXiv preprint arXiv:2508.08228, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    J. Lv, Y . Huang, M. Yan, J. Huang, J. Liu, Y . Liu, Y . Wen, X. Chen, and S. Chen. Gpt4motion: Scripting physical motions in text-to-video generation via blender-oriented gpt planning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1430–1440, 2024

    2024

  42. [42]

    R. Ma, A. G. Patil, M. Fisher, M. Li, S. Pirk, B.-S. Hua, S.-K. Yeung, X. Tong, L. Guibas, and H. Zhang. Language-driven synthesis of 3d scenes from scene databases.ACM Transactions on Graphics (TOG), 37(6):1–16, 2018

    2018

  43. [43]

    F. Meng, J. Liao, X. Tan, W. Shao, Q. Lu, K. Zhang, Y . Cheng, D. Li, Y . Qiao, and P. Luo. To- wards world simulator: Crafting physical commonsense-based benchmark for video generation. arXiv preprint arXiv:2410.05363, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Meshy: Generative ai for 3d content creation, 2024

    Meshy. Meshy: Generative ai for 3d content creation, 2024. Accessed: 2026-01-29

    2024

  45. [45]

    Motamed, M

    S. Motamed, M. Chen, L. Van Gool, and I. Laina. Travl: A recipe for making video-language models better judges of physics implausibility.arXiv preprint arXiv:2510.07550, 2025

    arXiv 2025

  46. [46]

    M¨uller, D

    M. M¨uller, D. Charypar, and M. Gross. Particle-based fluid simulation for interactive applica- tions. InProceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, pages 154–159, 2003

    2003

  47. [47]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023

    2023

  48. [48]

    Sora 2, 2025

    OpenAI. Sora 2, 2025. Video and audio generation model; released 30 Sep 2025

    2025

  49. [49]

    Parker-Holder and S

    J. Parker-Holder and S. Fruchter. Genie 3: A new frontier for world models. Google DeepMind Blog, 2025

    2025

  50. [50]

    Peebles and S

    W. Peebles and S. Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF international conference on computer vision, pages 4195–4205, 2023

    2023

  51. [51]

    J. Qiu, X. Qi, T. Zhang, X. Juan, J. Guo, Y . Lu, Y . Wang, Z. Yao, Q. Ren, X. Jiang, et al. Alita: Generalist agent enabling scalable agentic reasoning with minimal predefinition and maximal self-evolution.arXiv preprint arXiv:2505.20286, 2025

    Pith/arXiv arXiv 2025

  52. [52]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  53. [53]

    Shinn, F

    N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024. 12

    2024

  54. [54]

    Sohl-Dickstein, E

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. InInternational conference on machine learning, pages 2256–2265. pmlr, 2015

    2015

  55. [55]

    Sumers, S

    T. Sumers, S. Yao, K. Narasimhan, and T. L. Griffiths. Cognitive architectures for language agents.arXiv preprint arXiv:2309.02427, 2023

    Pith/arXiv arXiv 2023

  56. [56]

    C. Sun, J. Han, W. Deng, X. Wang, Z. Qin, and S. Gould. 3d-gpt: Procedural 3d modeling with large language models. In2025 International Conference on 3D Vision (3DV), pages 1253–1263. IEEE, 2025

    2025

  57. [57]

    T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, J. Wang, J. Zhang, J. Zhou, J. Wang, J. Chen, K. Zhu, K. Zhao, K. Yan, L. Huang, M. Feng, N. Zhang, P. Li, P. Wu, R. Chu, R. Feng, S. Zhang, S. Sun, T. Fang, T. Wang, T. Gui, T. Weng, T. Shen, W. Lin, W. Wang, W. Wang, W. Zhou, W. Wang, W. Shen, W. Yu, X. Shi, X....

    Pith/arXiv arXiv 2025

  58. [58]

    C. Wang, C. Chen, Y . Huang, Z. Dou, Y . Liu, J. Gu, and L. Liu. Physctrl: Generative physics for controllable and physics-grounded video generation. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems

  59. [59]

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y . Lin, et al. A survey on large language model based autonomous agents.arXiv preprint arXiv:2308.11432, 2023

    Pith/arXiv arXiv 2023

  60. [60]

    Y . Wang, X. Qiu, J. Liu, Z. Chen, J. Cai, Y . Wang, T.-H. Wang, Z. Xian, and C. Gan. Architect: Generating vivid and interactive 3d scenes with hierarchical 2d inpainting.Advances in Neural Information Processing Systems, 37:67575–67603, 2024

    2024

  61. [61]

    Z. Wang, D. Li, Y . Wu, T. He, J. Bian, and R. Jiang. Diffusion models in 3d vision: A survey. arXiv preprint arXiv:2410.04738, 2024

    Pith/arXiv arXiv 2024

  62. [62]

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversations. InFirst Conference on Language Modeling, 2024

    2024

  63. [63]

    Z. Xi, W. Chen, X. Guo, W. He, Y . Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, et al. The rise and potential of large language model based agents: A survey.arXiv preprint arXiv:2309.07864, 2023

    Pith/arXiv arXiv 2023

  64. [64]

    Xiang, G

    J. Xiang, G. Liu, Y . Gu, Q. Gao, Y . Ning, Y . Zha, Z. Feng, T. Tao, S. Hao, Y . Shi, et al. Pandora: Towards general world model with natural language actions and video states.arXiv preprint arXiv:2406.09455, 2024

    Pith/arXiv arXiv 2024

  65. [65]

    T. Xie, Z. Zong, Y . Qiu, X. Li, Y . Feng, Y . Yang, and C. Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4389–4398, June 2024

    2024

  66. [66]

    D. Xu, H. Liang, N. P. Bhatt, H. Hu, H. Liang, K. N. Plataniotis, and Z. Wang. Comp4d: Llm-guided compositional 4d scene generation.arXiv preprint arXiv:2403.16993, 2024

    Pith/arXiv arXiv 2024

  67. [67]

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. Swe- agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    2024

  68. [68]

    Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y . Yang, W. Hong, X. Zhang, G. Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. InThe Thirteenth International Conference on Learning Representations, 2024

    2024

  69. [69]

    S. Yin, J. Ge, Z. Z. Wang, C. Wang, X. Li, M. J. Black, T. Darrell, A. Kanazawa, and H. Feng. Vision-as-inverse-graphics agent via interleaved multimodal reasoning.arXiv preprint arXiv:2601.11109, 2026. 13

    Pith/arXiv arXiv 2026

  70. [70]

    H.-X. Yu, H. Duan, C. Herrmann, W. T. Freeman, and J. Wu. Wonderworld: Interactive 3d scene generation from a single image. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 5916–5926, 2025

    2025

  71. [71]

    H.-X. Yu, H. Duan, J. Hur, K. Sargent, M. Rubinstein, W. T. Freeman, F. Cole, D. Sun, N. Snavely, J. Wu, et al. Wonderjourney: Going from anywhere to everywhere. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6658–6667, 2024

    2024

  72. [72]

    L. Yu, Y . Cheng, K. Sohn, J. Lezama, H. Zhang, H. Chang, A. G. Hauptmann, M.-H. Yang, Y . Hao, I. Essa, et al. Magvit: Masked generative video transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10459–10469, 2023

    2023

  73. [73]

    Y . Yuan, X. Wang, T. Wickremasinghe, Z. Nadir, B. Ma, and S. H. Chan. Newtongen: Physics- consistent and controllable text-to-video generation via neural newtonian dynamics.arXiv preprint arXiv:2509.21309, 2025

    arXiv 2025

  74. [74]

    Zhang, W

    H. Zhang, W. Du, J. Shan, Q. Zhou, Y . Du, J. B. Tenenbaum, T. Shu, and C. Gan. Building co- operative embodied agents modularly with large language models. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  75. [75]

    Zhang, P

    Q. Zhang, P. Jing, H.-X. Yu, F. Ding, F. Nie, W. Wang, Y . Du, J. Zou, J. Wu, and B. Shuai. Physion-eval: Evaluating physical realism in generated video via human reasoning.arXiv preprint arXiv:2603.19607, 2026

    arXiv 2026

  76. [76]

    Zhang, H.-X

    T. Zhang, H.-X. Yu, R. Wu, B. Y . Feng, C. Zheng, N. Snavely, J. Wu, and W. T. Freeman. PhysDreamer: Physics-based interaction with 3d objects via video generation. InEuropean Conference on Computer Vision. Springer, 2024

    2024

  77. [77]

    Zhang, J

    Z. Zhang, J. Liao, M. Li, Z. Dai, B. Qiu, S. Zhu, L. Qin, and W. Wang. Tora: Trajectory-oriented diffusion transformer for video generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2063–2073, June 2025

    2063

  78. [78]

    A tracking shot of a green ball rolling on the floor while another yellow cylinder sits on the floor

    W. Zhao, J. P. Queralta, and T. Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In2020 IEEE symposium series on computational intelligence (SSCI), pages 737–744. IEEE, 2020. 14 A Broader Impact The development ofGS-Agentpresents significant positive potential while raising important ethical and societal consideratio...

    2020

  79. [79]

    This ensures entities do not fall through the world due to gravity

    Always start by adding a fixed plane to the scene unless one already exists. This ensures entities do not fall through the world due to gravity

  80. [80]

    If anything is unclear or missing, use ‘send_message‘ to request clarification from the Manager Agent

    Carefully interpret the instruction from the Manager Agent. If anything is unclear or missing, use ‘send_message‘ to request clarification from the Manager Agent

Showing first 80 references.