Pith. sign in

REVIEW 4 major objections 6 minor 64 references

3D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worlds

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a vision-language model trained on its own best-scoring action sequences can generate simulation-ready 3D environments from text prompts, and that synthetic data from those environments can replace human-crafted 3D…

desk verdict A substantial 3D scene-generation system that deserves a serious referee, but the prompt-alignment claim needs an external, non-CLIP validation before it can be taken at face value. read the letter →

arxiv 2507.06484 v2 pith:CYK5TBBT submitted 2025-07-09 cs.GR cs.CV

Fan-Yun Sun , Shengguang Wu , Christian Jacobsen , Thomas Yim , Haoming Zou , Alex Zook , Shangru Li , Yu-Hsin Chou
show 8 more authors
Ethem Can Xunlei Wu Clemens Eppner Valts Blukis Jonathan Tremblay Jiajun Wu Stan Birchfield Nick Haber
This is my paper
classification cs.GRcs.CV
keywords 3Dscenegenerationvision-language-actionmodelself-improvementfine-tuningCLIPscoresyntheticdatapretrainingscene-levelpolicyasset-levelpanoramicenvironment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that 3D environment building can be automated by treating it as a sequential decision-making problem solved by a vision-language model (VLM) that edits layout, materials, lighting, and assets in iterative rounds. Its central proposal is a self-improvement fine-tuning loop: the model samples multiple actions, keeps the action sequences whose rendered scenes score highest under CLIP alignment with the text prompt, and fine-tunes itself on those sequences. The authors show this loop makes generated scenes more prompt-aligned than prior layout-only methods, and that rendering millions of images from generated scenes can pretrain a vision encoder that outperforms one pretrained on the human-crafted Hypersim dataset at equal label count and approaches a model pretrained on roughly 400 times more real labels. If true, the contribution is a scalable source of 3D training data that does not require human artists.

What carries the argument

The load-bearing mechanism is the CLIP-scored self-improvement loop. At each step of the scene-level policy, the VLM receives multi-view renderings (top-down with coordinate marks, panoramic, side view with asset marks) and outputs action code in a Scene DSL, a structured Python-like representation of scene elements with placement, material, and lighting descriptors; multiple candidate actions are executed, the environment is re-rendered, and the sequences with the highest CLIP similarity to the prompt are retained as fine-tuning targets. An in-context library of past action snippets that raised CLIP alignment by at least 10% adds diversity and effective examples during data generation. A second, pixel-precise VLM handles small-asset placement on receptacle surfaces, using mesh-based surface detection and collision checks before placement.

What would settle it

Show humans a set of scenes generated by 3D-Generalist alongside scenes from a baseline and ask which better match the prompt; if human rankings correlate poorly with the CLIP scores that drove training, or if the fine-tuned model does not beat the base model in human ratings, the central claim fails its test.

Watch

Extended reading notes

Core claim

On the paper's own terms, 3D-Generalist establishes that a VLM can act as a generative policy for complete 3D rooms: given multi-view renderings of the current scene and a text prompt, it writes code in a Scene DSL that adds and repositions assets, swaps materials, and adjusts lighting. The self-improvement step is the crux: candidate action sequences are rolled out, scored by CLIP similarity between the resulting renderings and the prompt, and the top-scoring ones become supervised fine-tuning data for the next policy round. With three such rounds, the policy improves over its base model and over prior LLM/VLM methods on layout-based metrics, and the fine-tuning transfers to lower visual hallucination rates on general-domain benchmarks. Finally, the paper demonstrates that the pipeline scales: 12 million labels rendered from generated scenes let a Florence 2 encoder with a DaViT-B backbone reach 0.776 ImageNet top-1 accuracy after fine-tuning, versus 0.727 for the same architecture pretrained on Hypersim's 861K labels and 0.786 for the real-data model trained on 5 billion labels.

Load-bearing premise

The whole framework depends on CLIP score being a faithful measure of how well a rendered 3D scene matches the text prompt; if CLIP's notion of alignment diverges from human judgment, the self-improvement loop is optimizing the wrong target and the evaluations that use CLIP do not measure the claim.

Editorial extensions

If this is right

  • Natural-language 3D world building becomes a scaling-with-compute process: more compute buys more self-improvement rounds and more training data, rather than more manual annotation.
  • Synthetic data for vision pretraining can be produced on demand at tens of millions of labels, with a path to close the gap to real-data pretraining.
  • The self-improvement recipe transfers beyond 3D: fine-tuning on 3D action code reduces object hallucination rates on general-domain VLM benchmarks such as Object HalBench and AMBER.
  • The framework extends to a fully generative graphics pipeline by swapping repositories for text-to-3D, image-to-PBR, and 3D-asset generators, and to embodied AI by generating simulation environments for reinforcement learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because CLIP score serves as both the training reward and the headline evaluation metric for prompt alignment, the reported gains may not generalize to human judgment; a human-rated evaluation or a second alignment metric would test this directly.
  • The success of the loop suggests a general bootstrap pattern—use a frozen reward model to filter a model's own rollouts into training data—that could apply to other structured generation tasks such as robot manipulation plans or procedural level design.
  • The paper stops after three self-improvement rounds; it is untested whether additional rounds continue to improve prompt alignment or plateau, which would determine whether the loop is a genuinely scalable strategy.
  • If the 12M-label synthetic result is confirmed on more downstream tasks, it implies a data-efficiency ratio of roughly three orders of magnitude less data than real pretraining, which would make synthetic generation the default first step for open-vocabulary vision pretraining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes 3D-Generalist, a framework that recasts text-driven 3D environment generation as a sequential decision-making problem solved by vision-language action policies. Three modules are presented: Panoramic Environment Generation (diffusion-guided inverse graphics for room layout and fixtures), a Scene-Level Policy (a GPT-4o-based VLM that iteratively outputs code to refine assets, layout, materials, and lighting, improved by a CLIP-scored self-improvement fine-tuning loop), and an Asset-Level Policy (a Molmo-based VLM that places small objects on receptacle meshes with physical-plausibility checks). Experiments report physical plausibility and semantic coherence against LayoutGPT, Holodeck, and LayoutVLM (Table 1), a CLIP-score ablation of the scene-level self-improvement rounds (Table 2), transfer of the fine-tuned VLM to visual hallucination benchmarks (Table 3), and ImageNet-1K top-1 accuracy after pretraining Florence-2-style encoders on rendered synthetic scenes versus Hypersim and a real-data baseline (Table 4). The central claim is that self-improvement yields more prompt-aligned 3D environments and that the resulting synthetic data can replace or approach human-crafted and real-world data for vision pretraining.

Significance. If the claims are established, the paper would make an important step toward scalable, simulation-ready 3D content generation and provide evidence that VLM-generated synthetic environments can serve as training data for vision foundation models. The pipeline is modular and the downstream ImageNet experiment is a useful controlled comparison at matched label counts (861,080 labels), with the larger 12M-label run showing a nontrivial gain (0.776 vs. 0.727 for Hypersim and approaching 0.786 for the 5B-label real-data checkpoint). The hallucination-transfer result in Table 3 is also a plus, as it connects the 3D training objective to general-domain visual grounding. However, the principal evidence for the paper's headline claim of 'more prompt-aligned 3D environments' rests on a single CLIP-based score that is also used as the training selection criterion, so the central result is not independently established. The physics comparison in Table 1 is also structurally favorable to the proposed method, and the semantic judge is the same model family as the policy. These issues are correctable with additional evaluation, but they are load-bearing for the abstract's central claim.

major comments (4)
  1. [Sec. 3.2 and Table 2] The reported prompt-alignment improvement is circular. In Sec. 3.2, candidate action sequences are retained for supervised fine-tuning when they 'yield the highest CLIP scores between the environment renderings and the input prompt,' and the in-context library is curated from snippets with at least a 10% CLIP improvement. Table 2 then evaluates the resulting policies with the same CLIP score. The trend from 0.242 (no fine-tuning) to 0.254 (3 rounds, 3 actions) and 0.275 (3 rounds, 100 actions) therefore shows optimization of the training objective rather than independent evidence of prompt alignment. An external evaluation, such as human ratings or a non-CLIP automatic metric, is needed to support the central claim that self-improvement produces more prompt-aligned environments.
  2. [Sec. 4.1, Table 1] The physics comparison is structurally favorable to 3D-Generalist. The table caption notes that 'perfect physical plausibility can be achieved as the model can self-correct asset placement over rounds of iteration,' and Sec. 3.3 states that Asset-Level Policy only executes a placement after mesh collision checks, while the evaluation protocol says 'all assets are enforced to be placed, with remaining assets randomly placed if a method fails.' If baselines are not given the same iterative, collision-checked placement opportunity, the large gaps in Collision-Free score (99.0 vs. 76.8-83.8) and In-Boundary score (98.0 vs. 8.1-94.9) are partly by construction and do not by themselves establish that 3D-Generalist produces more physically plausible scenes.
  3. [Sec. 4.1, semantic coherence evaluation] The semantic coherence scores in Table 1 use GPT-4o as the judge, which is the same base model family as the Scene-Level Policy that is being evaluated. This provides only partial independence, and no evidence is given that GPT-4o's ratings correlate with human judgments of scene-prompt alignment. A small human-validation set for the judge, or a held-out judge from a different model family, would substantially strengthen the semantic comparison.
  4. [Sec. 4.1, Asset-Level Policy quantitative evaluation] The only quantitative metric for Asset-Level Policy is the maximum CLIP similarity between a rendered view and the input prompt over four camera positions. Although CLIP is not used as a training reward in this module, this evaluation inherits the same proxy-validity concern as Table 2: CLIP similarity is not established as a valid measure of the semantic and physical quality of small-object placement. The claim that the method places assets in a 'semantically aligned and physically plausible way' would be better supported by human ratings or by a non-CLIP automatic evaluation.
minor comments (6)
  1. [Supplementary Sec. 6.1] The instruction prompt contains the typo 'dictinoary'; it should be 'dictionary'.
  2. [Figure 4] The label 'random sample Function APIs' is unclear; consider rephrasing as 'randomly sample function APIs'.
  3. [Throughout] The paper alternates between '3D-GENERALIST' and '3D-Generalist'; please use one spelling consistently.
  4. [Table 3] The header '3D-GENERALIST(GPT-4o finetuned)' is ungrammatical; it should be '3D-GENERALIST (GPT-4o, fine-tuned)'.
  5. [Sec. 4.1] The subsection 'Additional Ablation on Fine-tuning' describes the visual hallucination benchmark results, but Table 3 is a transfer evaluation rather than an ablation of the self-improvement pipeline; consider renaming the subsection.
  6. [Table 4] The row labeled 'Florence 2 (real)' is ambiguous: it appears to be a publicly released checkpoint pretrained on 5B labels. Please clarify that this is a pretrained checkpoint rather than a dataset, and state explicitly whether the same fine-tuning recipe was applied to all rows.

Circularity Check

1 steps flagged · score 6.0 of 10

CLIP is the selection reward for self-improvement fine-tuning and the evaluation metric for prompt alignment, so Table 2's improvements are not independent; downstream ImageNet pretraining is separate and non-circular.

  1. fitted input called prediction [Sec. 3.2 (Self-Improvement Fine-tuning; In-Context Library in Supplementary 6.1) and Sec. 4.1 (Scene-Level Policy Evaluation, Table 2).]
    "At each state, multiple candidate actions are generated, and the top-scoring action sequences—those that yield the highest CLIP scores between the environment renderings and the input prompt—are retained. These top actions are then used to update the policy parameters via supervised fine-tuning... We report the CLIP score of the final generated scene."

    The exact score used to choose which action sequences become training targets is the same score later reported as evidence of prompt alignment. The supplementary in-context library is similarly 'a curated collection of code snippets that have demonstrated at least a 10% improvement in CLIP-alignment metrics.' Table 2's contrast between fine-tuned (0.254-0.275) and non-fine-tuned (0.242-0.252) policies is therefore a direct consequence of optimizing the evaluator rather than an independent measure of 'more prompt-aligned' scenes. Table 1's GPT-4o semantic scores and Table 4's ImageNet results are external to this loop, but the paper's central Table 2 claim for prompt alignment is not independently established.

full rationale

The circular step is specific and quoted: CLIP selects the self-improvement training data (Sec. 3.2 and supplementary in-context library) and CLIP is the Table 2 evaluation metric for prompt alignment. The reported gains are expected once a policy is trained to imitate CLIP-maximizing actions, so they cannot by themselves validate the claim that fine-tuning yields more prompt-aligned environments. The paper has non-circular components: Table 3 measures hallucination reduction on external benchmarks, and Table 4 pretrains Florence-v2 on generated images and tests on ImageNet-1K, which is independent of CLIP; however, those benchmarks do not measure per-scene prompt alignment, the property at the center of the paper's main claim. Table 1's semantic-coherence judge GPT-4o is the same base model family as the policy, giving only partial independence. The paper's self-citations (Holodeck, LayoutVLM) are used as baselines and metric protocols, not as load-bearing unverified premises, so they do not raise the score. Overall: one central evaluation loop is circular, but the paper contains independent downstream validation, so the score is partial (6/10), not total circularity.

Assumptions & free parameters 4 free parameters · 6 assumptions · 2 invented entities

The central claims rest on a small number of hand-chosen hyperparameters (self-improvement rounds, in-context library threshold, asset placement budget, collision threshold) and on domain assumptions that are not independently verified: CLIP as a prompt-alignment proxy, GPT-4o code generation reliability, multi-view renderings as sufficient state, and repository coverage. The invented artifacts (Scene DSL, in-context library) are design choices without external falsifiable handles.

free parameters (4)
  • number of self-improvement rounds = 3
    Table 2 ablates 0 to 3 rounds; 3 is the default. This is a hand-chosen hyperparameter that the claimed improvement scales with.
  • in-context library CLIP improvement threshold = 10%
    The in-context library retains snippets that yield at least a 10% CLIP improvement (Sec 6.1). This threshold is chosen without a principled basis and affects the diversity and quality of training data.
  • maximum asset placements per receptacle = 10
    Asset-level policy places up to 10 objects per receptacle (Sec 4.1), a fixed budget used in evaluation.
  • collision threshold for physics metric = 0.1 m^2
    A collision is counted when intersecting area exceeds 0.1 m^2 in Blender (Sec 4.1); this threshold is a hand-set evaluation parameter.
assumptions (6)
  • domain assumption CLIP score between a rendered image and the text prompt is a valid and sufficient measure of prompt alignment for 3D environments.
    Used as the reward for self-improvement data selection (Sec 3.2) and as the headline metric in Table 2. If CLIP is a weak proxy for human judgment, the central claim is not supported.
  • domain assumption GPT-4o, when prompted with multi-view renderings and a DSL, can generate syntactically and semantically valid action code that improves scenes.
    The scene-level policy relies on GPT-4o's code generation and the provided function APIs (Sec 3.2, supplementary). The paper does not quantify how often generated code fails to execute or produces invalid scenes.
  • domain assumption Rendered multi-view images with visual markers provide sufficient state for the VLM to make grounded 3D decisions.
    The policy only sees rendered 2D views (Figure 3); the paper assumes this is enough to infer positions, rotations, and scales for action code.
  • domain assumption The asset and material repositories (Objaverse filtered to 200K assets, ambientCG with 2,000 materials) are adequate for open-vocabulary scene generation.
    The system retrieves assets/materials from these repositories; if they lack objects needed for a prompt, prompt alignment fails regardless of policy quality.
  • domain assumption Florence-2 pretraining with detection, segmentation, and box-to-segmentation tasks on rendered synthetic images transfers to ImageNet classification.
    The downstream experiment (Table 4) assumes this transfer protocol is sound and that the comparison against Hypersim at equal label count is a fair test of data quality.
  • standard math Standard 3D geometry and ray casting from pixel locations and camera parameters recover 3D placement points.
    Used in Asset-Level Policy to convert pixel locations to 3D positions (Sec 3.3). This is textbook geometry and is not in question.
invented entities (2)
  • Scene DSL
    purpose: A domain-specific language that represents a 3D environment's floors, walls, ceilings, objects, materials, and lighting, and serves as the action space for the scene-level VLM policy.
    The DSL is a new representation introduced by the paper; it has no falsifiable external handle and its design choices (e.g., position as bounding-box center, z-axis rotation) are ad hoc.
  • In-context library
    purpose: A curated collection of action code snippets that improved CLIP scores by at least 10%, used to promote diversity during data generation.
    This is an algorithmic artifact introduced by the paper; its contents and curation threshold are not externally validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 3D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worlds." pith.science (2026). https://pith.science/paper/CYK5TBBT

@misc{pith2026250706484,
  author       = {Pith},
  title        = {Pith review of: 3D-Generalist: Self-Improving Vision-Language-Action Models for Crafting 3D Worlds},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYK5TBBT}},
  note         = {Machine review of arXiv:2507.06484}
}
read the original abstract

Despite large-scale pretraining endowing models with language and vision reasoning capabilities, improving their spatial reasoning capability remains challenging due to the lack of data grounded in the 3D world. While it is possible for humans to manually create immersive and interactive worlds through 3D graphics, as seen in applications such as VR, gaming, and robotics, this process remains highly labor-intensive. In this paper, we propose a scalable method for generating high-quality 3D environments that can serve as training data for foundation models. We recast 3D environment building as a sequential decision-making problem, employing Vision-Language-Models (VLMs) as policies that output actions to jointly craft a 3D environment's layout, materials, lighting, and assets. Our proposed framework, 3D-Generalist, trains VLMs to generate more prompt-aligned 3D environments via self-improvement fine-tuning. We demonstrate the effectiveness of 3D-Generalist and the proposed training strategy in generating simulation-ready 3D environments. Furthermore, we demonstrate its quality and scalability in synthetic data generation by pretraining a vision foundation model on the generated data. After fine-tuning the pre-trained model on downstream tasks, we show that it surpasses models pre-trained on meticulously human-crafted synthetic data and approaches results achieved with real data orders of magnitude larger.

Figures

Figures reproduced from arXiv: 2507.06484 by the authors.

Figure 1
Figure 1. Sample 3D environments generated by 3D-G [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of 3D-GENERALIST’s Panoramic Environment Generation. We use panoramic diffusion to generate a guiding 360° scene image, then extract the corners, windows, and doors information using a room layout estimation model, Grounded-SAM, and a VLM, respectively. These predictions are then used to construct the 3D room with fixtures procedurally. where precise, instance-level annotations are required or robotics appl… view at source ↗
Figure 3
Figure 3. Overview of our Multi-View Representation It. This illustrates how a 3D environment is rendered and fed as input to our VLM policy in Scene-Level Policy. The first view is overlaid with visual marks of the x-y coordinate system of the current environment, the second view is a panoramic render of the room, and the third is overlaid with the variable names for the existing asset instances. then, we use an inverse grap… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Overview of 3D-GENERALIST’s Scene-Level Policy. Starting with the current 3D scene, we render multiple views and combine the text prompt with an in-context example to guide a vision-language model (VLM) acting as an action policy. The VLM, during every round of the ite…
Figure 5
Figure 5. Figure 5 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Evaluation of the self-improving fine-tuning strategy. The top graph shows averaged results, indicating that GPT-4o im￾proves 3D prompt alignment iteratively only after self-improvement fine-tuning. The bottom examples highlight 3D-GENERALIST’s self-correction: refinin…
Figure 7
Figure 7. Figure 7: Comparison between the baseline inpainting method (left) and 3D-G [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: We experimented with two approaches for the asset level [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 34 canonical work pages

  1. [1]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  2. [2]

    Open-universe indoor scene generation using llm program synthesis and uncurated object databases.arXiv preprint arXiv:2403.09675, 2024

    Rio Aguina-Kang, Maxim Gumin, Do Heon Han, Stew- art Morris, Seung Jean Yoo, Aditya Ganeshan, R Kenny Jones, Qiuhong Anna Wei, Kailiang Fu, and Daniel Ritchie. Open-universe indoor scene generation using llm program synthesis and uncurated object databases.arXiv preprint arXiv:2403.09675, 2024. 3

    arXiv 2024

  3. [3]

    I-design: Personal- ized llm interior designer.arXiv preprint arXiv:2404.02838,

    Ata C ¸ elen, Guo Han, Konrad Schindler, Luc Van Gool, Iro Armeni, Anton Obukhov, and Xi Wang. I-design: Personal- ized llm interior designer.arXiv preprint arXiv:2404.02838,

    arXiv

  4. [4]

    Learning spatial knowledge for text to 3d scene generation

    Angel Chang, Manolis Savva, and Christopher D Manning. Learning spatial knowledge for text to 3d scene generation. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 2028–2038,

    work page 2014

  5. [5]

    Sceneseer: 3d scene design with natural language.arXiv preprint arXiv:1703.00050, 2017

    Angel X Chang, Mihail Eric, Manolis Savva, and Christo- pher D Manning. Sceneseer: 3d scene design with natural language.arXiv preprint arXiv:1703.00050, 2017. 2 9

    arXiv 2017

  6. [6]

    Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties

    Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. Spatialvlm: Endow- ing vision-language models with spatial reasoning capabili- ties. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 14455– 14465, 2024. 3

    work page 2024

  7. [7]

    Urdformer: A pipeline for constructing articulated simulation environments from real-world images

    Zoey Chen, Aaron Walsman, Marius Memmel, Kaichun Mo, Alex Fang, Karthikeya Vemuri, Alan Wu, Dieter Fox, and Abhishek Gupta. Urdformer: A pipeline for constructing articulated simulation environments from real-world images. arXiv preprint arXiv:2405.11656, 2024. 2

    arXiv 2024

  8. [8]

    Spatial- rgpt: Grounded spatial reasoning in vision language model

    An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Rui- han Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatial- rgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:2406.01584, 2024. 3

    arXiv 2024

Show all 64 references
  1. [9]

    Wordseye: An automatic text-to-scene conversion system

    Bob Coyne and Richard Sproat. Wordseye: An automatic text-to-scene conversion system. InProceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 487–496, 2001. 2

    2001

  2. [10]

    Procthor: Large-scale embodied ai using procedural generation.Ad- vances in Neural Information Processing Systems, 35:5982– 5994, 2022

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation.Ad- vances in Neural Information Processing Systems, 35:5982–...

    2022

  3. [11]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Reco...

    2023

  4. [12]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv preprint arXiv:2409.17146, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv preprint arXiv:2409.17146, 20...

    2024 arXiv

  5. [13]

    ImageNet: A large-scale hierarchical im- age database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical im- age database. InIEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. 9

    2009

  6. [14]

    Disentangled 3D scene genera- tion with layout learning.arXiv preprint arXiv:2402.16936,

    Dave Epstein, Ben Poole, Ben Mildenhall, Alexei A Efros, and Aleksander Holynski. Disentangled 3D scene genera- tion with layout learning.arXiv preprint arXiv:2402.16936,

    arXiv

  7. [15]

    Diffusion360: Seamless 360 degree panoramic im- age generation based on diffusion models.arXiv preprint arXiv:2311.13141, 2023

    Mengyang Feng, Jinlin Liu, Miaomiao Cui, and Xuansong Xie. Diffusion360: Seamless 360 degree panoramic im- age generation based on diffusion models.arXiv preprint arXiv:2311.13141, 2023. 3

    2023 arXiv

  8. [16]

    Layoutgpt: Compositional visual plan- ning and generation with large language models.Advances in Neural Information Processing Systems, 36, 2024

    Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Ar- jun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual plan- ning and generation with large language models.Advances in Neural Information Processing Systems, 36, 2024. 3, 7, 4

    2024

  9. [17]

    Example-based synthesis of 3d object arrangements.ACM Transactions on Graphics (TOG), 31(6):1–11, 2012

    Matthew Fisher, Daniel Ritchie, Manolis Savva, Thomas Funkhouser, and Pat Hanrahan. Example-based synthesis of 3d object arrangements.ACM Transactions on Graphics (TOG), 31(6):1–11, 2012. 3

    2012

  10. [18]

    Any- home: Open-vocabulary generation of structured and tex- tured 3d homes

    Rao Fu, Zehao Wen, Zichen Liu, and Srinath Sridhar. Any- home: Open-vocabulary generation of structured and tex- tured 3d homes. InEuropean Conference on Computer Vi- sion, pages 52–70. Springer, 2024. 4

    2024

  11. [19]

    Rel3D: A minimally contrastive benchmark for grounding spatial relations in 3d.Advances in Neural Information Pro- cessing Systems, 33:10514–10525, 2020

    Ankit Goyal, Kaiyu Yang, Dawei Yang, and Jia Deng. Rel3D: A minimally contrastive benchmark for grounding spatial relations in 3d.Advances in Neural Information Pro- cessing Systems, 33:10514–10525, 2020. 3

    2020

  12. [20]

    Text2Room: Extracting textured 3D meshes from 2D text-to-image models

    Lukas H ¨ollein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2Room: Extracting textured 3D meshes from 2D text-to-image models. InProceedings of the IEEE/CVF International Conference on Computer Vi- sion, pages 7909–7920, 2023. 7

    2023

  13. [21]

    3D-LLM: In- jecting the 3D world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494,

    Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3D-LLM: In- jecting the 3D world into large language models.Advances in Neural Information Processing Systems, 36:20482–20494,

  14. [22]

    Scenecraft: An llm agent for synthesizing 3d scenes as blender code

    Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A Ross, Cordelia Schmid, and Alireza Fathi. Scenecraft: An llm agent for synthesizing 3d scenes as blender code. InForty-first International Conference on Ma- chine Learning, 2024. 3, 4

    2024

  15. [23]

    Large language models cannot self-correct reasoning yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023. 2

    2023 arXiv

  16. [24]

    Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917, 2024

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning.arXiv preprint arXiv:2409.12917, 2024. 2

    2024 arXiv

  17. [25]

    InstructScene: Instruction- driven 3D indoor scene synthesis with semantic graph prior

    Chenguo Lin and Yadong Mu. InstructScene: Instruction- driven 3D indoor scene synthesis with semantic graph prior. arXiv preprint arXiv:2402.04717, 2024. 3

    2024 arXiv

  18. [26]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  19. [27]

    Material palette: Extraction of materials from a single image

    Ivan Lopes, Fabio Pizzati, and Raoul de Charette. Material palette: Extraction of materials from a single image. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4379–4388, 2024. 9

    2024

  20. [28]

    Language-driven synthe- sis of 3d scenes from scene databases.ACM Transactions on Graphics (TOG), 37(6):1–16, 2018

    Rui Ma, Akshay Gadi Patil, Matthew Fisher, Manyi Li, S¨oren Pirk, Binh-Son Hua, Sai-Kit Yeung, Xin Tong, Leonidas Guibas, and Hao Zhang. Language-driven synthe- sis of 3d scenes from scene databases.ACM Transactions on Graphics (TOG), 37(6):1–16, 2018. 2

    2018

  21. [29]

    RoboCasa: Large-scale simula- tion of everyday tasks for generalist robots.arXiv preprint arXiv:2406.02523, 2024

    Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Man- dlekar, and Yuke Zhu. RoboCasa: Large-scale simula- tion of everyday tasks for generalist robots.arXiv preprint arXiv:2406.02523, 2024. 2

    2024 arXiv

  22. [30]

    NVIDIA, Maciej Bala, Yin Cui, Yifan Ding, Yunhao Ge, Zekun Hao, Jon Hasselgren, Jacob Huffman, Jingyi Jin, 10 J.P. Lewis, Zhaoshuo Li, Chen-Hsuan Lin, Yen-Chen Lin, Tsung-Yi Lin, Ming-Yu Liu, Alice Luo, Qianli Ma, Ja- cob Munkberg, Stella Shi, Fangyin Wei, Donglai Xiang, Ji- a...

    2024 arXiv

  23. [31]

    Atiss: Autoregres- sive transformers for indoor scene synthesis.Advances in Neural Information Processing Systems, 34:12013–12026,

    Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregres- sive transformers for indoor scene synthesis.Advances in Neural Information Processing Systems, 34:12013–12026,

  24. [32]

    Advances in data- driven analysis and synthesis of 3d indoor scenes

    Akshay Gadi Patil, Supriya Gadi Patil, Manyi Li, Matthew Fisher, Manolis Savva, and Hao Zhang. Advances in data- driven analysis and synthesis of 3d indoor scenes. InCom- puter Graphics Forum, page e14927. Wiley Online Library,

  25. [33]

    Compositional 3d scene generation using locally conditioned diffusion

    Ryan Po and Gordon Wetzstein. Compositional 3d scene generation using locally conditioned diffusion. In2024 In- ternational Conference on 3D Vision (3DV), pages 651–663. IEEE, 2024. 2

    2024

  26. [34]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  27. [35]

    Lay-a-scene: Personalized 3d object arrangement using text-to-image priors.arXiv preprint arXiv:2406.00687, 2024

    Ohad Rahamim, Hilit Segev, Idan Achituve, Yuval Atzmon, Yoni Kasten, and Gal Chechik. Lay-a-scene: Personalized 3d object arrangement using text-to-image priors.arXiv preprint arXiv:2406.00687, 2024. 3

    2024 arXiv

  28. [36]

    Grounded sam: Assembling open-world models for diverse visual tasks,

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kun- chang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks,

  29. [37]

    Susskind

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic syn- thetic dataset for holistic indoor scene understanding. In International Conference on Computer Vision (ICCV) 2021,

    2021

  30. [38]

    Object hallucination in image cap- tioning.arXiv preprint arXiv:1809.02156, 2018

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image cap- tioning.arXiv preprint arXiv:1809.02156, 2018. 7, 4

    2018 arXiv

  31. [39]

    Controlroom3d: Room gen- eration using semantic proxy rooms

    Jonas Schult, Sam Tsai, Lukas H ¨ollein, Bichen Wu, Jialiang Wang, Chih-Yao Ma, Kunpeng Li, Xiaofang Wang, Felix Wimbauer, Zijian He, et al. Controlroom3d: Room gen- eration using semantic proxy rooms. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2024

  32. [40]

    Real-time automatic 3d scene generation from natural language voice and text de- scriptions

    Lee M Seversky and Lijun Yin. Real-time automatic 3d scene generation from natural language voice and text de- scriptions. InProceedings of the 14th ACM international conference on Multimedia, pages 61–64, 2006. 2

    2006

  33. [41]

    Horizonnet: Learning room layout with 1d represen- tation and pano stretch data augmentation

    Cheng Sun, Chi-Wei Hsiao, Min Sun, and Hwann-Tzong Chen. Horizonnet: Learning room layout with 1d represen- tation and pano stretch data augmentation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1047–1056, 2019. 4

    2019

  34. [42]

    LayoutVLM: Differentiable optimization of 3D layout via vision-language models.arXiv preprint arXiv:2412.02193,

    Fan-Yun Sun, Weiyu Liu, Siyi Gu, Dylan Lim, Goutam Bhat, Federico Tombari, Manling Li, Nick Haber, and Jiajun Wu. LayoutVLM: Differentiable optimization of 3D layout via vision-language models.arXiv preprint arXiv:2412.02193,

    arXiv

  35. [43]

    Factorsim: Generative simulation via factorized rep- resentation.Advances in Neural Information Processing Sys- tems, 37:87438–87472, 2024

    Fan-Yun Sun, Harini SI, Angela Yi, Yihan Zhou, Alex Zook, Jonathan Tremblay, Logan Cross, Jiajun Wu, and Nick Haber. Factorsim: Generative simulation via factorized rep- resentation.Advances in Neural Information Processing Sys- tems, 37:87438–87472, 2024. 9

    2024

  36. [44]

    Partial-view object view synthesis via filtering inversion

    Fan-Yun Sun, Jonathan Tremblay, Valts Blukis, Kevin Lin, Danfei Xu, Boris Ivanovic, Peter Karkus, Stan Birchfield, Dieter Fox, Ruohan Zhang, et al. Partial-view object view synthesis via filtering inversion. InInternational Conference on 3D Vision (3DV), pages 453–463, 2024. 9

    2024

  37. [45]

    Lgm: Large multi-view gaussian model for high-resolution 3d content creation

    Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. InEuropean Conference on Computer Vision, pages 1–18. Springer, 2024. 4

    2024

  38. [46]

    DiffuScene: Denoising diffu- sion models for generative indoor scene synthesis

    Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. DiffuScene: Denoising diffu- sion models for generative indoor scene synthesis. InPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20507–20518, 2024. 3, 7

    2024

  39. [47]

    An LLM-free multi-dimensional benchmark for MLLMs hallucination evaluation.arXiv preprint arXiv:2311.07397,

    Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. An LLM-free multi-dimensional benchmark for MLLMs hallucination evaluation.arXiv preprint arXiv:2311.07397,

    arXiv

  40. [48]

    Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models.arXiv preprint arXiv:2406.14852, 2024

    Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth a thou- sand words? delving into spatial reasoning for vision lan- guage models.arXiv preprint arXiv:2406.14852, 2024. 3

    2024 arXiv

  41. [49]

    Robogen: Towards unleashing infi- nite data for automated robot learning via generative simula- tion.arXiv preprint arXiv:2311.01455, 2023

    Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infi- nite data for automated robot learning via generative simula- tion.arXiv preprint arXiv:2311.01455, 2023. 9

    2023 arXiv

  42. [50]

    Architect: Generating vivid and interactive 3d scenes with hierarchical 2d inpainting, 2024

    Yian Wang, Xiaowen Qiu, Jiageng Liu, Zhehuan Chen, Jit- ing Cai, Yufei Wang, Tsun-Hsuan Wang, Zhou Xian, and Chuang Gan. Architect: Generating vivid and interactive 3d scenes with hierarchical 2d inpainting, 2024. 4

    2024

  43. [51]

    Florence-2: Advancing a unified representation for a variety of vision tasks

    Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), page...

    2024

  44. [52]

    InstantMesh: Efficient 3D mesh generation from a single image with sparse-view large reconstruction models.arXiv preprint arXiv:2404.07191,

    Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. InstantMesh: Efficient 3D mesh generation from a single image with sparse-view large reconstruction models.arXiv preprint arXiv:2404.07191,

    arXiv

  45. [53]

    Set-of-mark prompting unleashes 11 extraordinary visual grounding in GPT-4v.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes 11 extraordinary visual grounding in GPT-4v.arXiv preprint arXiv:2310.11441, 2023. 3

    2023 arXiv

  46. [54]

    Physcene: Physically interactable 3d scene synthe- sis for embodied ai

    Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. Physcene: Physically interactable 3d scene synthe- sis for embodied ai. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 16262–16272, 2024. 7, 4

    2024

  47. [55]

    Holodeck: Language guided gen- eration of 3d embodied ai environments

    Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Al- varo Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language guided gen- eration of 3d embodied ai environments. InProceedings of the IEEE/CVF Conference on Computer Vision and P...

    2024

  48. [56]

    The clutterpalette: An interactive tool for detailing indoor scenes

    Lap-Fai Yu, Sai-Kit Yeung, and Demetri Terzopoulos. The clutterpalette: An interactive tool for detailing indoor scenes. IEEE transactions on visualization and computer graphics, 22(2):1138–1148, 2015. 3

    2015

  49. [57]

    RLF-V: Towards trustworthy MLLMs via behavior alignment from fine-grained correctional hu- man feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. RLF-V: Towards trustworthy MLLMs via behavior alignment from fine-grained correctional hu- man feedback. InProceedings of the IEEE/CVF Conference on Co...

    2024

  50. [58]

    Clay: A controllable large-scale generative model for creat- ing high-quality 3d assets.ACM Transactions on Graphics (TOG), 43(4):1–20, 2024

    Longwen Zhang, Ziyu Wang, Qixuan Zhang, Qiwei Qiu, Anqi Pang, Haoran Jiang, Wei Yang, Lan Xu, and Jingyi Yu. Clay: A controllable large-scale generative model for creat- ing high-quality 3d assets.ACM Transactions on Graphics (TOG), 43(4):1–20, 2024. 3

    2024

  51. [59]

    SceneWiz3D: To- wards text-guided 3d scene composition.arXiv preprint arXiv:2312.08885, 2023

    Qihang Zhang, Chaoyang Wang, Aliaksandr Siarohin, Peiye Zhuang, Yinghao Xu, Ceyuan Yang, Dahua Lin, Bolei Zhou, Sergey Tulyakov, and Hsin-Ying Lee. SceneWiz3D: To- wards text-guided 3d scene composition.arXiv preprint arXiv:2312.08885, 2023. 7

    2023 arXiv

  52. [60]

    DreamScene360: Uncon- strained text-to-3D scene generation with panoramic gaus- sian splatting

    Shijie Zhou, Zhiwen Fan, Dejia Xu, Haoran Chang, Pradyumna Chari, Tejas Bharadwaj, Suya You, Zhangyang Wang, and Achuta Kadambi. DreamScene360: Uncon- strained text-to-3D scene generation with panoramic gaus- sian splatting. InEuropean Conference on Computer Vision, pages 324–...

    2025

  53. [61]

    GALA3D: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting.arXiv preprint arXiv:2402.07207, 2024

    Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong, Jinlin He, Zhi- wei Lin, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. GALA3D: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting.arXiv preprint arXiv:2402.07207, 2024. 2, 7 12 3D-GENERALIST: Vi...

    2024 arXiv

  54. [62]

    placements

    Additional Details of 3D-GENERALIST Here, we provide more detail for the implementation of var- ious components in 3D-GENERALIST. 6.1. Scene-Level Policy The base prompt we use as input to our VLM is: **Task**: As a programmer, you are required to complete an implementation. U...

  55. [63]

    Details of our Experiments We make design choices for our experiments to remain within a reasonable time and compute budget. Instead of leveraging text or image-to-3D asset generators [58], dif- fusion models for PBR material synthesis, or the latest state-of-the-art Vision-La...

  56. [64]

    receptacle objects

    is not anopen-vocabularylayout generation method, it can only generate layout for a fixed set of categories. 7.3. Asset-Level Policy’s Evaluation ProcTHOR [10] also has a method to place small objects on surfaces of “receptacle objects”. However, rather than 2https://ambientcg...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.