Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A zero-training pipeline turns 2D inpainting into interactable 3D scene generation by anchoring generated objects to known rendered depth.

desk verdict Architect is a clever, clearly-explained pipeline that controls camera and depth by inpainting into simulation-rendered backgrounds, but its load-bearing depth-rescale assumption is never validated against 3D ground truth. read the letter →

arxiv 2411.09823 v1 pith:J5A76BFH submitted 2024-11-14 cs.CV

Yian Wang , Xiaowen Qiu , Jiageng Liu , Zhehuan Chen , Jiting Cai , Yufei Wang , Tsun-Hsuan Wang , Zhou Xian
show 1 more author
Chuang Gan
This is my paper · ORCID
classification cs.CV
keywords 3Dscenegeneration2Dinpaintingdiffusionmodelsmonoculardepthestimationzero-shotembodiedAIinteractiveenvironmentshierarchical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Architect proposes that a pre-trained 2D image inpainting model can serve as the spatial planner for building interactive 3D scenes, without training a scene generator. The key move is to render an empty room in simulation, where camera parameters and depth are known exactly, and then ask a diffusion model to inpaint furniture and small objects into that image. Because the known background depth anchors the generated foreground, the depth predicted for the inpainted objects can be rescaled and back-projected to obtain 3D bounding boxes. Repeated hierarchical inpainting from multiple views fills large furniture first and then small objects, producing complex, open-vocabulary environments for embodied AI and robotics. If the approach works, it turns internet-scale 2D image priors directly into interactable simulation scenes.

What carries the argument

The central object is the controlled inpainted image: a simulation render of an empty scene with known camera parameters and ground-truth depth, used as the conditioning template for a latent diffusion inpainting model. The mechanism that carries the argument is the affine depth rescale: for each inpainted image, reference pixels are sampled from the non-masked background (all non-masked pixels for large furniture; pixels on the supporting furniture for small objects), and the monocular depth estimate is rescaled with a scale and offset computed from those reference pairs. This converts the scale-ambiguous predicted depth into the metric depth coordinate of the render, so back-projection with the known camera gives geometrically consistent 3D bounding boxes. The hierarchical loop of render, inpaint, perceive, place, and repeat is what extends a single view into a full multi-scale scene.

What would settle it

Render a known empty room, inpaint a single object at a known position and distance, then compare the rescaled predicted depth inside the inpainted mask to the ground-truth depth from the render; if the per-pixel error is large or grows with distance, the back-projection claim fails, and this measurement is not reported in the paper.

Watch

Extended reading notes

Core claim

The discovery is that a diffusion inpainting model can be made to inherit the geometry of a simulation-rendered background. Starting from a photo-realistic empty room rendered with known camera intrinsics and ground-truth depth, Architect inpaints a masked foreground region. The paper argues that the geometric cues visible in the background inform the inpainting model's camera parameters, so the generated image is effectively captured from the same camera; and because the known background depth provides reference pixels, the estimated depth of the inpainted objects can be rescaled affinely with a per-image scale and offset, then back-projected to 3D. From the inpainted image, off-the-shelf perception models name, detect, and segment the new objects, a monocular depth estimator supplies relative depth, and DBSCAN cleans the point clouds before axis-aligned bounding boxes are produced. Those boxes constrain placement of retrieved or generated assets, and the loop repeats at a finer scale for small objects inside or on top of furniture. The paper claims this yields diverse, realistic, interactive scenes from text, floor plans, or pre-arranged scenes without any layout training.

Load-bearing premise

The load-bearing assumption is that the depth predicted for a freshly inpainted object lines up with the known background depth after one simple rescale—that is, that the depth estimator sees both in the same coordinate system.

Editorial extensions

If this is right

  • Text-to-scene generation becomes a zero-shot retrieval-and-placement process: any room type expressible by an inpainting model can be turned into an interactive simulation without layout training data.
  • Scenes can be seeded from different starting points—text, floor plans, or already-arranged environments—and refined iteratively, since each inpainting step only needs a rendered view and a mask.
  • Small-object placement, which LLM-based scene designers struggle with, is inherited from the 2D model's spatial priors rather than from explicit rules.
  • Generated scenes support downstream embodied tasks: long-horizon robot manipulation and navigation trajectories can be synthesized on the resulting assets.
  • Scene editing becomes possible because the final state is a list of named objects with positions, scales, bounding boxes, and orientations that an LLM can call as APIs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct validation of the affine-depth assumption would be the cheapest test: render a known scene, inpaint an object at a known location, and compare the rescaled predicted depth inside the mask against the ground-truth depth from the render; the paper reports similarity scores but not this error.
  • If the affine assumption holds only approximately, errors will compound with iteration, since each newly placed object is anchored to previously placed ones through the rendered scene; later small-object placements inherit the drift.
  • The method's dependence on an asset database for large furniture means diversity is bounded by retrieval even though the 2D prior is open-vocabulary; the paper's text-to-3D furniture pipeline is a step toward removing that bound.
  • The same render-empty-background and inpaint-foreground recipe could extend beyond rooms to streetscapes, stores, or outdoor scenes wherever a simulator can supply camera and depth, effectively turning the 2D model into a scene layout prior for arbitrary embodied domains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents Architect, a zero-shot pipeline that generates interactive 3D scenes by iteratively inpainting foreground objects into images rendered from an empty simulated room. Starting from a floor plan, an LLM proposes inpainting prompts, a diffusion inpainting model fills masked regions, and visual perception models (GPT-4V, Grounding-DINO, SAM) detect and segment objects. Monocular depth from Marigold is rescaled against known rendered depth in non-masked regions and then back-projected using known camera parameters to produce 3D bounding boxes, which are used to retrieve or generate 3D assets and place them in simulation. The pipeline operates hierarchically, first placing large furniture and then small objects, and is claimed to support text, floor-plan, and pre-arranged-scene inputs. The authors compare against Holodeck, Text2Room, and DiffuScene with image-text alignment metrics, a GPT-4o ranking, and a user study, and they demonstrate several embodied robot tasks in the generated scenes.

Significance. If the central claim holds, Architect is a significant systems contribution: it generates open-vocabulary, interactive, cluttered 3D scenes without training a scene model, leveraging the strong layout priors of pretrained 2D diffusion models and grounding them with known camera and depth information from simulation. The paper's strengths include a clear and modular pipeline, zero-shot generalization beyond household scenes (e.g., grocery stores, hospitals), flexible initialization from text or floor plans, and concrete demonstrations of downstream embodied tasks. The authors also report a user study and a set of quantitative comparisons. However, the quantitative evidence is almost entirely 2D image-text alignment or subjective preference; the load-bearing assertion of accurate 2D-to-3D lifting is not directly validated. The paper's own checklist explicitly acknowledges that error bars are not reported, and the depth-rescaling step in Section 3.3 is unchecked inside the inpainted regions where the generated objects actually live. Because the contribution's value depends on producing geometrically correct placements, this validation gap is central rather than cosmetic.

major comments (3)
  1. [§1 and §3.3, Eq. (1)] The central claim that inpainting into a rendered background controls camera parameters and depth scale so that images can be accurately back-projected is not validated inside the inpainted regions. Eq. (1) rescales Marigold's predicted depth with a single affine map computed from reference pixels in the non-masked background. This corrects a global scale and offset but cannot correct local geometric errors in the generated foreground, such as an object placed at the wrong distance, an incorrect floor contact, or an implausible object size. The paper provides no direct comparison of the resulting 3D bounding boxes against ground truth. I request a synthetic evaluation: generate scenes with known object placements, run the full pipeline, and report per-object center error, 3D IoU, and scale error, with separate breakdowns for the large-furniture and small-object branches. Without such an experiment, the contribution bullet claiming 'accurate 2D to 3D lifting' is unsupported.
  2. [§4, Tables 2 and 4] The quantitative evaluation in Tables 2 and 4 uses CLIPScore, BLIPScore, VQAScore, GPT-4o ranking, and a user study. These metrics measure 2D image-text alignment and holistic visual preference; they cannot detect a misplaced object, a wrong depth scale, a collision, or an incorrect bounding-box size. A method that produces photorealistic but geometrically wrong scenes could score well on all of these. The user-study 'Layout Correctness' is a subjective one-to-five score, not a spatial accuracy measurement. To support the paper's placement claims, the evaluation needs direct 3D metrics against known layouts, or at minimum quantitative comparisons of wall/floor contact, inter-object penetration, and bounding-box alignment with the inpainted images.
  3. [Table 3] The 'Inpaint vs. Placed' similarity in Table 3 measures the agreement between the inpainted image and a rendered image of the scene produced by the same pipeline. This is a consistency check, not an accuracy measure: both images can agree while sharing the same geometric errors, and a high 2D similarity can coexist with incorrect depth or placement. The 'LLM Placement VQScore' likewise compares one placement method against another without grounding in true positions. Additionally, the paper reports no error bars or confidence intervals anywhere; checklist item 7 explicitly states that error bars are not reported because of computational expense. Given that all quantitative claims are aggregate scores, the absence of variance information makes it impossible to assess whether reported differences are meaningful.
minor comments (6)
  1. [Throughout] There are numerous typos and inconsistencies: 'hierachical' in the abstract, 'previos' in the Table 1 caption, 'devided' in Appendix A.1, and inconsistent spelling of 'DiffuScene'/'Diffscene' and 'GPT4v'/'GPT-4V'. A careful proofreading pass is needed.
  2. [§3.2] The 'predefined criterion' for filtering generated inpainted images is not specified. Since this filtering directly affects which images proceed to visual perception and placement, the threshold and the number of generated candidates per mask should be stated.
  3. [§4 and Appendix D] The user study says all participants were volunteers without compensation, which is acceptable, but the paper should state how many scenes per method were rated, how the 23 scenes were sampled, and whether the raters were shown matched room captions; otherwise the reported averages are hard to interpret.
  4. [Appendix A.4] The claim that an 84-degree FOV yields approximately 95 percent of a square room visible from a corner-to-corner view should be justified with a formula or citation; the geometry is not obvious without knowing the room aspect ratio and camera height.
  5. [§4.3 and Figure 6] The caption 'Figure 6 right part' appears twice and is redundant; the text refers to 'Figure 6 right part' where 'Figure 6, right' would suffice. Also, the object-generation subsection would benefit from quantitative or at least more systematic qualitative evidence, as it currently reports only examples.
  6. [Appendix A.2] The asset-selection procedure uses scale similarity and CLIP feature similarity, but the relative weights or thresholds for combining them are not given; without these, the placement and retrieval behavior is not fully reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the depth-rescale step is a calibration against known rendered depth, not a self-defined prediction; the central 3D-lifting claim is under-validated but not circular.

full rationale

The load-bearing step is the affine rescale of Marigold depth in Section 3.3 using reference pixels from the known rendered depth. This is not circular: the scale and shift are fit to known background depths and then applied to inpainted foreground depths, which is an extrapolation that can fail rather than a quantity equal to its input by construction. The claim that inpainting 'effectively controls the camera parameters and depth scale' is grounded in the use of simulation-rendered views with known intrinsics and ground-truth depth, followed by a calibration; no equation in the paper reduces the output to the input. RoboGen is cited by overlapping authors, but only as inspiration for the embodied-task data-collection extension, not as evidence for the core pipeline, so the self-citation is not load-bearing. The evaluation weakness is real but is a validation gap, not circularity: Table 3's 'Inpaint vs. Placed' similarity compares outputs of the same pipeline, and NeurIPS Checklist item 7 explicitly says no error bars are reported; a synthetic ground-truth 3D bounding-box test would be needed to support the spatial-accuracy claim. None of these limitations makes the derivation equivalent to its inputs, so no circular step is exhibited.

Assumptions & free parameters 7 free parameters · 7 assumptions · 0 invented entities

The paper introduces no new particles, forces, dimensions, or conserved quantities; it is a software pipeline over existing models and datasets. The central claim rests on assumptions about pretrained model behavior and several hand-set heuristics rather than a new theoretical derivation.

free parameters (7)
  • Camera FOV and height for room views = 84 degrees; 1.8 m height
    Chosen as 'standard' in Appendix A.4; determines rendered background and back-projection geometry.
  • Room occupancy stopping threshold = 0.7
    Stops inpainting from new views when room occupancy exceeds 0.7 in Appendix A.4; affects scene completeness and clutter.
  • Minimum recognized-object criterion = unspecified
    Section 3.2 filters inpainted images when recognized objects fall below a 'predefined criterion', introducing a hand-set quality gate.
  • Cube fill dimensions for small objects = slightly smaller; fixed height
    Section 3.1 uses a cube within or atop the furniture bbox to define the inpainting mask; cube size directly controls where and how small objects are generated.
  • Depth-rescale reference pixel set = hand-chosen per tier
    Section 3.3 uses all non-masked pixels for large furniture and non-masked furniture pixels for small objects; this choice affects the scale and offset of predicted depth.
  • Mask erosion and Gaussian blur amounts = unspecified
    Section 3.2 mentions erosion and blur to soften masks but gives no parameters.
  • DBSCAN clustering parameters = unspecified
    Section 3.3 uses DBSCAN to remove outliers and form object bboxes; eps and min samples are not reported.
assumptions (7)
  • domain assumption Pre-trained 2D image generation and inpainting models encode realistic furniture layouts and inter-object spatial configurations.
    Invoked in Section 1 as the core motivation: 'pre-trained image-based generative models are able to better capture scene and object configurations.'
  • ad hoc to paper Inpainting a foreground into a rendered background forces the generated objects to be consistent with the background's camera and depth scale.
    Central mechanism stated in Section 1: background geometric cues inform camera parameters and depth scale; no direct geometric evaluation is provided.
  • domain assumption Marigold or similar monocular depth predictors produce relative depth that can be converted to metric depth by an affine rescale using sparse reference pixels.
    Section 3.3 rescales De using min and max and mean of reference pixels from ground-truth depth; assumes the same affine relation holds in newly inpainted regions.
  • domain assumption GPT-4V and Grounded-SAM correctly detect and segment objects in inpainted images.
    Section 3.3 relies on these models to obtain object bboxes and masks; errors propagate to placement.
  • domain assumption Retrieved 3D assets from Objaverse and PartNetMobility are faithful stand-ins for objects in the inpainted image.
    Section 3.4 retrieves and scales assets by bbox similarity; mismatch between asset geometry and generated image is a known limitation acknowledged in Section 5.
  • domain assumption The constraint search over bboxes can place furniture without collisions while matching the image.
    Section 3.4 and Appendix A.1 assume DFS with greedy pruning over derived constraints avoids collision and alignment issues.
  • domain assumption LLM-generated positive and negative prompts improve inpainting quality.
    Section 3.2 asks LLMs to propose lacking and reached-limit objects; this assumes the LLM's scene-composition knowledge helps the diffusion model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting." pith.science (2026). https://pith.science/paper/J5A76BFH

@misc{pith2026241109823,
  author       = {Pith},
  title        = {Pith review of: Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J5A76BFH}},
  note         = {Machine review of arXiv:2411.09823}
}
read the original abstract

Creating large-scale interactive 3D environments is essential for the development of Robotics and Embodied AI research. Current methods, including manual design, procedural generation, diffusion-based scene generation, and large language model (LLM) guided scene design, are hindered by limitations such as excessive human effort, reliance on predefined rules or training datasets, and limited 3D spatial reasoning ability. Since pre-trained 2D image generative models better capture scene and object configuration than LLMs, we address these challenges by introducing Architect, a generative framework that creates complex and realistic 3D embodied environments leveraging diffusion-based 2D image inpainting. In detail, we utilize foundation visual perception models to obtain each generated object from the image and leverage pre-trained depth estimation models to lift the generated 2D image to 3D space. Our pipeline is further extended to a hierarchical and iterative inpainting process to continuously generate placement of large furniture and small objects to enrich the scene. This iterative structure brings the flexibility for our method to generate or refine scenes from various starting points, such as text, floor plans, or pre-arranged environments.

Figures

Figures reproduced from arXiv: 2411.09823 by the authors.

Figure 1
Figure 1. We present ARCHITECT, a generative framework to create diverse, realistic, and complex Embodied AI scenes. Leveraging 2D diffusion models, ARCHITECT generates scenarios in an open-vocabulary manner. Here, we showcase two cases in detail: an apartment and a grocery store. Collecting or generating large-scale training data has recently emerged as a promising direction for advancing Robotics and Embodied AI research. A… view at source ↗
Figure 2
Figure 2. Demonstration of our pipeline that generate complex interactive environment starting from [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. We compare ARCHITECT with other methods in both household scenes(living room and dining room) and other non-household scenes. We only compared the household scene generated by Diffuscene due to its limitations in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Two robot manipulation tasks generated in our scene setting. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Left: the robot organizes the room by pushing the chair under the table and pushing the keyboard inside the table. Right: the robot opens the fridge door, grasps the mango and puts it into the fridge, opens the kitchen-dining room door, grasps the soda can and puts it …
Figure 6
Figure 6. Figure 6: Examples of generated small objects and large furniture. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Comparison of living room and dining room scene generated by [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Comparison of four scenes generated by A [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Demonstration of comparison between different mask, different small objects placement [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: The example questionnaire for participants. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Generative Physical AI in Vision: A Survey

    cs.CV 2025-01 conditional novelty 6.0 of 10

    A structured review that categorizes physics-aware generative models in vision into explicit-simulation and implicit-learning families and proposes six integration paradigms.

Reference graph

Works this paper leans on

72 extracted references · 33 canonical work pages · cited by 1 Pith paper

  1. [1]

    Open-universe indoor scene generation using llm program synthesis and uncurated object databases

    Rio Aguina-Kang, Maxim Gumin, Do Heon Han, Stewart Morris, Seung Jean Yoo, Aditya Ganeshan, R Kenny Jones, Qiuhong Anna Wei, Kailiang Fu, and Daniel Ritchie. Open-universe indoor scene generation using llm program synthesis and uncurated object databases. arXiv preprint arXiv:2403.09675, 2024. 10

    arXiv 2024

  2. [2]

    Zoedepth: Zero-shot transfer by combining relative and metric depth

    Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288, 2023

    arXiv 2023

  3. [3]

    Imitating task and motion planning with visuomotor transformers

    Murtaza Dalal, Ajay Mandlekar, Caelan Garrett, Ankur Handa, Ruslan Salakhutdinov, and Dieter Fox. Imitating task and motion planning with visuomotor transformers. arXiv preprint arXiv:2305.16309, 2023

    arXiv 2023

  4. [4]

    Procthor: Large-scale embodied ai using procedural generation

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 35:5982–5994, 2022

    2022

  5. [5]

    Objaverse-xl: A universe of 10m+ 3d objects

    Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram V oleti, Samir Yitzhak Gadre, Eli VanderBilt, Aniruddha Kembhavi, Carl V ondrick, Georgia Gkioxari, Kiana Ehsani, Ludwig Schmidt, and Ali Farhadi. Objaverse-xl: A universe of 10m+ 3d objects. arXiv preprint arXiv:2307.05663, 2023

    arXiv 2023

  6. [6]

    Objaverse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142–13153, 2023

    2023

  7. [7]

    Learning controllable content generators

    Sam Earle, Maria Edwards, Ahmed Khalifa, Philip Bontrager, and Julian Togelius. Learning controllable content generators. In 2021 IEEE Conference on Games (CoG), pages 1–9. IEEE, 2021

    work page 2021

  8. [8]

    Layoutgpt: Compositional visual planning and generation with large language models

    Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems, 36, 2024

    2024

Show all 72 references
  1. [9]

    3d-front: 3d furnished rooms with layouts and semantics

    Huan Fu, Bowen Cai, Lin Gao, Lingxiao Zhang, Cao Li, Qixun Zeng, Chengyue Sun, Yiyun Fei, Yu Zheng, Ying Li, Yi Liu, Peng Liu, Lin Ma, Le Weng, Xiaohang Hu, Xin Ma, Qian Qian, Rongfei Jia, Binqiang Zhao, and Hao Zhang. 3d-front: 3d furnished rooms with layouts and semantics. a...

    2011 arXiv

  2. [10]

    3d-future: 3d furniture shape with texture

    Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: 3d furniture shape with texture. arXiv preprint arXiv:2009.09633, 2020

    2009 arXiv

  3. [11]

    3d-front: 3d furnished rooms with layouts and semantics

    Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10933– 10...

    2021

  4. [12]

    Tetsphere splatting: Representing high-quality geometry with lagrangian volumetric meshes

    Minghao Guo, Bohan Wang, Kaiming He, and Wojciech Matusik. Tetsphere splatting: Representing high-quality geometry with lagrangian volumetric meshes. arXiv preprint arXiv:2405.20283, 2024

    2024 arXiv

  5. [13]

    Scaling up and distilling down: Language-guided robot skill acquisition

    Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guided robot skill acquisition. In Conference on Robot Learning, pages 3766–3777. PMLR, 2023

    2023

  6. [14]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. CoRR, abs/2104.08718, 2021. URL https://arxiv.org/abs/2104.08718

    2021 arXiv

  7. [15]

    Text2room: Extracting textured 3d meshes from 2d text-to-image models

    Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7909–7920, 2023

    2023

  8. [16]

    Repurposing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  9. [17]

    Pcgrl: Procedural content generation via reinforcement learning

    Ahmed Khalifa, Philip Bontrager, Sam Earle, and Julian Togelius. Pcgrl: Procedural content generation via reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 16, pages 95–101, 2020. 11

    2020

  10. [18]

    Dbscan: Past, present and future

    Kamran Khan, Saif Ur Rehman, Kamran Aziz, Simon Fong, and Sababady Sarasvady. Dbscan: Past, present and future. In The fifth international conference on the applications of digital information and web technologies (ICADIWT 2014), pages 232–238. IEEE, 2014

    2014

  11. [19]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023

    2023 arXiv

  12. [20]

    Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation

    Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In Conference on...

    2023

  13. [21]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    2023

  14. [22]

    Lucid- dreamer: Towards high-fidelity text-to-3d generation via interval score matching, 2023

    Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Lucid- dreamer: Towards high-fidelity text-to-3d generation via interval score matching, 2023

    2023

  15. [23]

    Towards language-guided interactive 3d generation: Llms as layout interpreter with generative feedback

    Yiqi Lin, Hao Wu, Ruichen Wang, Haonan Lu, Xiaodong Lin, Hui Xiong, and Lin Wang. Towards language-guided interactive 3d generation: Llms as layout interpreter with generative feedback. arXiv preprint arXiv:2305.15808, 2023

    2023 arXiv

  16. [24]

    Evaluating text-to-visual generation with image-to-text generation

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.01291, 2024

    2024 arXiv

  17. [25]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023

    2023 arXiv

  18. [26]

    M. Matl. PyRender. https://github.com/mmatl/pyrender, 2019

    2019

  19. [27]

    Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding

    Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogn...

    2019

  20. [28]

    Gpt-4v (ision) system card

    R OpenAI. Gpt-4v (ision) system card. Citekey: gptvision, 2023

    2023

  21. [29]

    Atiss: Autoregressive transformers for indoor scene synthesis

    Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. Advances in Neural Information Processing Systems, 34:12013–12026, 2021

    2021

  22. [30]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023

    2023

  23. [31]

    Barron, and Ben Mildenhall

    Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion, 2022

    2022

  24. [32]

    Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d

    Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. In Proceedings of the IEEE/CVF Conference on Computer...

    2024

  25. [33]

    Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai

    Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint...

    2021 arXiv

  26. [34]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. CoRR, abs/2102.12092, 2021. URL https://arxiv.org/abs/2102.12092

    2021 arXiv

  27. [35]

    Grounded sam: Assembling open-world models for diverse visual tasks, 2024

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024. 12

    2024

  28. [36]

    High-resolution image synthesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021

    2021

  29. [37]

    Controlroom3d: Room generation using semantic proxy rooms

    Jonas Schult, Sam Tsai, Lukas Höllein, Bichen Wu, Jialiang Wang, Chih-Yao Ma, Kunpeng Li, Xiaofang Wang, Felix Wimbauer, Zijian He, et al. Controlroom3d: Room generation using semantic proxy rooms. arXiv preprint arXiv:2312.05208, 2023

    2023 arXiv

  30. [38]

    Mvdream: Multi- view diffusion for 3d generation

    Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang. Mvdream: Multi- view diffusion for 3d generation. arXiv:2308.16512, 2023

    2023 arXiv

  31. [39]

    Diffuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis

    Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Diffuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthesis. arXiv preprint arXiv:2303.14207, 2023

    2023 arXiv

  32. [40]

    Dreamgaussian: Generative gaussian splatting for efficient 3d content creation

    Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653, 2023

    2023 arXiv

  33. [41]

    Gensim: Generating robotic simulation tasks via large language models

    Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models. In Arxiv, 2023

    2023

  34. [42]

    Robogen: Towards unleashing infinite data for automated robot learning via generative simulation

    Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint arXiv:2311.01455, 2023

    2023 arXiv

  35. [43]

    Pro- lificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation, 2023

    Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation, 2023

    2023

  36. [44]

    Visual room rearrange- ment

    Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrange- ment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5922–5931, 2021

    2021

  37. [45]

    Anyhome: Open-vocabulary generation of structured and textured 3d homes

    Zehao Wen, Zichen Liu, Srinath Sridhar, and Rao Fu. Anyhome: Open-vocabulary generation of structured and textured 3d homes. arXiv preprint arXiv:2312.06644, 2023

    2023 arXiv

  38. [46]

    Chang, Leonidas J

    Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part-based interactive environment. In The IEEE Conference on Computer Vision and P...

    2020

  39. [47]

    In- stantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruc- tion models

    Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. In- stantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruc- tion models. arXiv preprint arXiv:2404.07191, 2024

    2024 arXiv

  40. [48]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. arXiv preprint arXiv:2401.10891, 2024

    2024 arXiv

  41. [49]

    Physcene: Physically interactable 3d scene synthesis for embodied ai

    Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. Physcene: Physically interactable 3d scene synthesis for embodied ai. arXiv preprint arXiv:2404.09465, 2024

    2024 arXiv

  42. [50]

    Holodeck: Language guided generation of 3d embodied ai environments

    Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language guided generation of 3d embodied ai environments. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition...

    2024

  43. [51]

    Luminous: Indoor scene generation for embodied ai challenges

    Yizhou Zhao, Kaixiang Lin, Zhiwei Jia, Qiaozi Gao, Govind Thattai, Jesse Thomason, and Gaurav S Sukhatme. Luminous: Indoor scene generation for embodied ai challenges. arXiv preprint arXiv:2111.05527, 2021

    2021 arXiv

  44. [52]

    Luisarender: A high-performance rendering framework with layered and unified interfaces on stream architectures

    Shaokun Zheng, Zhiqian Zhou, Xin Chen, Difei Yan, Chuyan Zhang, Yuefeng Geng, Yan Gu, and Kun Xu. Luisarender: A high-performance rendering framework with layered and unified interfaces on stream architectures. ACM Transactions on Graphics (TOG), 41(6):1–19, 2022. 13 6 Appendi...

    2022

  45. [53]

    Description should not be too long

  46. [54]

    You should only give the result and no unnecessary words

  47. [55]

    Don’t describe the positional relationship between objects

  48. [56]

    Classification can only be **floor-object**, **wall-object**

  49. [57]

    other furniture

    Please pay attention to only large furniture like sofa, table, lamp, shelf, and ignore small objects like bottles or books. The prompt above is fed into GPT-4V along with an image generated by a 2D inpainting model. The prompt asks GPT-4V to recognize all objects in the inpain...

  50. [58]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The paper’s contributions, a zero-shot generative pipeline that creates diverse, complex, and realistic 3D interactive...

  51. [59]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper discuss the limitations in Section 5. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the...

  52. [60]

    Thus it does not provide full set of assumptions and proof

    Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 22 Answer: [NA] Justification: The paper does not provide theoretical results, it provides a practical 3D scene generatio...

  53. [61]

    All the information needed to reproduce the main experi- mental results are provides, thus, the paper is easy to be reproduced

    Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  54. [62]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The code will be made publi...

  55. [63]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: The ARCHITECT method that the...

  56. [64]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Error bars are not reported because it would be too...

  57. [65]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We provide experiments ...

  58. [66]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: The research conducted in the paper conform, in every respect, with the NeurIPS...

  59. [67]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the positive and negative social impacts thoroughly in Appendix F. Guidelines: • The answer NA...

  60. [68]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses...

  61. [69]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The creators or ...

  62. [70]

    Guidelines: • The answer NA means that the paper does not release new assets

    New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The new ARCHITECT method is well documented and the documentation is provided alongside the assets. Guidelines: • Th...

  63. [71]

    The participants are volunteers with no compensation

    Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  64. [72]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.