Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SceneFactor generates 3D scenes from text, then lets users edit them by moving boxes.

desk verdict Factored text-to-layout-to-geometry diffusion with box-based editing is a real step forward, but the editing procedure is underspecified and the locality claim needs hard evidence. read the letter →

arxiv 2412.01801 v2 pith:U4JQKN7J submitted 2024-12-02 cs.CV

Alexey Bokhovkin , Quan Meng , Shubham Tulsiani , Angela Dai This is my paper · ORCID
classification cs.CV
keywords 3Dscenegenerationlatentdiffusiontext-guidedsemanticlayouteditingVQ-VAEoutpaintingindoorscenes
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SceneFactor tries to make large-scale 3D indoor scene generation from text both high-quality and locally editable. Its key move is to break generation into two stages: first synthesize a coarse semantic map of 3D boxes from the text, then synthesize geometry conditioned on that map. Because edits are performed on the box map, a user can add, remove, move, resize, or replace an object by manipulating two box corners, and only the edited region is re-synthesized. If the claim holds, this gives content creators a practical middle ground between writing a new text prompt and re-generating an entire scene.

What carries the argument

The load-bearing mechanism is the pair of aligned latent grids $f_S$ and $f_G$, produced by two 3D VQ-VAEs, together with the diffusion models defined over them. The semantic diffusion $\Psi_S$ generates $f_S$ from text; the geometric diffusion $\Psi_G$ generates $f_G$ from $f_S$ using convolutional attention with window size 3 so the geometry reads local semantic neighborhoods. Editing works by replacing the semantic box in the region $\mathcal{R}_S$, filling the corresponding region of $f_G$ with Gaussian noise, and denoising only that region while the rest of the grid is held fixed. Chunk-wise outpainting with overlapping sliding windows extends the scheme to arbitrary scene sizes.

What would settle it

Take a generated scene, edit one semantic box, fill only that box's geometric region with Gaussian noise, re-synthesize, and compare the decoded geometry outside the bounding box before and after the edit; if voxels outside the region change beyond the model's decoding noise, the locality assumption fails. A stronger check would compare the locally edited scene against a full re-synthesis with the same edited semantic map: the editing claim stands only if the two agree outside the edited box.

Watch

Extended reading notes

Core claim

The paper's central claim is that factoring text-to-scene generation through a proxy semantic space of 3D boxes makes 3D scene generation more tractable and editable. A semantic latent diffusion model consumes a text caption and produces a coarse box layout; a geometric latent diffusion model consumes that layout through spatial cross-attention and produces a truncated unsigned distance field for the geometry. The two latent grids are kept at equal resolution so they align exactly, and editing means changing boxes in the semantic map and re-running only the affected geometric region from Gaussian noise. The paper reports that this factored approach improves geometric quality and text adherence over direct text-to-geometry baselines, while enabling object addition, removal, replacement, rescaling, and movement without full-scene re-synthesis.

Load-bearing premise

The editing procedure assumes that re-synthesizing only the edited region of the geometric latent grid, while holding the rest fixed, produces geometry that is consistent with the unchanged surroundings.

Editorial extensions

If this is right

  • Users can add, remove, move, resize, or replace objects in a generated scene by editing two corners of a semantic box, with no full-scene re-synthesis.
  • Scene size is not fixed: chunk-by-chunk outpainting with overlapping windows yields arbitrarily sized rooms.
  • Decoupling layout from geometry lets the semantic stage align the scene to text before any geometric detail is committed, reducing floaters and incoherent arrangements.
  • Because the semantic map is explicit, edited results keep global consistency outside the edited region, provided the locality assumption in the edit procedure holds.
  • The same factored pipeline outperforms direct text-to-geometry diffusion baselines on geometric quality and text-consistency metrics in the reported experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The editing procedure's locality could be stress-tested by measuring whether geometry outside the edited bounding box changes when only $\mathcal{R}_S$ is re-noised; any measurable drift would indicate the effective receptive field leaks across the boundary.
  • A natural extension is to let the semantic stage produce open-vocabulary object categories instead of a closed 10-class vocabulary, which would broaden edit operations beyond categories seen in the training data.
  • The same factored semantic-geometry idea could apply to other structured outputs, such as outdoor scenes or articulated objects, wherever a coarse layout can mediate between language and dense geometry.
  • Linking text to semantic boxes opens a path to language-driven editing: a user could say 'make the sofa larger,' and a parser could perform the box manipulation automatically.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SceneFactor, a two-stage latent diffusion approach for text-guided 3D indoor scene generation. A semantic VQ-VAE encodes coarse semantic box layouts, and a text-conditioned diffusion model generates these layouts; a geometric VQ-VAE encodes truncated unsigned distance fields, and a second diffusion model synthesizes geometry conditioned on the semantic layout via spatial cross-attention with window-3 convolutional attention. Large scenes are produced by chunk-based outpainting, with RePaint-style inpainting equations for the known regions. The paper claims that this factorization enables localized editing: users manipulate semantic boxes (add, remove, replace, resize, move) and only the edited geometric region is re-synthesized. Experiments compare geometry quality and text consistency against several baselines and include a multi-part perceptual study.

Significance. If the claims hold, SceneFactor would be a notable advance: it targets editable large-scale text-to-3D-scene generation, and the factored semantic-proxy representation makes local editing conceptually simple. The paper's strengths include a clearly described two-stage pipeline, explicit RePaint-style outpainting equations (Eqs. 8-10), a large-scale training setup on 3D-FRONT/3D-FUTURE, and quantitative evaluations on geometry and text consistency, including a perceptual study. However, the central editing claim is currently not sufficiently supported: the exact diffusion-inpainting procedure for edits is underspecified, and the only direct evidence of edit locality is a small unary user study. The methodological issues in the editing evaluation are load-bearing because localized editing is the main stated contribution.

major comments (4)
  1. [Sec. 3.5] The editing protocol is underspecified in a way that directly affects the central claim. The text says to 'fill only R_S in the grid F_G with Gaussian noise and re-generate geometry for it,' but it does not state the starting timestep, the noise level of the known region at that timestep, or how the known region is resampled during the reverse process. If the reverse process starts at t=T with the known region held at clean latent values, the model input is a mixture of isotropic noise and clean signal, which is out-of-distribution and likely produces boundary artifacts; if it starts at an intermediate timestep, the known region must be noised at each step as in the paper's own outpainting equations (8)-(10). The manuscript should specify the algorithm precisely and verify that the known region is processed consistently with the diffusion training distribution.
  2. [Sec. 3.5 and Supplement Sec. 8] There is no quantitative measure of edit locality. The only evidence that the scene outside the editing region R_S remains consistent is a unary perceptual question answered by 21 participants. Because the geometric diffusion uses convolutional attention with window size 3 (Sec. 3.3), cross-boundary influence is mechanistically possible, so a numerical locality metric is needed — for example, Chamfer distance or 1-NNA between the geometry outside R_S before and after editing, or a comparison between localized editing and full re-synthesis. This evidence is necessary to support the claim of 'localized editing ... without requiring re-synthesis of the full scene.'
  3. [Tables 1, 3, 4, 9, 10] The reported geometry and text-consistency metrics are point estimates with no error bars, standard deviations, or significance tests. Since diffusion sampling is stochastic and the test set has only 250 scenes, the claimed improvements over baselines could be within run-to-run variation. The authors should report results over multiple seeds (or at least repeated sampling) and, where relevant, a significance test.
  4. [Sec. 4.2 and Supplement Sec. 8] The neural listener evaluation in Table 2 (and Table 8 in the supplement) uses the same BERT text encoder as SceneFactor for the language encoder. This can introduce a bias in favor of the proposed method, since the text representation is not encoder-independent. The CLIP score in Table 3 also uses the maximum over five rendered views per chunk; the paper should clarify whether the same rule is applied to all baselines and show sensitivity to the number and choice of views.
minor comments (5)
  1. [Sec. 3.5, 'Moving an object'] The text says the geometric features are translated 'from R1_S to R1_S' — presumably this should be 'from R1_S to R2_S'. Please fix the typo.
  2. [Supplement, Table 6 caption] The caption says 'as independent chunks (left) and as chunks of outpainted 3D scenes (right)', but the table contains only independent-chunk columns; adjust the caption to match the table.
  3. [Sec. 4.1 and Supplement Sec. 6] The paper reports 3 million crops for VQ-VAE training and a 6000/250 train/test scene split, but it would help to state how many chunks are used for training the diffusion models and how the 250 test scenes are chunked.
  4. [Sec. 3.3] The v-parameterization notation is confusing: v_t is defined as sqrt(alpha_t) epsilon_t - sqrt(1-alpha_t) x_t, while Eq. (5) defines x_t = sqrt(alpha_bar_t) x_0 + sqrt(1-alpha_bar_t) epsilon. Please clarify the exact relation and ensure subscripts (alpha_t vs alpha_bar_t) are consistent.
  5. [Sec. 4.3 and Fig. 5] The BlockFusion comparisons are generated unconditionally, while the other baselines and SceneFactor are text-conditioned. Since the main comparison is against conditional generation, please state explicitly whether all geometry metrics are computed in the same conditional setting, or discuss why an unconditional BlockFusion comparison is informative.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the factored diffusion derivation is self-contained, and the main claims are grounded in external benchmarks and ground-truth geometry.

full rationale

SceneFactor's derivation chain is not circular. The semantic and geometric latent spaces are learned with VQ-VAE reconstruction objectives (Eqs. 1-3), the semantic diffusion is trained with a text-conditioned latent diffusion objective (Eq. 6), and the geometry diffusion is trained with a semantic-conditioned objective (Eq. 7). The editing procedure in Sec. 3.5 takes user-specified semantic box manipulations and re-synthesizes the corresponding geometric latent region through the trained conditional model; this is an application of the model, not a prediction forced by a fitted input. The central generation claims are evaluated against external references: CLIP score uses a pretrained CLIP model, and the geometric metrics (MMD/COV/1-NNA) are computed against ground-truth point clouds. The only mild methodological caveat is that the neural listener evaluation (supplement Sec. 8) uses SceneFactor's text encoder as the listener's language encoder, which could modestly favor SceneFactor in Tab. 2; however, this is an evaluation-alignment concern rather than a circular derivation, and text adherence is also measured by independent CLIP scores. The editing procedure in Sec. 3.5 is underspecified regarding the inpainting schedule (starting timestep and noising of the known region), which leaves the locality claim unverified; this is a completeness and correctness concern, not a circularity. No self-citation chain, uniqueness theorem, or ansatz imported from prior author work is load-bearing. The paper therefore does not reduce any of its predictions to its own inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's central claims rest on standard diffusion/VQ-VAE machinery plus the domain assumption that a coarse semantic box map is a sufficient and locally compositional condition for high-fidelity scene geometry. The only hand-chosen free parameters that materially affect the claim are the semantic vocabulary size, chunk scales, and latent codebook sizes.

free parameters (3)
  • Semantic class count = 10 channels (free space, wall/floor, 8 object categories)
    The closed vocabulary of 10 semantic classes in the proxy map limits what can be generated; the paper acknowledges this in Limitations. It is a hand-chosen design tied to the 3D-FRONT dataset.
  • Chunk sizes = 2.7m and 5.4m crops, voxel 4.2cm
    Training chunk dimensions and voxel resolution are chosen manually and affect the scale and fidelity of generated scenes.
  • Codebook sizes = 8192 semantic, 32768 geometric VQ-VAE
    Latent codebook sizes are hand-picked hyperparameters for the VQ-VAE compression; they affect reconstruction fidelity and editability.
assumptions (4)
  • standard math DDPM forward/reverse process assumptions (Markovian Gaussian noise, learned reverse denoiser)
    The method follows Ho et al. 2020 and uses v-parameterization; standard background for diffusion models.
  • domain assumption VQ-VAE latent manifolds are smooth and compositional enough for diffusion and localized editing
    The paper relies on the learned discrete latent spaces producing meaningful spatial structure; stated in Sec. 3.2 as enabling 'smooth manifolds for efficient generation and editing'. If the latent geometry is not compositionally editable, box edits in S may not transfer cleanly to fG.
  • domain assumption The semantic box layout is a sufficient condition to determine high-fidelity geometry
    The geometry diffusion model maps from one-hot semantic chunks to geometry; this assumes the coarse 10-class layout carries enough information to reconstruct detailed geometry. The paper's 'w/o sem stage' ablation supports this, but it is still an assumption about the dataset and task.
  • domain assumption RePaint-style inpainting on latent grids produces globally consistent large scenes
    Chunk-by-chunk outpainting uses masked denoising (Eqs. 8-10); this assumes overlapping known latents anchor the unknown regions without introducing seams. The paper decodes the full geometric grid at once to avoid seams, but the underlying assumption remains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation." pith.science (2026). https://pith.science/paper/U4JQKN7J

@misc{pith2026241201801,
  author       = {Pith},
  title        = {Pith review of: SceneFactor: Factored Latent 3D Diffusion for Controllable 3D Scene Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U4JQKN7J}},
  note         = {Machine review of arXiv:2412.01801}
}
read the original abstract

We present SceneFactor, a diffusion-based approach for large-scale 3D scene generation that enables controllable generation and effortless editing. SceneFactor enables text-guided 3D scene synthesis through our factored diffusion formulation, leveraging latent semantic and geometric manifolds for generation of arbitrary-sized 3D scenes. While text input enables easy, controllable generation, text guidance remains imprecise for intuitive, localized editing and manipulation of the generated 3D scenes. Our factored semantic diffusion generates a proxy semantic space composed of semantic 3D boxes that enables controllable editing of generated scenes by adding, removing, changing the size of the semantic 3D proxy boxes that guides high-fidelity, consistent 3D geometric editing. Extensive experiments demonstrate that our approach enables high-fidelity 3D scene synthesis with effective controllable editing through our factored diffusion approach.

Figures

Figures reproduced from arXiv: 2412.01801 by the authors.

Figure 1
Figure 1. SceneFactor factors the complex task of text-guided 3D scene generation into forming a coarse semantic structure, followed by [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Method overview. We formulate text-guided 3D scene generation as a factored diffusion process, first generating a coarse [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Chunk-based 3D scene generation. Left: Chunks for a [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Scene editing. SceneFactor enables seamless localized editing through easy manipulation of the 3D semantic box map. We [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparisons to state-of-the-art diffusion-based 3D scene generative approaches BlockFusion [ [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison with state of the art on text-guided scene chunk generation using Qwen1.5 captions. In comparison with [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Additional qualitative scene editing results. Generated scenes and their corresponding semantic maps are shown in the top row, [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Perceptual study of the quality of text-guided 3D indoor scene generation and editing. (a) Unary study on perceptual geometric [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Additional qualitative comparisons for scene generation in comparison with SDFusion [ [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Additional qualitative comparisons for scene generation in comparison with SDFusion [ [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Additional qualitative comparisons to state-of-the-art diffusion-based 3D generative approaches PVD [ [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Additional qualitative comparisons to state-of-the-art diffusion-based 3D generative approaches PVD [ [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EditVerse3D: High-Quality 3D Object Editing with Region-Aware Learning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An end-to-end 3D editing framework achieves high-fidelity local edits from coarse bounding boxes and 2D image prompts using region-aware loss reweighting and a large-scale parts-derived training dataset.

Reference graph

Works this paper leans on

86 extracted references · 54 canonical work pages · cited by 1 Pith paper

  1. [1]

    Shapeglot: Learning language for shape differentiation

    Panos Achlioptas, Judy Fan, Robert Hawkins, Noah Good- man, and Leonidas J Guibas. Shapeglot: Learning language for shape differentiation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8938– 8947, 2019. 7

    2019

  2. [2]

    Rio Aguina-Kang, Maxim Gumin, Do Heon Han, Stewart Morris, Seung Jean Yoo, Aditya Ganeshan, R. K. Jones, Qi- uhong Anna Wei, Kailiang Fu, and Daniel Ritchie. Open- universe indoor scene generation using llm program synthe- sis and uncurated object databases. ArXiv, abs/2403.09675,

    arXiv

  3. [3]

    Blended diffusion for text-driven editing of natural images

    Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18208–18218, 2022. 2

    2022

  4. [4]

    Cc3d: Layout-conditioned generation of compositional 3d scenes

    Sherwin Bahmani, Jeong Joon Park, Despoina Paschalidou, Xingguang Yan, Gordon Wetzstein, Leonidas Guibas, and Andrea Tagliasacchi. Cc3d: Layout-conditioned generation of compositional 3d scenes. In 2023 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 7137– 7147, 2023. 3

    2023

  5. [5]

    Text2live: Text-driven layered image and video editing

    Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kas- ten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In European Conference on Computer Vision, pages 707–723. Springer, 2022. 2

    2022

  6. [6]

    Chan, Koki Nagano, Matthew A

    Eric R. Chan, Koki Nagano, Matthew A. Chan, Alexan- der W. Bergman, Jeong Joon Park, Axel Levy, Miika Ait- tala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. GeNVS: Generative novel view synthesis with 3D-aware dif- fusion models. In arXiv, 2023. 2

    2023

  7. [7]

    Shapenet: An information-rich 3d model repository

    Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015. 2

    arXiv 2015

  8. [8]

    Learning graph variational au- toencoders with constraints and structured priors for condi- tional indoor 3d scene generation

    Aditya Chattopadhyay, Xi Zhang, David Paul Wipf, Himan- shu Arora, and Ren ´e Vidal. Learning graph variational au- toencoders with constraints and structured priors for condi- tional indoor 3d scene generation. In 2023 IEEE/CVF Win- ter Conference on Applications of Computer Vision (WACV), pages 785–794, 2023. 3

    2023

Show all 86 references
  1. [9]

    Learning generative models of 3d structures

    Siddhartha Chaudhuri, Daniel Ritchie, Jiajun Wu, Kai Xu, and Hao Zhang. Learning generative models of 3d structures. Computer Graphics Forum, 39(2):643–666, 2020. 2

    2020

  2. [10]

    Fan- tasia3d: Disentangling geometry and appearance for high- quality text-to-3d content creation

    Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fan- tasia3d: Disentangling geometry and appearance for high- quality text-to-3d content creation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 2

    2023

  3. [11]

    SDFusion: Multimodal 3d shape completion, reconstruction, and generation

    Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexan- der G Schwing, and Liang-Yan Gui. SDFusion: Multimodal 3d shape completion, reconstruction, and generation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4456–4465, 2023. 2, 3...

    2023

  4. [12]

    Diffusion-sdf: Conditional generative modeling of signed distance func- tions

    Gene Chou, Yuval Bahat, and Felix Heide. Diffusion-sdf: Conditional generative modeling of signed distance func- tions. 2023. 2, 5

    2023

  5. [13]

    ProcTHOR: Large-Scale Embodied AI Using Procedural Generation

    Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, and Roozbeh Mottaghi. ProcTHOR: Large-Scale Embodied AI Using Procedural Generation. In NeurIPS, 2022. Outstanding Paper Award. 3

    2022

  6. [14]

    Turboedit: Text-based image editing using few-step diffusion models, 2024

    Gilad Deutch, Rinon Gal, Daniel Garibi, Or Patashnik, and Daniel Cohen-Or. Turboedit: Text-based image editing using few-step diffusion models, 2024. 2

    2024

  7. [15]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In North American Chapter of the Association for Computational Linguistics ,

  8. [16]

    Graph-to-3d: End-to-end generation and ma- nipulation of 3d scenes using scene graphs

    Helisa Dhamo, Fabian Manhardt, Nassir Navab, and Fed- erico Tombari. Graph-to-3d: End-to-end generation and ma- nipulation of 3d scenes using scene graphs. In IEEE Inter- national Conference on Computer Vision (ICCV), 2021. 3

    2021

  9. [17]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. 2021. 5

    2021

  10. [18]

    Hyperdiffusion: Generating implicit neural fields with weight-space diffusion

    Ziya Erkoc ¸, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generating implicit neural fields with weight-space diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14300–14310, 2023. 2

    2023

  11. [19]

    Ctrl- room: Controllable text-to-3d room meshes generation with layout constraints

    Chuan Fang, Xiaotao Hu, Kunming Luo, and Ping Tan. Ctrl- room: Controllable text-to-3d room meshes generation with layout constraints. arXiv preprint arXiv:2310.03602, 2023. 3

    2023

  12. [20]

    Scenescape: Text-driven consistent scene generation

    Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel. Scenescape: Text-driven consistent scene generation. arXiv preprint arXiv:2302.01133, 2023. 2

    2023 arXiv

  13. [21]

    3d-front: 3d furnished rooms with layouts and semantics

    Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Bin- qiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 10933–10942,

  14. [22]

    3d-future: 3d fur- niture shape with texture

    Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: 3d fur- niture shape with texture. International Journal of Computer Vision, 129:3313–3337, 2021. 6, 1 9

    2021

  15. [23]

    Scenehgn: Hierarchical graph net- works for 3d indoor scene generation with fine-grained ge- ometry

    Lin Gao, Jia-Mu Sun, Kaichun Mo, Yu-Kun Lai, Leonidas Guibas, and Jie Yang. Scenehgn: Hierarchical graph net- works for 3d indoor scene generation with fine-grained ge- ometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP:1–18, 2023. 3

    2023

  16. [24]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. 5

    2016 arXiv

  17. [25]

    Jonathan Ho, Ajay Jain, and P. Abbeel. Denoising diffusion probabilistic models. ArXiv, abs/2006.11239, 2020. 5

    2006 arXiv

  18. [26]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2

    2020

  19. [27]

    Long short-term memory

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural computation, 9:1735–80, 1997. 4

    1997

  20. [28]

    Text2room: Extracting textured 3d meshes from 2d text-to-image models

    Lukas H ¨ollein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner. Text2room: Extracting textured 3d meshes from 2d text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7909–7920, 2023. 2, 7, 8, 3, 4, 5

    2023

  21. [29]

    Robust water- tight manifold surface generation method for shapenet mod- els

    Jingwei Huang, Hao Su, and Leonidas Guibas. Robust water- tight manifold surface generation method for shapenet mod- els. arXiv preprint arXiv:1802.01698, 2018. 1

    2018 arXiv

  22. [30]

    Diffindscene: Diffusion-based high-quality 3d indoor scene generation

    Xiaoliang Ju, Zhaoyang Huang, Yijin Li, Guofeng Zhang, Yu Qiao, and Hongsheng Li. Diffindscene: Diffusion-based high-quality 3d indoor scene generation. 2023. 3

    2023

  23. [31]

    Shap-e: Generating condi- tional 3d implicit functions, 2023

    Heewoo Jun and Alex Nichol. Shap-e: Generating condi- tional 3d implicit functions, 2023. 3

    2023

  24. [32]

    Clip-mesh: Generating textured meshes from text using pretrained image-text models

    Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Popa Tiberiu. Clip-mesh: Generating textured meshes from text using pretrained image-text models. SIGGRAPH Asia 2022 Conference Papers, 2022. 2

    2022

  25. [33]

    Adam: A method for stochastic optimization

    Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , San Diega, CA, USA,

  26. [34]

    Semcity: Semantic scene genera- tion with triplane diffusion

    Jumin Lee, Sebin Lee, Changho Jo, Woobin Im, Juhyeong Seon, and Sung-Eui Yoon. Semcity: Semantic scene genera- tion with triplane diffusion. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition ,

  27. [35]

    Grains: Generative re- cursive autoencoders for indoor scenes

    Manyi Li, Akshay Gadi Patil, Kai Xu, Siddhartha Chaudhuri, Owais Khan, Ariel Shamir, Changhe Tu, Baoquan Chen, Daniel Cohen-Or, and Hao Zhang. Grains: Generative re- cursive autoencoders for indoor scenes. ACM Transactions on Graphics, 37, 2018. 3

    2018

  28. [36]

    Diffusion- sdf: Text-to-shape via voxelized diffusion

    Muheng Li, Yueqi Duan, Jie Zhou, and Jiwen Lu. Diffusion- sdf: Text-to-shape via voxelized diffusion. In CVPR, 2023. 2, 3

    2023

  29. [37]

    Editvae: Unsupervised parts-aware controllable 3d point cloud shape generation

    Shidi Li, Miaomiao Liu, and Christian Walder. Editvae: Unsupervised parts-aware controllable 3d point cloud shape generation. Proceedings of the AAAI Conference on Artificial Intelligence, 36:1386–1394, 2022. 3

    2022

  30. [38]

    Magic3d: High-resolution text-to-3d content creation

    Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2023. 2

    2023

  31. [39]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2017. 7, 5

    2017

  32. [40]

    Repaint: Inpainting using denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022. 2, 5

    2022

  33. [41]

    AutoSDF: Shape priors for 3d completion, reconstruction and generation

    Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shub- ham Tulsiani. AutoSDF: Shape priors for 3d completion, reconstruction and generation. In CVPR, 2022. 2, 3

    2022

  34. [42]

    T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453, 2023. 2

    2023 arXiv

  35. [43]

    Difffacto: Con- trollable part-based 3d point cloud generation with cross dif- fusion

    Kiyohiro Nakayama, Mikaela Angelina Uy, Jiahui Huang, Shi-Min Hu, Ke Li, and Leonidas Guibas. Difffacto: Con- trollable part-based 3d point cloud generation with cross dif- fusion. In International Conference on Computer Vision (ICCV), 2023. 3

    2023

  36. [44]

    Polygen: An autoregressive generative model of 3d meshes

    Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International conference on machine learning, pages 7220–7229. PMLR, 2020. 2

    2020

  37. [45]

    Learning 3d scene priors with 2d supervision

    Yinyu Nie, Angela Dai, Xiaoguang Han, and Matthias Nießner. Learning 3d scene priors with 2d supervision. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 792–802, 2023. 3

    2023

  38. [46]

    Atiss: Autoregres- sive transformers for indoor scene synthesis

    Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregres- sive transformers for indoor scene synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 3, 4, 5

    2021

  39. [47]

    Advances in data-driven analysis and syn- thesis of 3d indoor scenes

    Akshay Patil, Supriya Patil, Manyi Li, Matthew Fisher, and Manolis Savva. Advances in data-driven analysis and syn- thesis of 3d indoor scenes. Computer Graphics Forum, 43,

  40. [48]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  41. [49]

    Barron, and Ben Milden- hall

    Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Milden- hall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv,

  42. [50]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of th...

    2021

  43. [51]

    Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies

    Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hierarchies. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE ,

  44. [52]

    High-resolution image syn- thesis with latent diffusion models, 2021

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models, 2021. 2

    2021

  45. [53]

    Geodiffuser: Geometry-based image editing with diffusion models, 2024

    Rahul Sajnani, Jeroen Vanbaar, Jie Min, Kapil Katyal, and Srinath Sridhar. Geodiffuser: Geometry-based image editing with diffusion models, 2024. 2

    2024

  46. [54]

    Controlroom3d: Room generation using semantic proxy rooms

    Jonas Schult, Sam Tsai, Lukas H ¨ollein, Bichen Wu, Jialiang Wang, Chih-Yao Ma, Kunpeng Li, Xiaofang Wang, Felix Wimbauer, Zijian He, Peizhao Zhang, Bastian Leibe, Peter Vajda, and Ji Hou. Controlroom3d: Room generation using semantic proxy rooms. In IEEE Conference on Compute...

    2024

  47. [55]

    3d-aware indoor scene synthesis with depth priors

    Zifan Shi, Yujun Shen, Jiapeng Zhu, Dit-Yan Yeung, and Qifeng Chen. 3d-aware indoor scene synthesis with depth priors. 2022. 3

    2022

  48. [56]

    3d neural field generation using triplane diffusion, 2023

    J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Ji- ajun Wu, , and Gordon Wetzstein. 3d neural field generation using triplane diffusion, 2023. 2, 7, 8, 1, 4, 5, 9

    2023

  49. [57]

    Meshgpt: Generating triangle meshes with decoder-only transformers

    Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias Nießner. Meshgpt: Generating triangle meshes with decoder-only transformers. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, 2024. 2

    2024

  50. [58]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 2

    2015

  51. [59]

    Roomdreamer: Text-driven 3d indoor scene synthesis with coherent geome- try and texture, 2023

    Liangchen Song, Liangliang Cao, Hongyu Xu, Kai Kang, Feng Tang, Junsong, Yuan, and Yang Zhao. Roomdreamer: Text-driven 3d indoor scene synthesis with coherent geome- try and texture, 2023. 2

    2023

  52. [60]

    Dual diffusion implicit bridges for image-to-image transla- tion

    Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image transla- tion. In International Conference on Learning Representa- tions, 2023. 2

    2023

  53. [61]

    Diffuscene: Denoising dif- fusion models for gerative indoor scene synthesis

    Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Jus- tus Thies, and Matthias Nießner. Diffuscene: Denoising dif- fusion models for gerative indoor scene synthesis. In Pro- ceedings of the ieee/cvf conference on computer vision and pattern recognition, 2024. 3

    2024

  54. [62]

    Introducing qwen1.5, 2024

    Qwen Team. Introducing qwen1.5, 2024. 7, 2, 4

    2024

  55. [63]

    Generating part-aware editable 3d shapes without 3d supervision

    Konstantinos Tertikas, Despoina Paschalidou, Boxiao Pan, Jeong Joon Park, Mikaela Angelina Uy, Ioannis Emiris, Yan- nis Avrithis, and Leonidas Guibas. Generating part-aware editable 3d shapes without 3d supervision. In Proceedings IEEE Conf. on Computer Vision and Pattern Reco...

    2023

  56. [64]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 4

    2017

  57. [65]

    Deep convolutional priors for indoor scene synthe- sis

    Kai Wang, Manolis Savva, Angel X Chang, and Daniel Ritchie. Deep convolutional priors for indoor scene synthe- sis. ACM Transactions on Graphics (TOG), 37(4):70, 2018. 3

    2018

  58. [66]

    Dual octree graph networks for learning adaptive volumetric shape rep- resentations

    Peng-Shuai Wang, Yang Liu, and Xin Tong. Dual octree graph networks for learning adaptive volumetric shape rep- resentations. ACM Transactions on Graphics (TOG), 41:1 – 15, 2022. 1

    2022

  59. [67]

    Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distilla- tion

    Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distilla- tion. arXiv preprint arXiv:2305.16213, 2023. 2

    2023 arXiv

  60. [68]

    Lego-net: Learning regular rearrangements of ob- jects in rooms

    Qiuhong Anna Wei, Sijie Ding, Jeong Joon Park, Rahul Sajnani, Adrien Poulenard, Srinath Sridhar, and Leonidas Guibas. Lego-net: Learning regular rearrangements of ob- jects in rooms. arXiv preprint arXiv:2301.09629, 2023. 3

    2023 arXiv

  61. [69]

    Tenenbaum

    Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Joshua B. Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. In Neural Information Processing Systems, 2016. 2

    2016

  62. [70]

    Blockfusion: Expand- able 3d scene generation using latent tri-plane extrapolation

    Zhennan Wu, Yang Li, and Han Yan. Blockfusion: Expand- able 3d scene generation using latent tri-plane extrapolation. https://synthical.com/article/66e48646- f127-4f88-bc78-f293758c0986 , 2024. 3, 7, 8, 1, 2, 4, 5, 6, 9

    2024

  63. [71]

    Empirical evaluation of rectified activations in convolutional network

    Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network

  64. [72]

    Frankenstein: Gen- erating semantic-compositional 3d scenes in one tri-plane

    Han Yan, Yang Li, Zhennan Wu, Shenzhou Chen, Weixuan Sun, Taizhang Shang, Weizhe Liu, Tian Chen, Xiaqiang Dai, Chao Ma, Hongdong Li, and Pan Ji. Frankenstein: Gen- erating semantic-compositional 3d scenes in one tri-plane. ArXiv, abs/2403.16210, 2024. 3

    2024 arXiv

  65. [73]

    Mitra, Dani Lischin- ski, Danny Cohen-Or, and Hui Huang

    Xingguang Yan, Liqiang Lin, Niloy J. Mitra, Dani Lischin- ski, Danny Cohen-Or, and Hui Huang. Shapeformer: Transformer-based shape completion via sparse representa- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2022. 2, 3

    2022

  66. [74]

    G. Yang, X. Huang, Z. Hao, M. Liu, S. Belongie, and B. Hariharan. Pointflow: 3d point cloud generation with con- tinuous normalizing flows. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4540–4549, Los Alamitos, CA, USA, 2019. IEEE Computer Society. 7

    2019

  67. [75]

    Holodeck: Language guided generation of 3d embodied ai environ- ments

    Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Al- varo Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, Chris Callison-Burch, Mark Yatskar, Aniruddha Kembhavi, and Christopher Clark. Holodeck: Language guided generation of 3d embodied ai environ- ...

    2023 arXiv

  68. [76]

    Commonscenes: Generating commonsense 3d indoor scenes with scene graph diffusion

    Guangyao Zhai, Evin Pınar ¨Ornek, Shun-Cheng Wu, Yan Di, Federico Tombari, Nassir Navab, and Benjamin Busam. Commonscenes: Generating commonsense 3d indoor scenes with scene graph diffusion. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 3

    2023

  69. [77]

    Text2nerf: Text-driven 3d scene generation with neural radiance fields

    Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao. Text2nerf: Text-driven 3d scene generation with neural radiance fields. arXiv preprint arXiv:2305.11588, 2023. 2

    2023 arXiv

  70. [78]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2

    2023

  71. [79]

    Fast 3d indoor 11 scene synthesis by learning spatial relation priors of objects

    Song-Hai Zhang, Shao-Kui Zhang, Wei-Yu Xie, Cheng- Yang Luo, Yong-Liang Yang, and Hongbo Fu. Fast 3d indoor 11 scene synthesis by learning spatial relation priors of objects. IEEE Transactions on Visualization and Computer Graph- ics, 28(9):3082–3092, 2022. 3

    2022

  72. [80]

    Michelangelo: Conditional 3d shape generation based on shape-image-text aligned latent representation

    Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng, Rui Wang, Pei Cheng, BIN FU, Tao Chen, Gang YU, and Shenghua Gao. Michelangelo: Conditional 3d shape generation based on shape-image-text aligned latent representation. In Thirty- seventh Conference on Neural Information Processing ...

    2023

  73. [81]

    Locally attentional sdf diffusion for controllable 3d shape generation

    Xin-Yang Zheng, Hao Pan, Peng-Shuai Wang, Xin Tong, Yang Liu, and Heung-Yeung Shum. Locally attentional sdf diffusion for controllable 3d shape generation. ACM Trans- actions on Graphics (SIGGRAPH), 42(4), 2023. 3

    2023

  74. [82]

    3d shape generation and completion through point-voxel diffusion

    Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision (ICCV) , pages 5826–5835, 2021. 2, 7, 8, 1, 4, 5, 9 12 SceneFactor: Factored Latent 3D Diffusio...

    2021

  75. [83]

    To make 3D-FRONT [21] data suitable for training and testing, we first combine 3D furniture and 3D scene meshes using 3D-FRONT annotation

    Data Processing Geometry. To make 3D-FRONT [21] data suitable for training and testing, we first combine 3D furniture and 3D scene meshes using 3D-FRONT annotation. 3D- FUTURE [22] models are preliminarily converted into high-quality watertight meshes using the Manifold [29] a...

  76. [84]

    Additional Results Additional Comparison to Diffusion-based Methods. Fig. 6, 11 and 12 show additional qualitative compar- isons with state-of-the-art baselines on scene chunk gener- ation using synthetic and Qwen-refined captions. PVD [82] model uses explicit point cloud diff...

  77. [85]

    Baseline Evaluation Setup Metrics. Following the works for 3D shape generation, we use the following metrics on point clouds extracted from mesh surfaces: MMD(Sg, Sr) = 1 |Sr| X Y ∈Sr min X∈Sg D(X, Y), COV(Sg, Sr) = |{argminY ∈Sr D(X, Y)|X ∈ Sg}| |Sr| , 1-NNA(Sg, Sr) = P X∈Sg ...

  78. [86]

    Implementation Details Our method is implemented using PyTorch. Semantic and geometric VQ-V AE models are trained with an Adam [33] 4 Method Independent chunks MMD↓ COV↑ 1-NNA (0.5) CD EMD CD EMD CD EMD w/o latent 0.263 0.473 0.335 0.344 0.784 0.784 Ours 0.222 0.458 0.495 0.49...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.