{"id":"ac2bdd95-767c-42f5-a115-9bd5c5904d18","arxiv_id":"2411.09823","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A diffusion-inpainting pipeline generates interactive 3D scenes by iteratively adding furniture and small objects to rendered views, then back-projecting them to 3D with rescaled depth.","lead":"Architect builds interactive 3D scenes by painting furniture into rendered images of empty rooms with a diffusion model, then lifting the painted images to 3D using depth estimation and object detection. It gives embodied AI researchers a way to generate diverse training environments from text without manual 3D modeling.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Affine depth rescale in §3.3 is unvalidated inside inpainted regions; a synthetic ground-truth 3D bbox test is needed before the back-projection claim is supported.","rationale":"The reader's weakest assumption and my concern coincide: the affine rescale is a global correction, while the error that matters is local and inside the inpainted mask. This is not a disagreement with consensus; it is an internal gap between the method and its claim. The rest of the pipeline—retrieval, search-based placement, and rendering—is described in enough detail and could be correct if the depth step works. I see no reason to move the verdict to reject, because the central mechanism is plausible and directly testable. The reader's CONDITIONAL verdict should stand, with the condition being a 3D ground-truth validation of the depth-to-bbox step before accepting the quantitative comparative claims. The paper's own limitation section mentions only furniture retrieval, not depth accuracy, which further supports requiring this validation.","tokens_in":19854,"tokens_out":4285,"duration_ms":47076,"concrete_test":"Using the same Pyrender/LuisaRender setup, render a room background with known camera and ground-truth depth, place a known 3D mesh at a known pose, render the composite, mask out the object, then run the exact Section 3.3 perception chain: Marigold depth, min/max affine rescale over unmasked pixels, back-projection, and SAM/DBSCAN bbox extraction. Compare predicted bbox center, extent, and floor-contact distance against ground truth across at least 50 objects spanning near/far distances and large/small furniture. If median translation error exceeds roughly 10 cm or bbox IoU falls below about 0.5, the affine-depth assumption fails for inpainted regions and the central 3D claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 assumes Marigold's monocular depth for freshly inpainted foreground lies in the same affine family as its depth for the known rendered background. The rescale uses only min/max and means over unmasked reference pixels, so it corrects a global scale and shift but cannot correct local geometry errors. If the diffusion model places an object at the wrong distance, scale, or floor contact, Marigold will produce a plausible but locally incorrect depth and the extracted 3D bounding box inherits that error. The paper provides no direct 2D-to-3D accuracy metric, no ground-truth bbox comparison, and no error bars (checklist item 7 explicitly says error bars are not reported). Tables 2 and 4 measure rendered-image text alignment and user preference, not spatial correctness. Table 3's 'Inpaint vs. Placed' similarity is a 2D image metric and can be high even when depth or placement is wrong. The small-object branch is especially fragile because reference pixels are taken from the non-masked parts of large furniture, where depth discontinuities are large. Thus the central claim—that hierarchical inpainting into a rendered background controls camera parameters and depth scale so accurate 3D back-projection is possible—is untested precisely where it is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Architect, a zero-shot pipeline that generates interactive 3D scenes by iteratively inpainting foreground objects into images rendered from an empty simulated room. Starting from a floor plan, an LLM proposes inpainting prompts, a diffusion inpainting model fills masked regions, and visual perception models (GPT-4V, Grounding-DINO, SAM) detect and segment objects. Monocular depth from Marigold is rescaled against known rendered depth in non-masked regions and then back-projected using known camera parameters to produce 3D bounding boxes, which are used to retrieve or generate 3D assets and place them in simulation. The pipeline operates hierarchically, first placing large furniture and then small objects, and is claimed to support text, floor-plan, and pre-arranged-scene inputs. The authors compare against Holodeck, Text2Room, and DiffuScene with image-text alignment metrics, a GPT-4o ranking, and a user study, and they demonstrate several embodied robot tasks in the generated scenes.","tokens_in":20207,"tokens_out":4067,"duration_ms":47102,"significance":"If the central claim holds, Architect is a significant systems contribution: it generates open-vocabulary, interactive, cluttered 3D scenes without training a scene model, leveraging the strong layout priors of pretrained 2D diffusion models and grounding them with known camera and depth information from simulation. The paper's strengths include a clear and modular pipeline, zero-shot generalization beyond household scenes (e.g., grocery stores, hospitals), flexible initialization from text or floor plans, and concrete demonstrations of downstream embodied tasks. The authors also report a user study and a set of quantitative comparisons. However, the quantitative evidence is almost entirely 2D image-text alignment or subjective preference; the load-bearing assertion of accurate 2D-to-3D lifting is not directly validated. The paper's own checklist explicitly acknowledges that error bars are not reported, and the depth-rescaling step in Section 3.3 is unchecked inside the inpainted regions where the generated objects actually live. Because the contribution's value depends on producing geometrically correct placements, this validation gap is central rather than cosmetic.","major_comments":[{"comment":"The central claim that inpainting into a rendered background controls camera parameters and depth scale so that images can be accurately back-projected is not validated inside the inpainted regions. Eq. (1) rescales Marigold's predicted depth with a single affine map computed from reference pixels in the non-masked background. This corrects a global scale and offset but cannot correct local geometric errors in the generated foreground, such as an object placed at the wrong distance, an incorrect floor contact, or an implausible object size. The paper provides no direct comparison of the resulting 3D bounding boxes against ground truth. I request a synthetic evaluation: generate scenes with known object placements, run the full pipeline, and report per-object center error, 3D IoU, and scale error, with separate breakdowns for the large-furniture and small-object branches. Without such an experiment, the contribution bullet claiming 'accurate 2D to 3D lifting' is unsupported.","section":"§1 and §3.3, Eq. (1)"},{"comment":"The quantitative evaluation in Tables 2 and 4 uses CLIPScore, BLIPScore, VQAScore, GPT-4o ranking, and a user study. These metrics measure 2D image-text alignment and holistic visual preference; they cannot detect a misplaced object, a wrong depth scale, a collision, or an incorrect bounding-box size. A method that produces photorealistic but geometrically wrong scenes could score well on all of these. The user-study 'Layout Correctness' is a subjective one-to-five score, not a spatial accuracy measurement. To support the paper's placement claims, the evaluation needs direct 3D metrics against known layouts, or at minimum quantitative comparisons of wall/floor contact, inter-object penetration, and bounding-box alignment with the inpainted images.","section":"§4, Tables 2 and 4"},{"comment":"The 'Inpaint vs. Placed' similarity in Table 3 measures the agreement between the inpainted image and a rendered image of the scene produced by the same pipeline. This is a consistency check, not an accuracy measure: both images can agree while sharing the same geometric errors, and a high 2D similarity can coexist with incorrect depth or placement. The 'LLM Placement VQScore' likewise compares one placement method against another without grounding in true positions. Additionally, the paper reports no error bars or confidence intervals anywhere; checklist item 7 explicitly states that error bars are not reported because of computational expense. Given that all quantitative claims are aggregate scores, the absence of variance information makes it impossible to assess whether reported differences are meaningful.","section":"Table 3"}],"minor_comments":[{"comment":"There are numerous typos and inconsistencies: 'hierachical' in the abstract, 'previos' in the Table 1 caption, 'devided' in Appendix A.1, and inconsistent spelling of 'DiffuScene'/'Diffscene' and 'GPT4v'/'GPT-4V'. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The 'predefined criterion' for filtering generated inpainted images is not specified. Since this filtering directly affects which images proceed to visual perception and placement, the threshold and the number of generated candidates per mask should be stated.","section":"§3.2"},{"comment":"The user study says all participants were volunteers without compensation, which is acceptable, but the paper should state how many scenes per method were rated, how the 23 scenes were sampled, and whether the raters were shown matched room captions; otherwise the reported averages are hard to interpret.","section":"§4 and Appendix D"},{"comment":"The claim that an 84-degree FOV yields approximately 95 percent of a square room visible from a corner-to-corner view should be justified with a formula or citation; the geometry is not obvious without knowing the room aspect ratio and camera height.","section":"Appendix A.4"},{"comment":"The caption 'Figure 6 right part' appears twice and is redundant; the text refers to 'Figure 6 right part' where 'Figure 6, right' would suffice. Also, the object-generation subsection would benefit from quantitative or at least more systematic qualitative evidence, as it currently reports only examples.","section":"§4.3 and Figure 6"},{"comment":"The asset-selection procedure uses scale similarity and CLIP feature similarity, but the relative weights or thresholds for combining them are not given; without these, the placement and retrieval behavior is not fully reproducible.","section":"Appendix A.2"}],"recommendation":"major_revision","confidential_remarks":"This is a solid systems paper with a plausible and useful pipeline, and I would be willing to accept it after the authors add a direct validation of the 2D-to-3D back-projection step. The missing geometric evaluation is the main risk: the paper's own contribution statement promises accurate 3D lifting, but the current experiments only establish realism and text-image alignment. I also noticed that the paper claims reproducibility in the checklist while leaving several pipeline parameters unspecified; this should be tightened during revision. I do not see grounds for rejection, because the core assumption is testable and the authors can address it with a synthetic ground-truth experiment within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Architect is worth a serious referee. The core idea is genuinely neat: instead of trying to recover camera parameters and depth scale from a fully generated image, the authors render an empty simulation scene with known camera and ground-truth depth, then inpaint the foreground into that image. The diffusion model sees background geometry and the inpainting constraint, so the result plausibly inherits the camera and depth scale. That combination—sim-rendered background plus hierarchical inpainting plus depth-rescaled back-projection—is new relative to the cited Text2Room, Holodeck, and DiffuScene baselines, and the paper positions it honestly. The qualitative results are impressive, the user study is real (115 volunteers, instructions included in the appendix), and the pipeline is described in enough detail to reimplement.\n\nThe soft spot is exactly where the stress-test note lands. Section 3.3 rescales Marigold's predicted depth with a global scale and shift fit to reference pixels outside the mask. That corrects a global affine ambiguity, but it cannot fix local geometry errors inside the inpainted region. If the diffusion model draws the sofa too close to the camera or floating off the floor, Marigold will produce a smooth but locally wrong depth, and the extracted 3D bounding box inherits the error. The paper provides no direct 2D-to-3D accuracy metric: no ground-truth bbox comparison, no error bars (the checklist admits this). Table 3's 'Inpaint vs. Placed' similarity is a 2D rendered-image metric; it can be high even when placements are geometrically wrong. The small-object branch is the fragile one, because the reference pixels come from non-masked parts of large furniture, where depth discontinuities are large.\n\nThat said, I don't think the paper collapses. The pipeline has enough error-tolerant machinery—DBSCAN outlier removal, retrieval with scale similarity, a constraint-based placement search rather than direct bbox-centering—that moderate depth errors may not be fatal. And the central mechanism is plausible: inpainting into a known-render background really does constrain the camera better than pure text-to-image generation. But 'plausible' is not 'shown.' The fix is straightforward and not expensive: render a known synthetic scene, run the full pipeline, and compare predicted object bounding boxes and floor contact to ground truth across, say, twenty rooms with error bars. That experiment would either confirm the affine-depth assumption or reveal where it breaks.\n\nThe citation pattern is fine; the relevant baselines are cited, and the claims about LLM spatial limitations are supported by their own comparisons. The paper is coherent and honest about its main limitation (dependence on a furniture database). Bottom line: send it to review. A good referee should ask for the 3D-accuracy experiment, but the idea is useful and the writeup is clear. I'd bring it to a reading group and would cite it if I worked on scene generation.","headline":"Architect is a clever, clearly-explained pipeline that controls camera and depth by inpainting into simulation-rendered backgrounds, but its load-bearing depth-rescale assumption is never validated against 3D ground truth.","tokens_in":20696,"tokens_out":2627,"would_cite":true,"duration_ms":26875,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A zero-training pipeline turns 2D inpainting into interactable 3D scene generation by anchoring generated objects to known rendered depth.","keywords":["3D scene generation","2D inpainting","diffusion models","monocular depth estimation","zero-shot generation","embodied AI","interactive environments","hierarchical generation"],"falsifier":"Render a known empty room, inpaint a single object at a known position and distance, then compare the rescaled predicted depth inside the inpainted mask to the ground-truth depth from the render; if the per-pixel error is large or grows with distance, the back-projection claim fails, and this measurement is not reported in the paper.","tokens_in":19706,"feed_emoji":"🏠","tokens_out":5967,"duration_ms":55573,"temperature":0.7,"pith_summary":"Architect proposes that a pre-trained 2D image inpainting model can serve as the spatial planner for building interactive 3D scenes, without training a scene generator. The key move is to render an empty room in simulation, where camera parameters and depth are known exactly, and then ask a diffusion model to inpaint furniture and small objects into that image. Because the known background depth anchors the generated foreground, the depth predicted for the inpainted objects can be rescaled and back-projected to obtain 3D bounding boxes. Repeated hierarchical inpainting from multiple views fills large furniture first and then small objects, producing complex, open-vocabulary environments for embodied AI and robotics. If the approach works, it turns internet-scale 2D image priors directly into interactable simulation scenes.","feed_headline":"Inpaint a rendered room, get an interactive 3D scene","feed_subtitle":"Architect anchors diffusion output to known depth so each painted object lands at a real 3D position.","key_machinery":"The central object is the controlled inpainted image: a simulation render of an empty scene with known camera parameters and ground-truth depth, used as the conditioning template for a latent diffusion inpainting model. The mechanism that carries the argument is the affine depth rescale: for each inpainted image, reference pixels are sampled from the non-masked background (all non-masked pixels for large furniture; pixels on the supporting furniture for small objects), and the monocular depth estimate is rescaled with a scale and offset computed from those reference pairs. This converts the scale-ambiguous predicted depth into the metric depth coordinate of the render, so back-projection with the known camera gives geometrically consistent 3D bounding boxes. The hierarchical loop of render, inpaint, perceive, place, and repeat is what extends a single view into a full multi-scale scene.","core_discovery":"The discovery is that a diffusion inpainting model can be made to inherit the geometry of a simulation-rendered background. Starting from a photo-realistic empty room rendered with known camera intrinsics and ground-truth depth, Architect inpaints a masked foreground region. The paper argues that the geometric cues visible in the background inform the inpainting model's camera parameters, so the generated image is effectively captured from the same camera; and because the known background depth provides reference pixels, the estimated depth of the inpainted objects can be rescaled affinely with a per-image scale and offset, then back-projected to 3D. From the inpainted image, off-the-shelf perception models name, detect, and segment the new objects, a monocular depth estimator supplies relative depth, and DBSCAN cleans the point clouds before axis-aligned bounding boxes are produced. Those boxes constrain placement of retrieved or generated assets, and the loop repeats at a finer scale for small objects inside or on top of furniture. The paper claims this yields diverse, realistic, interactive scenes from text, floor plans, or pre-arranged scenes without any layout training.","pith_inferences":["A direct validation of the affine-depth assumption would be the cheapest test: render a known scene, inpaint an object at a known location, and compare the rescaled predicted depth inside the mask against the ground-truth depth from the render; the paper reports similarity scores but not this error.","If the affine assumption holds only approximately, errors will compound with iteration, since each newly placed object is anchored to previously placed ones through the rendered scene; later small-object placements inherit the drift.","The method's dependence on an asset database for large furniture means diversity is bounded by retrieval even though the 2D prior is open-vocabulary; the paper's text-to-3D furniture pipeline is a step toward removing that bound.","The same render-empty-background and inpaint-foreground recipe could extend beyond rooms to streetscapes, stores, or outdoor scenes wherever a simulator can supply camera and depth, effectively turning the 2D model into a scene layout prior for arbitrary embodied domains."],"forward_implications":["Text-to-scene generation becomes a zero-shot retrieval-and-placement process: any room type expressible by an inpainting model can be turned into an interactive simulation without layout training data.","Scenes can be seeded from different starting points—text, floor plans, or already-arranged environments—and refined iteratively, since each inpainting step only needs a rendered view and a mask.","Small-object placement, which LLM-based scene designers struggle with, is inherited from the 2D model's spatial priors rather than from explicit rules.","Generated scenes support downstream embodied tasks: long-horizon robot manipulation and navigation trajectories can be synthesized on the resulting assets.","Scene editing becomes possible because the final state is a list of named objects with positions, scales, bounding boxes, and orientations that an LLM can call as APIs."],"supporting_citations":[{"why":"Marigold monocular depth estimator whose output is rescaled with reference background depth before back-projection.","marker":"[16]"},{"why":"SD-XL inpainting model that generates foreground objects conditioned on the rendered empty-scene image.","marker":"[30]"},{"why":"Grounded-SAM pipeline that detects and segments each inpainted object for later point-cloud extraction.","marker":"[35]"},{"why":"GPT-4V that names objects and supplies descriptions and inpainting prompts.","marker":"[28]"},{"why":"Holodeck, the baseline and source of the constraint-search placement procedure used to avoid collisions.","marker":"[50]"},{"why":"LuisaRender, the ray-tracing renderer that provides the photo-realistic empty-scene image and ground-truth depth.","marker":"[52]"},{"why":"Objaverse, the asset database from which large furniture is retrieved for placement.","marker":"[6]"}],"fun_headline_variants":["Depth-aware 2D inpainting builds interactive 3D scenes","Inpaint a room in 2D, then step inside in 3D","Hierarchical inpainting turns 2D images into explorable 3D","From 2D paint to 3D world: Architect's hierarchy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the depth predicted for a freshly inpainted object lines up with the known background depth after one simple rescale—that is, that the depth estimator sees both in the same coordinate system.","fun_headline_variants_meta":{"raw":{"variants":["Depth-aware 2D inpainting builds interactive 3D scenes","Inpaint a room in 2D, then step inside in 3D","Hierarchical inpainting turns 2D images into explorable 3D","From 2D paint to 3D world: Architect's hierarchy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000398,"raw_usage":{"total_tokens":2099,"prompt_tokens":979,"completion_tokens":1120,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":1036}},"tokens_in":595,"tokens_out":1120,"duration_ms":9957,"temperature":1.0,"reasoning_tokens":1036,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:16:36.510415+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render a known empty room, inpaint a single object at a known position and distance, then compare the rescaled predicted depth inside the inpainted mask to the ground-truth depth from the render; if the per-pixel error is large or grows with distance, the back-projection claim fails, and this measurement is not reported in the paper.","supporting_citations":[{"cited_title":"Grounded sam: Assembling open-world models for diverse visual tasks, 2024","cited_arxiv_id":null,"evidence_quote":"Grounded-SAM pipeline that detects and segments each inpainted object for later point-cloud extraction."},{"cited_title":"Holodeck: Language guided generation of 3d embodied ai environments","cited_arxiv_id":null,"evidence_quote":"Holodeck, the baseline and source of the constraint-search placement procedure used to avoid collisions."},{"cited_title":"Luisarender: A high-performance rendering framework with layered and unified interfaces on stream architectures","cited_arxiv_id":null,"evidence_quote":"LuisaRender, the ray-tracing renderer that provides the photo-realistic empty-scene image and ground-truth depth."}],"review_version":1}