{"id":"791e07a6-1bdb-4c6c-9a0c-05dedd79a2c4","arxiv_id":"2412.03558","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MIDI extends pre-trained image-to-3D object generators to multi-instance diffusion with a multi-instance attention mechanism, producing spatially coherent 3D scenes from a single image in one pass.","lead":"MIDI turns a single photo into a 3D scene by generating all visible objects at once with a diffusion model, instead of building them one by one. The approach could make 3D scene creation for games, film, and augmented reality faster and more consistent.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never explains how the frozen object-centric VAE can encode/decode instances in a shared global coordinate frame, yet the central claim of simultaneous spatially coherent generation depends on exactly this.","rationale":"The reader's weakest assumption correctly identifies the shared-coordinate VAE issue, and my reading agrees with the conditional verdict. The concern is reinforced by the paper's own limitation statement (Sec. 10) that generation occurs in a global normalized space, which makes the missing encoding/decoding mechanism more concrete rather than less. However, the paper provides substantial quantitative and qualitative evidence, including ablations, so the correct response is to require clarification and verification rather than reject the work. I would keep the CONDITIONAL verdict and ask the authors to specify exactly how ground-truth latents are constructed for scene instances, whether the VAE is frozen or fine-tuned, how object positions are represented in the latent, and to provide the probe described above. The absence of released code and error bars further supports a conditional rather than unconditional acceptance, but does not change the central recommendation.","tokens_in":16670,"tokens_out":5320,"duration_ms":58731,"concrete_test":"Use the frozen VAE to encode and decode ground-truth 3D-Front instances in global scene coordinates: sample 20 test scenes, extract each instance's point cloud with normals in the normalized global frame, compute z_i = E(x_i^global), then decode with D(z_i) at the same global query points, and measure per-instance F-score and bounding-box IoU against the GT object in scene coordinates. If reconstructed objects are not correctly localized (e.g., centered at the origin or severely deformed), the direct-combination assumption is falsified and an undeclared pose/canonicalization mechanism must exist. If the authors instead used canonical latents plus a pose head, that head and its loss must be specified and evaluated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the pre-trained, object-centric VAE can encode and decode multiple instances in a shared global coordinate frame, so that denoised latents can be 'decoded and directly combined' into a scene (Sec. 4.1). The base VAE is described (Sec. 3 and supp. Sec. 7) as compressing point positions and normals and decoding to SDF values, trained on single objects in canonical normalized space. The paper never specifies how a scene-coordinate instance is encoded into a latent compatible with this decoder, nor how the decoder knows where to place the object in the scene. Sec. 10 admits generation happens in a normalized global space from -1 to 1, which confirms global-coordinate latents but does not explain how a VAE trained on centered objects produces valid SDFs at translated positions. If each z_i is canonical, pose information is absent and combining requires an undeclared layout step; if each z_i is global, the frozen VAE is out-of-distribution. Eq. (3) requires ground-truth latents z_i for scene instances, but the source and coordinate frame of these targets is unspecified. This gap is load-bearing because the central claim of simultaneous spatially coherent generation collapses if the latent representation cannot carry both shape and placement through the frozen VAE.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MIDI, a method for generating a 3D scene composed of multiple object instances from a single scene image in one diffusion pass. MIDI extends a pre-trained image-to-3D object diffusion model by denoising the latents of all instances simultaneously with a shared denoising network, introducing multi-instance attention layers so that each instance's tokens attend to tokens of all other instances, and conditioning on local object images, masks, and a global scene image. The model is fine-tuned on 3D-Front with mixed training on Objaverse, using a rectified-flow loss extended to multiple instances. The paper reports state-of-the-art quantitative results on 3D-Front and BlendSwap, qualitative results on real-world and stylized images, and an ablation study over the number of multi-instance attention layers, global conditioning, and mixed single-object training.","tokens_in":16961,"tokens_out":3693,"duration_ms":36304,"significance":"If the central claim holds, MIDI is a meaningful advance: it replaces multi-stage object-by-object scene generation with a single simultaneous multi-instance diffusion process, leveraging strong object priors and a novel cross-instance attention mechanism. The paper includes a clear ablation isolating the multi-instance attention contribution, reports runtime, and demonstrates qualitative generalization to real and stylized images. A notable strength is the explicit use of single-object data as regularization to preserve the base model's generalization. However, the method rests on an underspecified assumption about how the frozen object-centric VAE represents and decodes instances in a shared global coordinate frame, and the quantitative evaluation lacks uncertainty estimates and uses an unreleased base model. These issues are load-bearing for the central claim of spatially coherent simultaneous generation and for reproducibility, respectively.","major_comments":[{"comment":"The paper never specifies how the pre-trained single-object VAE encodes and decodes multiple instances in a shared global scene coordinate system, even though this is essential to the claim that denoised latents can be 'decoded and directly combined' into a coherent scene. The loss in Eq. (3) requires ground-truth latents {z_i0} for scene instances, but the source and coordinate frame of these latents are undefined. If each z_i0 is a canonical object-centric latent, it carries shape but no pose, and the 'directly combined' statement in Sec. 4.1 is unsupported without a hidden layout step. If each z_i0 is in global scene coordinates, then the frozen VAE—trained on canonical normalized objects as described in Sec. 3 and Supp. Sec. 7—is out-of-distribution, because point positions at arbitrary scene translations are far from its training distribution. The limitation section (Sec. 10) confirms that generation happens in a normalized global space from -1 to 1, but it does not explain how the decoder maps a global-coordinate latent back to an SDF field at the correct scene position, nor how ground-truth global latents are produced for training. This gap is load-bearing: the central claim of simultaneous spatially coherent generation collapses if the latent representation cannot carry both shape and placement through the frozen VAE. The manuscript must specify the encoding of scene instances during training, the coordinate frame of the latents, and the decoding/placement procedure.","section":"Sec. 4.1, Eq. (3); Sec. 10; Supp. Sec. 7"},{"comment":"The quantitative comparison in Table 1 reports a single number per metric with no error bars, no variance over multiple runs or seeds, and no statistical significance tests. Given that the claimed improvements are large (e.g., IoU-B 0.518 vs. 0.392 on 3D-Front), the absence of any uncertainty estimate makes it impossible to judge whether the margin is stable. In addition, the real-world evaluation in Sec. 5.3 is qualitative only, using 10 selected scenes, so the paper's claim of 'strong generalization' to real images is not quantitatively supported. The authors should provide error bars or multiple-seed results for the main tables, and, if possible, quantitative metrics on at least a subset of the real-world data with a clear evaluation protocol.","section":"Table 1; Sec. 5.3"},{"comment":"The base model is described as 'our own image-to-3D object generation model' and is not released, which makes the experiments non-reproducible for other researchers. Since the paper's central contribution is a paradigm to 'extend pre-trained image-to-3D object generation models,' it would be substantially strengthened by demonstrating the method on at least one publicly available base model (e.g., CLAY, Michelangelo, or Direct3D) in addition to the authors' internal model. Without this, the reader cannot verify whether the reported gains come from the proposed multi-instance attention and training scheme or from properties of the unreleased base model.","section":"Sec. 5.1; Sec. 4"},{"comment":"The ablation study in Table 2 is reported without variance or repeated runs, and is conducted only on 3D-Front. The conclusion that K=5 is optimal is based on a single configuration. Moreover, the K=0 row reports only scene-level metrics (CD-S and F-Score-S) and leaves object-level metrics as '–', because the model cannot generate separable multi-instances; this is acknowledged in the text, but it means the comparison between K=0 and K=5 is not on the same set of metrics. The authors should either provide a way to measure object-level quality for K=0 or acknowledge more clearly that the K=0 baseline is not directly comparable for those metrics.","section":"Table 2"}],"minor_comments":[{"comment":"In Eq. (3), the expectation is written over {z_i}, y, {epsilon_i}, t, but y is defined only loosely as the composite image; clarify that y includes the global scene image, all local object images, and all masks, and make the dependence explicit in the notation.","section":"Sec. 4.1"},{"comment":"The sentence 'We set the resolution of y to 512' is ambiguous: specify whether this is 512×512 pixels and how the variable-length number of instances (N up to 7) is accommodated in the batching and conditioning.","section":"Sec. 5.1"},{"comment":"The supplementary material writes 'for t = 1, · · ·, T' for the rectified flow perturbation, while the main text (Sec. 4.3) correctly states t is sampled from 0 to 1. This inconsistency should be corrected, as it creates confusion about the time indexing.","section":"Supp. Sec. 7, Eq. (4)"},{"comment":"There is a typo: 'qualitiative' should be 'qualitative'.","section":"Sec. 5.2"},{"comment":"The sentence 'and then inferences the geometry in a new image' uses the noun 'inferences' as a verb; it should be 'infers the geometry from a new image'.","section":"Sec. 1"},{"comment":"The stylized-image evaluation compares MIDI exclusively with REPARO. This is understandable given the input type, but the paper should state explicitly that the comparison is limited and not claim parity with a full benchmark on this data.","section":"Sec. 5.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is promising, and the experiments show strong results if the method works as described. The main concern is the missing specification of the latent coordinate frame and how the frozen VAE handles global scene placements; this is a technical gap that could be resolved in revision by adding a detailed description of the training-time encoding and inference-time decoding procedure, possibly with a clarifying figure. The use of an unreleased base model is a reproducibility concern that becomes more serious given that the contribution is explicitly a way to extend such models; I would encourage the editor to ask for either release of the base model or a demonstration on a public alternative. The lack of any variance estimates in the quantitative tables is also worth addressing, but it is secondary to the coordinate-frame issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the quick read on MIDI. It proposes a genuinely new formulation: extend a single-object image-to-3D diffusion model to denoise all scene instances at once, using a multi-instance attention layer that lets each object's tokens attend to the whole scene. That is not what the multi-stage compositional methods do, and the reported gains are large: IoU-B jumps from 0.392 (DiffCAD) to 0.518 on 3D-Front, with better F-scores and 40s runtime. The mixed training with Objaverse to preserve object priors is sensible, and the qualitative results on synthetic, real, and stylized images are worth looking at.\n\nThe soft spot is real and load-bearing. The paper never explains how the frozen object-centric VAE represents and decodes instances in a shared global coordinate frame. The base VAE compresses point positions and normals to latents and decodes to SDF values; it is trained on individual normalized objects. The paper says latents can be 'decoded and directly combined' (Sec 4.1), and the limitations section (Sec 10) confirms generation happens in a global -1..1 coordinate space. But if each z_i is in object-canonical coordinates, pose information is missing and a combination step is silently required; if each z_i is in global coordinates, the frozen VAE is being fed out-of-distribution inputs and the decoder has no mechanism to place geometry anywhere but its training origin. Eq. (3) needs ground-truth latents for scene instances, but their coordinate frame is never stated. This is not a minor exposition issue—the central claim of simultaneous spatially coherent generation depends on this mechanism.\n\nSecondary issues: no error bars or multiple seeds; K=5 (number of multi-instance attention layers) is selected from the test split; and the base model is the authors' own unreleased model, so the results are not independently reproducible. The paper would be substantially stronger with a released base model or code, and with a validation split for hyperparameters.\n\nAll that said, the idea is good enough to warrant a serious referee. The coordinate handling may be fixable with a clarification or a small architectural change (e.g., fine-tuning the VAE or adding an explicit pose latent), but the authors need to actually provide that detail. I'd send this to peer review with a major-revision request: explain the coordinate frame, add error bars, use a validation split, and release what you can. It's a promising direction, just not fully baked as written.\n\nBest.","headline":"A novel and promising multi-instance diffusion formulation, but the missing coordinate-frame explanation for the frozen VAE makes the central mechanism unverifiable.","tokens_in":17476,"tokens_out":3530,"would_cite":false,"duration_ms":33364,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a single diffusion pass can generate an entire 3D scene with multiple, spatially coherent objects from one image.","keywords":["multi-instance diffusion","3D scene generation","single-image 3D","diffusion transformer","spatial relationships","image-to-3D generation","multi-instance attention","scene layout generation"],"falsifier":"Generate a scene with two visually identical objects, decode the raw per-instance meshes without any alignment step, and measure the volume IoU of their bounding boxes against the ground-truth layout. If the latent carries no usable position information, the two instances will collapse toward the same location or swap places, which would break the reported spatial-coherence claim.","tokens_in":16492,"feed_emoji":"🏠","tokens_out":7058,"duration_ms":60879,"temperature":0.7,"pith_summary":"This paper claims that generating a full 3D scene from a single image can be done in one diffusion pass, rather than by segmenting the image, generating each object separately, and then optimizing their relative positions. The authors take a pre-trained image-to-3D object generation model and equip it with multi-instance attention, so the latent codes of all objects in the scene are denoised simultaneously and can see each other through the attention layers. Because inter-object interactions are modeled inside the generation process, the resulting instances share a global coordinate frame and preserve the spatial layout of the input. If correct, this would make single-image 3D scene generation faster and less error-prone than multi-stage pipelines, and it would carry the generalization of large object generators over to scenes. On synthetic and real benchmarks the method reports large gains, such as improving volume IoU of bounding boxes on 3D-Front from 0.392 (DiffCAD) to 0.518.","feed_headline":"One diffusion pass turns a single image into a 3D scene","feed_subtitle":"MIDI generates all objects at once with correct spatial layout, beating multi-stage pipelines on synthetic and real benchmarks.","key_machinery":"The load-bearing mechanism is the multi-instance attention layer: a modification of the original object self-attention in which the feature tokens of instance $i$ no longer query only themselves but attend to the concatenated tokens $\\{f^j\\}_{j=1}^N$ of all instances. This lets the denoising network reason about cross-instance interactions in latent space. The other pieces are: concurrent denoising of $N$ latent codes with a shared noise level $t$ and a rectified-flow loss, and a composite conditioning image $y \\in \\mathbb{R}^{h \\times w \\times 7}$ that stacks each object's RGB crop, its mask, and the global scene image before a DINO-initialized ViT encoder. Together they let the model produce per-instance latents $\\{z_0^i\\}$ that can be decoded and directly combined into a scene.","core_discovery":"The central claim is that an image-to-3D object diffusion model, pre-trained on large single-object data, can be fine-tuned into a multi-instance diffusion model that generates all objects of a scene at once with correct spatial relationships. The key change is to convert a subset of the transformer self-attention layers into multi-instance attention, so that every token in every instance attends to the tokens of all other instances during denoising. At training time, the model is supervised with scene-level data (3D-Front) to learn inter-object interactions, while single-object data (Objaverse) is mixed in 30% of the time to preserve the pre-trained generalization. Conditioning is a 7-channel composite of the object's RGB image, its mask, and the global scene image, encoded by a ViT and injected through cross-attention. The authors report state-of-the-art results on 3D-Front, BlendSwap, Matterport3D, and ScanNet, and also on stylized images from text-to-image models, claiming that simultaneous multi-instance generation avoids the error accumulation of compositional pipelines.","pith_inferences":["Because the model's latent has no explicit per-object position token, scaling to a large number of objects may degrade; a testable extension is to inject explicit layout or pose tokens into the multi-instance attention, which the paper does not explore.","The attention mechanism is order-agnostic, so permuting the instance order at inference should not change the output; if it does, the model has learned order-specific artifacts rather than true cross-instance geometry. This is a cheap consistency check the authors did not report.","The same simultaneous-denoising idea could transfer to dynamic scenes or human-object interaction, but the paper acknowledges its current training data only supports simple static interactions.","The reported IoU-B gains mix geometry quality and layout accuracy; separating the two, for instance by evaluating with ground-truth object shapes, would clarify how much of the improvement comes from the spatial-coherence mechanism versus the stronger object prior."],"forward_implications":["Scene generation becomes a single feed-forward diffusion pass, cutting runtime to about 40 seconds per scene on an A100, compared with minutes for compositional baselines.","Inter-object spatial relationships are baked into the generation process, so the output layout matches the input image better than pipelines that optimize layouts after generating objects one by one.","The method inherits the generalization of pre-trained object generators, as shown by results on real-world scans and on stylized images from text-to-image models.","Fine-tuning with a small scene dataset is enough if single-object data is used as regularization, because the pre-trained object prior is largely retained.","The recipe is described as applicable to pre-trained image-to-3D object generation models generally, not to a single backbone."],"supporting_citations":[{"why":"The pre-trained image-to-3D object generation backbone (CLAY-style) whose VAE and DiT denoiser MIDI extends to multiple instances.","marker":"[78]"},{"why":"3D-Front supplies the scene-level training and test data with paired images, object instances, and layouts that teach inter-object interactions.","marker":"[15]"},{"why":"Objaverse provides single-object data used for 30% mixed regularization to preserve the pre-trained object prior.","marker":"[9]"},{"why":"DiffCAD is the retrieval-based baseline whose 3D-Front IoU-B of 0.392 is the key comparison for MIDI's 0.518.","marker":"[17]"},{"why":"REPARO is the compositional generation baseline that MIDI compares against on stylized images and real data.","marker":"[21]"},{"why":"Gen3DSR is a divide-and-conquer compositional baseline used in the quantitative comparisons.","marker":"[11]"},{"why":"DINOv2 initializes the ViT image encoder that processes the 7-channel composite conditioning.","marker":"[49]"},{"why":"Rectified flow defines the diffusion loss and the linear noise schedule used in Eq. (3).","marker":"[42]"}],"fun_headline_variants":["Single image to full 3D scene in one diffusion pass","Diffusion generates all 3D scene objects simultaneously","Multi-instance diffusion: one image to coherent 3D scenes","MIDI: one-shot multi-object 3D scene generation","From one image to all 3D scene objects in a single pass"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument rests on the assumption that a VAE trained on single objects can encode and decode several objects inside one shared 3D coordinate frame, so that the decoded instances sit at the right places in the scene without any post-hoc alignment; the paper does not specify where the positional information lives in the latent.","fun_headline_variants_meta":{"raw":{"variants":["Single image to full 3D scene in one diffusion pass","Diffusion generates all 3D scene objects simultaneously","Multi-instance diffusion: one image to coherent 3D scenes","MIDI: one-shot multi-object 3D scene generation","From one image to all 3D scene objects in a single pass"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000812,"raw_usage":{"total_tokens":3570,"prompt_tokens":964,"completion_tokens":2606,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":2519}},"tokens_in":580,"tokens_out":2606,"duration_ms":15827,"temperature":1.0,"reasoning_tokens":2519,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:15:18.745232+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a scene with two visually identical objects, decode the raw per-instance meshes without any alignment step, and measure the volume IoU of their bounding boxes against the ground-truth layout. If the latent carries no usable position information, the two instances will collapse toward the same location or swap places, which would break the reported spatial-coherence claim.","supporting_citations":[{"cited_title":"Clay: A controllable large-scale generative model for creat- ing high-quality 3d assets","cited_arxiv_id":null,"evidence_quote":"The pre-trained image-to-3D object generation backbone (CLAY-style) whose VAE and DiT denoiser MIDI extends to multiple instances."},{"cited_title":"Diffcad: Weakly-supervised probabilistic cad model retrieval and alignment from an rgb image","cited_arxiv_id":null,"evidence_quote":"DiffCAD is the retrieval-based baseline whose 3D-Front IoU-B of 0.392 is the key comparison for MIDI's 0.518."}],"review_version":1}