{"id":"40276f84-3565-4b55-904e-1f960d1329f2","arxiv_id":"2412.11457","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MOVIS adds depth and mask conditioning, an auxiliary mask-prediction task, and a timestep curriculum to a view-conditioned diffusion model, improving multi-object novel view synthesis and cross-view consistency.","lead":"MOVIS is a method for generating consistent new views of indoor scenes with several objects from a single photo. It matters because without structural conditioning, camera movement causes furniture to shift or distort, a failure that standard image-similarity scores miss.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative evidence stops at white-background composites; real indoor scenes are only qualitative, so the title-level claim rests on an untested transfer assumption.","rationale":"The reader's weakest_assumption is the same concern I find most load-bearing: the central claim of strong generalization to realistic indoor scenes is supported only by qualitative examples on SUNRGB-D and 3D-FRONT, while all quantitative evidence is on synthetic white-background composites. I checked the other candidate weaknesses. The scheduler's 'fine detail' phase uses a mean timestep of 500, which is not small in absolute terms, and this weakens the paper's narrative about the scheduler; however, the ablation shows a consistent improvement from the scheduler and this does not affect the comparison against baselines. The MASt3R-based Hit Rate and Dist metrics are applied uniformly to all methods and are corroborated by foreground IoU, so even if the matching metric is noisy, it is not the sole support for the placement claim. The unresolved issue is the domain gap: the proposed IoU metric is inapplicable to real images, the limitations section concedes that background texture is not modeled, and no quantitative experiment on a full indoor scene with background is provided. Because 3D-FRONT has renderable ground truth, the missing test is concrete and feasible. This confirms the reader's CONDITIONAL verdict rather than moving it.","tokens_in":24847,"tokens_out":6532,"duration_ms":63461,"concrete_test":"Render at least 100 held-out 3D-FRONT room scenes with full backgrounds, using the paper's 256x256 setup and similar camera ranges, and compute Table 1 metrics for MOVIS and the baselines: PSNR/SSIM/LPIPS, foreground IoU against ground-truth masks, and Hit Rate/Dist. Use the same DepthFM/SAM preprocessing as the paper's real-image protocol. Also report the same metrics on the white-background C3DFS test set for calibration. If foreground IoU or PSNR degrades by more than roughly 15 points or 3 dB relative to C3DFS, the transfer assumption fails and the indoor-scenes claim is unsupported; if not, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every quantitative result in Table 1 and the ablations is obtained on synthetic composites of 3-6 furniture objects rendered on white backgrounds (C3DFS, Objaverse, Room-Texture); real-world SUNRGB-D and synthetic 3D-FRONT appear only as qualitative figures. The title-level claim about indoor scenes therefore rests on an untested transfer assumption: that a model trained with rendered depth/instance masks and white backgrounds will work when given DepthFM/SAM estimates on full indoor images with walls, floors, and clutter. The paper's own foreground-IoU metric thresholds gray level 250 to separate objects from a white background (Sec. B.3), so it cannot be applied to real scenes at all, and the limitations section explicitly disclaims background modeling. Since 3D-FRONT is a synthetic dataset with renderable ground truth, a quantitative evaluation there is feasible and its absence is the single largest gap between the evidence and the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MOVIS, a view-conditioned diffusion model for multi-object novel view synthesis (NVS). The method augments a Stable Diffusion backbone with three components: (i) structure-aware input conditioning using the input-view depth map and instance mask, (ii) an auxiliary task that predicts the target-view object mask from the U-Net's final features, and (iii) a structure-guided timestep sampling scheduler whose Gaussian mean decays linearly during training to shift from global placement learning to fine-grained detail recovery. The authors introduce a new synthetic dataset, C3DFS, composed of 3-6 furniture objects on white backgrounds, and propose two cross-view consistency metrics (Hit Rate and Nearest Matching Distance) based on MASt3R matching, in addition to foreground IoU. They compare against Zero-1-to-3, ZeroNVS, and Free3D on C3DFS, Objaverse composites, and Room-Texture, with qualitative results on 3D-FRONT and SUNRGB-D, and report consistent improvements in PSNR/SSIM/LPIPS, IoU, and cross-view consistency.","tokens_in":24978,"tokens_out":4091,"duration_ms":36555,"significance":"If the transfer to real indoor scenes holds, the work is a useful practical contribution: the structure conditioning, auxiliary mask prediction, and curriculum-style timestep sampling are simple and appear to yield large gains on multi-object placement and cross-view consistency relative to strong baselines. The C3DFS dataset with disjoint furniture splits is a valuable benchmark, and the MASt3R-based evaluation protocol addresses a real gap in the NVS literature. The main caveat is that all quantitative evidence is confined to synthetic composites with white backgrounds; the real-world and full-scene claims rest on qualitative demonstrations only. The paper also ships a clear ablation study showing each component contributes, though without error bars or significance tests.","major_comments":[{"comment":"All quantitative results -- image-level metrics, foreground IoU, Hit Rate, and Dist -- are computed on synthetic composites rendered on white backgrounds (C3DFS, Objaverse, Room-Texture). The real-world SUNRGB-D and synthetic 3D-FRONT appear only in qualitative figures (Fig. 4 and Figs. S.9, S.10). Because the title and abstract claim multi-object NVS for indoor scenes and the introduction emphasizes generalization to realistic datasets, the absence of quantitative evaluation on a full-scene dataset is a load-bearing gap. I request a quantitative evaluation on 3D-FRONT, which has renderable ground truth, and preferably also on SUNRGB-D with an adapted protocol (e.g., SAM-based masks and background-agnostic metrics).","section":"Sec. 4.1 (Datasets), Table 1, and Fig. 4"},{"comment":"The foreground IoU metric computes the foreground mask by thresholding the generated image as M = IL < 250, explicitly exploiting the white background of the synthetic composites. This metric cannot be applied to real indoor images containing walls, floors, and clutter. Consequently, the placement evidence for the paper's central claim is limited to the synthetic white-background setting. Please add a background-agnostic placement metric, such as per-instance IoU using SAM-estimated masks, and report it on at least one scene-level dataset.","section":"Sec. B.3 (Metrics, IoU)"},{"comment":"All numerical results are single-run point estimates with no error bars, confidence intervals, or significance tests. Some improvements are modest (e.g., PSNR 10.014 vs. 9.623 on Room-Texture), so it is important to verify that the reported gains are stable across at least two or three training seeds, or to provide a statistical test over test-set samples, before concluding that MOVIS 'significantly outperforms' the baselines.","section":"Tables 1, 2, and S.5"},{"comment":"The structure-guided scheduler has several hand-chosen hyperparameters (mu_global=1000, mu_local=500, sigma=200, warmup of 4000 steps, decay over 2000 steps, final 6000 steps). The ablation compares only the linear-decay schedule against three alternatives and is performed only on C3DFS. Since the scheduler is a central contribution, please include a small sensitivity analysis (varying sigma and the decay length) and at least one result on a held-out dataset such as Objaverse to show that the choice is not overfit to C3DFS.","section":"Sec. 3.3 and Table S.3"}],"minor_comments":[{"comment":"Line 13 contains a typo: 'Hits ← −Hits + 1' should be 'Hits ← Hits + 1'.","section":"Algorithm 1"},{"comment":"The sentence 'We also evaluate our model on diverse indoor scenes from both the synthetic dataset 3D-FRONT and the real-world dataset SUNRGB-D' is imprecise: the evaluation on these datasets is qualitative only, while the quantitative evaluation is on synthetic composites. Please reword to distinguish these clearly.","section":"Sec. 4.1"},{"comment":"The auxiliary mask loss is applied to latent mask features, as described in Sec. A.3, but the main text does not specify that the supervision is in latent space. Please state this explicitly in the main text to avoid the impression that the loss is computed on decoded masks.","section":"Sec. 3.2, Eq. (3)"},{"comment":"The caption appears to contain the run-together text 'NVSCross View Matching'; please insert a space.","section":"Fig. 1 caption"},{"comment":"The sentence 'This choice stems from the recent advancements in object segmentation [33], while we leave the background modeling for future work' is a significant scope limitation that is deferred to the supplementary material. I recommend stating this limitation prominently in the main text, since it directly bears on the indoor-scene claim.","section":"Sec. 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a credible contribution within the synthetic white-background domain, and the proposed components are well motivated. The main risk is that the title and abstract overstate the indoor-scene claim relative to the quantitative evidence. I would ask the authors to add a quantitative evaluation on 3D-FRONT (or another full-scene dataset with ground truth) and to consider either adapting the metrics to real scenes or narrowing the title/claims to foreground-object NVS on synthetic composites. I would not reject the paper: the core ideas are sound and the synthetic evidence is substantial, but the load-bearing transfer assumption must be tested before the indoor-scene claim can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the Pith Report. I read the paper myself and largely agree with the conditional verdict. What's actually new: the paper identifies a concrete failure mode—single-object view-conditioned diffusion models fall apart when asked to synthesize novel views of multiple objects—and shows that injecting input depth and object masks, adding auxiliary novel-view mask prediction, and annealing the timestep sampling distribution together fix much of it. The gains are not huge in PSNR/SSIM/LPIPS, but the placement and consistency numbers are: IoU jumps from ~34 to 58 on C3DFS, Hit Rate from ~4 to 19. That is a real effect, not noise. The ablations are clean, the furniture split into disjoint train/test sets is fair, and re-training Zero-1-to-3 on C3DFS for comparison is the right call. The C3DFS dataset and the MASt3R-based cross-view consistency metrics are useful artifacts even if never released.\n\nThe soft spots are real but not fatal. The largest is the gap between the title and the evidence: every quantitative result is on synthetic foreground composites rendered on white backgrounds. SUNRGB-D and 3D-FRONT appear only as qualitative figures, and the foreground IoU metric uses a gray-level threshold of 250, so it cannot even be computed on real scenes. The paper's own limitations section admits it does not model background texture. That is honest, but it means the 'indoor scenes' claim is a transfer hypothesis, not a demonstrated result. Since 3D-FRONT is synthetic with renderable ground truth, adding quantitative numbers there is feasible and should be required before the title-level claim is made. Second, no code, checkpoints, or dataset release is stated, so the numbers cannot be reproduced today. Third, the tables lack error bars; with only 5k test composites, a few seeds would settle whether the ~0.2 PSNR advantages are meaningful. A minor point: the scheduler story says small timesteps refine detail, but the implemented mean floors at 500, which is mid-noise, not small; the ablation justifies the schedule empirically, but the prose should be tightened.\n\nWho should read this: anyone working on diffusion-based NVS, scene-level generation, or view-consistency evaluation. It deserves a serious referee. My recommendation: send to peer review, require either quantitative 3D-FRONT (or similar) results or a scoped title/claim, and ask for artifacts.","headline":"Solid multi-object NVS extension with honest limitations; the indoor-scenes claim outruns the quantitative evidence, which stops at white-background composites.","tokens_in":25569,"tokens_out":3294,"would_cite":true,"duration_ms":28596,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a view-conditioned diffusion model equipped with structure-aware inputs, an auxiliary mask-prediction task, and a structure-guided timestep scheduler synthesizes consistent novel views of multi-object indoor scenes…","keywords":["novel view synthesis","diffusion models","multi-object scenes","structure-aware conditioning","depth maps","instance masks","cross-view consistency","timestep sampling"],"falsifier":"Take a real indoor dataset with ground-truth novel views and per-instance masks, run the released MOVIS model with DepthFM/SAM estimates, and compute foreground IoU and MASt3R Hit Rate against ground truth; if under realistic clutter and backgrounds these metrics fall to the level of the Zero-1-to-3 baseline, the title-level claim about indoor scenes would be unsupported. A cheaper check is to compare MOVIS outputs using ground-truth depth and masks versus DepthFM/SAM estimates on the same inputs; a large drop would indicate the transfer depends on the estimators rather than on learned structure awareness.","tokens_in":24585,"feed_emoji":"🛋️","tokens_out":8120,"duration_ms":69108,"temperature":0.7,"pith_summary":"This paper aims to show that the single-object trick of fine-tuning a diffusion model as a novel-view synthesizer can be pushed to multi-object indoor scenes, provided the model is made structurally aware. It argues that feeding depth and instance masks from the input view, adding a target-view mask prediction head, and scheduling sampled noise timesteps so training first learns global placement and later local detail are what let the model place objects correctly and keep shape and appearance consistent under new viewpoints. The payoff, if true, is an end-to-end image-conditioned synthesizer that handles several furniture objects at once, with evaluation metrics that actually measure cross-view consistency rather than only pixel similarity.","feed_headline":"Depth and object masks fix multi-object novel views","feed_subtitle":"A view-conditioned diffusion model places and renders furniture consistently in unseen viewpoints.","key_machinery":"The load-bearing mechanism is a re-purposed latent diffusion denoiser with three additions. Structure-aware feature amalgamation VAE-encodes the input image, normalized depth, and instance-mask image and concatenates their latents with the noised target-view latent. An auxiliary head on the final denoising U-Net layer predicts the target-view object mask, supervised with weight $\\gamma = 0.1$. The structure-guided timestep sampling scheduler draws $t \\sim \\mathcal{N}(\\mu(s), \\sigma)$ with $\\sigma = 200$, where $\\mu(s)$ decays linearly from 1000 to 500 over training, so early training emphasizes global object placement and later training emphasizes fine-grained geometry and appearance. The paper shows through DDIM rollouts that global placement is fixed at large timesteps while mask boundaries sharpen only at small timesteps, which is the observation the scheduler is designed around.","core_discovery":"On the paper's own terms, MOVIS establishes that a view-conditioned diffusion model can move from single-object to multi-object novel view synthesis when it is given structural information and trained to reproduce structure. The central empirical claim is that MOVIS substantially outperforms Zero-1-to-3, ZeroNVS, and Free3D on multi-object synthesis and cross-view consistency. On the paper's C3DFS test set it reports PSNR 17.432 versus 14.811 for the best baseline, foreground IoU 58.1 versus 34.4, and a MASt3R-based Hit Rate of 19.3 versus 4.8, with gains persisting on the Objaverse and Room-Texture generalization sets. The paper attributes these gains to three design choices and isolates them in ablations: removing the scheduler drops IoU from 58.1 to 49.1, removing mask prediction to 54.7, and removing depth input to 57.2.","pith_inferences":["The paper's own limitation statement concedes that multi-view consistency among synthesized images is not guaranteed and background texture is not modeled; a natural next test is whether adding background modeling or training on real RGB-D scenes with backgrounds removes the synthetic-to-real gap in quantitative metrics.","The timestep-schedule insight is likely not specific to novel view synthesis: any diffusion task with a coarse-to-fine structure, such as layout-conditioned generation or compositional image editing, could benefit from a mean-shifting noise schedule that first forces global arrangement and later allocates capacity to details.","The foreground-IoU and MASt3R matching metrics could double as a lightweight proxy for 3D awareness in other single-image generative models, since they detect whether objects move or deform correctly with viewpoint.","Because inference on real images depends on DepthFM and SAM estimates, the method inherits their failure modes; comparing outputs with ground-truth versus estimated depth and masks would show how much of the reported gap is due to the structure conditions themselves."],"forward_implications":["A single image of several objects can be turned into a new viewpoint with each object retaining its identity, position, and rough geometry, which is what image-to-3D and scene reconstruction pipelines need as a first stage.","The three components do independent work: ablations show the scheduler matters most, then mask prediction, then depth input, so future models can adopt components selectively.","Because the auxiliary head outputs novel-view masks, editing operations like object removal under a new viewpoint follow directly from thresholding the predicted mask.","The proposed Hit Rate and nearest-matching-distance metrics give a quantitative handle on cross-view consistency that PSNR, SSIM, and LPIPS miss, and can be applied to any novel-view synthesis method.","Demonstrated generalization to Objaverse, Room-Texture, 3D-FRONT, and SUNRGB-D (the real dataset tested qualitatively) suggests a model trained only on synthetic furniture composites can transfer when background is not the focus."],"supporting_citations":[{"why":"Supplies the view-conditioned diffusion paradigm this work extends and one of the baselines it must beat.","marker":"[46]"},{"why":"Provides the pre-trained latent diffusion denoiser and VAE that MOVIS fine-tunes for multi-object synthesis.","marker":"[66]"},{"why":"Supplies the 3D-FUTURE furniture models used to build the C3DFS training and test composites.","marker":"[16]"},{"why":"Supplies the Objaverse objects used for the synthetic generalization evaluation.","marker":"[12]"},{"why":"DepthFM estimates the input-view depth at inference on real and unannotated images.","marker":"[18]"},{"why":"SAM estimates the input-view instance masks at inference and motivates the foreground-object focus.","marker":"[33]"},{"why":"MASt3R produces the input-to-target image matches on which the new Hit Rate and distance metrics are computed.","marker":"[35]"},{"why":"ZeroNVS is the scene-level baseline trained on real-world mixtures that MOVIS compares against.","marker":"[67]"},{"why":"Free3D is the strongest existing baseline for cross-view consistency in the comparison tables.","marker":"[97]"}],"fun_headline_variants":["MOVIS: structure-aware diffusion for multi-object novel views","Depth and masks make multi-object novel views consistent","Structure-aware sampling boosts multi-object view synthesis","Structural cues unlock consistent multi-object novel views"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that training on synthetic white-background composites of three to six furniture objects, with ground-truth depth and masks during training and DepthFM/SAM estimates during inference, transfers to real indoor scenes with backgrounds and clutter; the paper tests SUNRGB-D only qualitatively.","fun_headline_variants_meta":{"raw":{"variants":["MOVIS: structure-aware diffusion for multi-object novel views","Depth and masks make multi-object novel views consistent","Structure-aware sampling boosts multi-object view synthesis","Structural cues unlock consistent multi-object novel views"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000631,"raw_usage":{"total_tokens":2947,"prompt_tokens":1013,"completion_tokens":1934,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":1884}},"tokens_in":629,"tokens_out":1934,"duration_ms":13911,"temperature":1.0,"reasoning_tokens":1884,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:54:51.252541+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a real indoor dataset with ground-truth novel views and per-instance masks, run the released MOVIS model with DepthFM/SAM estimates, and compute foreground IoU and MASt3R Hit Rate against ground truth; if under realistic clutter and backgrounds these metrics fall to the level of the Zero-1-to-3 baseline, the title-level claim about indoor scenes would be unsupported. A cheaper check is to compare MOVIS outputs using ground-truth depth and masks versus DepthFM/SAM estimates on the same inputs; a large drop would indicate the transfer depends on the estimators rather than on learned structure awareness.","supporting_citations":[{"cited_title":"Zero-1-to-3: Zero-shot one image to 3d object","cited_arxiv_id":null,"evidence_quote":"Supplies the view-conditioned diffusion paradigm this work extends and one of the baselines it must beat."},{"cited_title":"High-resolution image syn- thesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained latent diffusion denoiser and VAE that MOVIS fine-tunes for multi-object synthesis."},{"cited_title":"Segment anything","cited_arxiv_id":null,"evidence_quote":"SAM estimates the input-view instance masks at inference and motivates the foreground-object focus."},{"cited_title":"Ground- ing image matching in 3d with mast3r","cited_arxiv_id":null,"evidence_quote":"MASt3R produces the input-to-target image matches on which the new Hit Rate and distance metrics are computed."},{"cited_title":"Zeronvs: Zero-shot 360- degree view synthesis from a single image","cited_arxiv_id":null,"evidence_quote":"ZeroNVS is the scene-level baseline trained on real-world mixtures that MOVIS compares against."},{"cited_title":"Free3d: Consistent novel view synthesis without 3d representation","cited_arxiv_id":null,"evidence_quote":"Free3D is the strongest existing baseline for cross-view consistency in the comparison tables."}],"review_version":1}