{"id":"6a282a1c-7919-4a50-906a-cf0028a6b7df","arxiv_id":"2504.18448","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"NoiseController decomposes initial diffusion noise into scene-level foreground/background and shared/residual components, then collaborates them across views and frames, improving multi-view video consistency on nuScenes.","lead":"This paper introduces NoiseController, a method for generating multi-view videos by splitting the initial diffusion noise into background and foreground, and then into shared and residual pieces, so different views stay consistent while keeping variety. Because consistent synthetic driving videos can train perception models, the method is aimed at autonomous driving applications and reports gains in generation quality and downstream detection and segmentation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The foreground/background decomposition assumes 3D-box masks are accurate in the latent space, yet the paper provides no validation or sensitivity analysis of this assumption.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the correctness of the binary masks in the latent space is essential to the noise decomposition and joint denoising. This concern is more fundamental than the equation typos or missing hyperparameters because those are presentation issues that could be corrected, whereas the mask assumption underpins the method's core mechanism. If the masks are inaccurate, the claimed foreground/background separation is invalid, and the reported gains in controllability and consistency lose the stated interpretation. Since the reader already issued a CONDITIONAL verdict and this concern is consistent with that verdict, no change is recommended. The proposed concrete test, an IoU analysis and a mask-perturbation ablation, would directly settle whether the concern lands.","tokens_in":13547,"tokens_out":9905,"duration_ms":92708,"concrete_test":"Compute the overlap between the binary masks obtained by projecting nuScenes 3D boxes onto each camera view and resizing to the VAE latent resolution, and ground-truth foreground/background masks derived from instance segmentation or LiDAR-projected depth. Report the mean IoU over the validation set and across all views/frames; if the mean IoU is below 0.5, the masked noise targets in Eq. (6) are mis-specified. As a complementary check, retrain the model with the box masks randomly shifted or dilated by a few pixels; if FVD/FID remain within 5% of the reported values, mask accuracy is not load-bearing, whereas substantial degradation would confirm the concern.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The method's central mechanism, multi-level noise decomposition and joint denoising, relies on binary masks derived from 3D object boxes to partition each frame's latent noise into background and foreground components. These masks are used both to construct the ground-truth regression targets N^B_t and N^F_t in Eq. (6) and to mask the two U-Nets' predictions in Eq. (7). If the box-to-latent masks are misaligned because of projection errors, VAE downsampling, or box inaccuracies, the regression targets are not true foreground/background noises, and the two U-Nets are trained to denoise arbitrary spatial partitions. The paper gives no analysis of mask accuracy in the latent space, no ablation on mask perturbation, and no evidence that the learned decomposition corresponds to actual objects. Consequently, the claimed improvements in foreground/background controllability (Table 5) and multi-view consistency may stem primarily from the collaboration matrices and the extra U-Net capacity rather than from a meaningful noise decomposition. This is load-bearing because the paper's key novelty, capturing distinct motion properties of foreground and background, collapses if the masks do not separate those properties.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NoiseController, a method for multi-view video generation that operates by manipulating initial latent noise. It introduces a two-level noise decomposition: scene-level decomposition into foreground and background noises via 3D-box-derived binary masks, and individual-level decomposition of each scene-level noise into shared and residual components. An inter-view spatiotemporal collaboration matrix and an intra-view impact collaboration matrix are learned to produce shared components from preceding frames, and two parallel U-Nets denoise background and foreground jointly. Experiments on nuScenes report FVD 122.9 and FID 14.65 at 224x400, and FVD 87.23 and FID 13.72 at 512x1024, outperforming prior multi-view video generation methods. The authors also report that generated videos improve downstream BEVFusion detection and CVT segmentation, and they show compatibility of the framework with MagicDrive and TrackDiffusion.","tokens_in":13701,"tokens_out":7166,"duration_ms":71972,"significance":"If the technical claims are correct and reproducible, NoiseController is a meaningful contribution to multi-view driving video generation. The paper provides a coherent set of ablations for its three main modules, studies the sliding window length and the two collaboration matrices, evaluates both generation quality and downstream perception, and demonstrates that the framework can be plugged into existing baselines. These are substantive strengths. However, several core equations are internally inconsistent, the values of the variance-splitting hyperparameters are not reported, and the mask construction is not validated; these issues currently prevent the reader from verifying the central mechanism or reproducing the results.","major_comments":[{"comment":"The decomposition identity is not valid as written. With M^F = 1 - M^B, the equation N^B + N^F = epsilon^B \\odot M^B + epsilon^F \\odot M^F equals epsilon only on the background region where epsilon equals epsilon^B and on the foreground region where epsilon equals epsilon^F. If epsilon^B and epsilon^F are independently sampled from N(0, I), as stated, the sum is a newly generated noise and not the original epsilon. This makes the ground-truth regression targets N^D_t and the training loss in Eqs. (9)-(10) ill-defined. Please either define N^B and N^F as masked versions of the same initial noise, or describe the procedure as composing a new noise rather than decomposing the original, and correct the distributional statements because masked noise is not distributed as N(0, I).","section":"Sec. 4.1, Eq. (1) and Sec. 4.2, Eq. (6)"},{"comment":"Equation (8) is internally inconsistent. The symbol epsilon^B is introduced as a tensor initialized to a constant coefficient eta^2/(eta^2+1), but the same symbol is used as a noise quantity. The summand (S_i \\cdot epsilon^D_i \\odot I^B_k) \\odot (S_i \\cdot epsilon^D_i \\odot I^B_k) squares the collaboration term, which does not match the linear collaboration operation in Eq. (5). In addition, k is not defined inside the summation over i. Since L_C = L_B + L_F is part of the training objective, this equation must be corrected and its notation clarified.","section":"Sec. 4.3, Eq. (8)"},{"comment":"The hyperparameters eta and lambda define the variance split between shared and residual components, and they also enter the collaboration loss L_C. Their values are never reported in the implementation details or elsewhere, and no ablation over these parameters is given. Without these values, the individual-level decomposition and the training objective cannot be reproduced.","section":"Sec. 4.1 and Sec. 5.1"},{"comment":"For n > 1, the regression target N^B_{n,t} is compared with a sum of terms S_i \\cdot \\hat N^B_{i,t} \\odot I_{min}, but according to Eq. (5) the shared component for frame n is formed from the scene-level noises epsilon^D_i before masking, and the full noise at frame n also contains the independently sampled residual component. As printed, Eq. (9) does not compare the same quantities that the generation procedure produces, so the loss may not minimize the intended prediction error. Please reconcile Eq. (9) with Eq. (5).","section":"Sec. 4.3, Eq. (9)"},{"comment":"The central foreground/background separation relies on binary masks derived from 3D object boxes and mapped into the latent space, but the paper provides no details of how the masks are projected, no visualization of the masks in latent space, and no sensitivity analysis with respect to mask perturbation or projection error. Because the decomposition is the paper's main novelty, this missing validation leaves a load-bearing assumption unsupported.","section":"Sec. 4.2, Figure 4 and Sec. 5.1"},{"comment":"The high-resolution variant NoiseController* is reported with state-of-the-art results (FVD 87.23, FID 13.72 at 512x1024), but the asterisk is unexplained and the main text gives no training or adaptation details for this variant. Additionally, no error bars or number of evaluation seeds are reported for the FVD/FID numbers, so the claimed margins over SubjectDrive and Panacea and the stated 30.7% and 30.0% improvements are not statistically grounded.","section":"Sec. 5.4, Table 4 and Sec. 5.1"}],"minor_comments":[{"comment":"In Eq. (5), the index k is used inside the summation over i but is not defined; it should be k = i - (n - K) as implied by the preceding discussion.","section":"Eq. (5)"},{"comment":"The phrase \"We respect 6-view noises\" should be rephrased, for example to \"We use the 6-view noises of the preceding K frames\".","section":"Figure 3 caption"},{"comment":"The caption contains the typo \"stree-view generation methods\" and should read \"street-view\".","section":"Table 5 caption"},{"comment":"The conclusion contains the typo \"multi-leve noise decomposition\".","section":"Conclusion"},{"comment":"The comparison mixes methods evaluated at different resolutions (224x400, 256x448, 256x512, 512x1024, 848x1600); since FVD and FID are resolution-dependent, the table should state this caveat or restrict the headline comparison to matched resolutions.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is potentially solid, but the main text contains several equations that cannot be parsed as written, and the reported high-resolution result is not accompanied by implementation details. I would encourage the editor to require a carefully corrected revision rather than rejecting the work, since the ablations and downstream experiments suggest that the overall framework, after clarification, could make a useful contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: NoiseController is a solid engineering paper, not a conceptual breakthrough. The genuinely new piece is the two-level noise decomposition—scene-level foreground/background on top of the individual-level shared/residual split—plus learned inter-view and intra-view collaboration matrices and two parallel denoising U-Nets. That combination goes beyond VideoFusion, Preserve Your Own Correlation, and FreeNoise, and the ablation table earns each module a place. The headline numbers on nuScenes (FVD 122.9, FID 14.65 at 224x400 vs MagicDrive's 177.3/20.92) are large and internally consistent, and the downstream perception gains (BEVFusion mAP 36.07, NDS 43.54; CVT vehicle IoU 40.97, road IoU 80.13) support the claim that the generated data is usable. I don't see circularity: evaluation uses external metrics and prior-art baselines, and the collaboration matrices are learned, not hand-fitted to the reported numbers. The citation pattern looks appropriate; same-group citations are not padding.\n\nSoft spots, in proportion:\n\n- Equation (8) is internally inconsistent as written: it squares the collaboration term and the text says 'LB + LB' instead of 'LB + LF'. Likely a typo, but in a methods paper that still blocks reconstruction of the loss.\n- eta and lambda, the variance-splitting hyperparameters, are never given values. Small omission, easy fix, but it matters for reproducibility.\n- No error bars, no code, and the 512x1024 result is a single table row. Given how much of the contribution is empirical, I'd want at least a seed-variance statement before believing the margins are stable.\n- The mask concern raised in the stress test has real teeth. The scene-level decomposition relies on binary masks derived from 3D boxes, projected into latent space and used both as regression targets and as prediction masks. There is no validation that those masks align with the VAE's latent semantics, and no sensitivity analysis to box noise or misalignment. I don't think this kills the paper—the collaboration matrices and extra U-Net capacity may carry much of the gain, and the ablation (row 3 vs row 5) suggests decomposition adds something—but the foreground/background motion story is exactly as strong as the mask assumption, and that assumption is unexamined.\n\nWho this is for: researchers working on controllable driving-scene generation and synthetic-data augmentation for perception. It deserves a serious referee; the right outcome is probably a revise-and-resubmit asking for equation typo fixes, hyperparameter values, error bars, and a mask-robustness experiment.\n\nBest","headline":"A practical multi-view driving video generator with a plausible noise-decomposition recipe; the gains over MagicDrive look real, but the paper needs equation fixes, hyperparameter reporting, and mask-sensitivity analysis before I'd trust it beyond the reported setting.","tokens_in":14292,"tokens_out":3025,"would_cite":true,"duration_ms":30521,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NoiseController claims that decomposing the initial latent noise into scene-level foreground/background and individual-level shared/residual components, then collaborating them across views and frames, yields multi-view videos with…","keywords":["multi-view video generation","noise decomposition","spatiotemporal consistency","diffusion models","autonomous driving","joint denoising","noise collaboration","latent noise control"],"falsifier":"Render the mask premise false in a controlled way: take the trained model and feed it box masks that are shifted, dilated, or replaced by random binary masks, keeping everything else fixed. If FVD and FID stay near 122.9 and 14.65, the gains do not depend on mask accuracy; if quality collapses, the reported consistency gains are contingent on precise foreground and background separation in the latent domain.","tokens_in":13283,"feed_emoji":"🚗","tokens_out":7928,"duration_ms":69670,"temperature":0.7,"pith_summary":"The paper tries to establish that the initial noise of a multi-view video diffusion model can be structured so that cross-view and cross-frame consistency is built into the starting point of denoising, instead of being left to attention mechanisms alone. It decomposes each view's initial latent noise into scene-level background and foreground noise, further splits each into shared and residual components, then collaborates these components across six views and a sliding window of past frames before two parallel U-Nets denoise them. If this is right, it matters because the reported gains are concrete: on the driving benchmark, FVD falls from 177.3 to 122.9 and FID from 20.92 to 14.65, and training perception models on the generated videos improves detection and BEV segmentation over training without augmentation.","feed_headline":"Noise decomposition cuts video FVD from 177 to 123","feed_subtitle":"Splitting noise into foreground, background, shared, and residual parts makes multi-view driving video more consistent.","key_machinery":"The load-bearing object is the two-level decomposition of the initial latent noise. At the scene level, masks derived from 3D object boxes split each view's noise into background noise and foreground noise, each assumed standard Gaussian. At the individual level, each scene-level noise is split into a shared component, drawn with a controlled variance that enforces consistency, and a residual component, drawn with complementary variance to preserve diversity. The collaboration machinery consists of an inter-view spatiotemporal collaboration matrix and an intra-view impact collaboration matrix, computed over a sliding window of K frames; these produce the shared components for the next frame from the preceding K frames of all six views. Two parallel denoising U-Nets, one for background and one for foreground, then predict masked noise components that are summed into the final prediction. The argument is carried by the claim that this decomposition and collaboration makes the starting points of denoising consistent across views and frames, so attention alone no longer has to do that work.","core_discovery":"At its core, the paper claims that consistency in multi-view video generation can be achieved by engineering the initial noise rather than by stacking more attention. For every view and frame, the initial latent noise is decomposed with binary masks from 3D object boxes into a background noise and a foreground noise, each modeled as standard Gaussian noise. Each of those is further split into a shared component that carries what should stay the same across views and frames, and a residual component that preserves per-frame diversity. Two learned collaboration matrices—one across views and time, one across background and foreground within a view—propagate information over a sliding window of K frames to build the shared components of the next frame, and two parallel U-Nets denoise background and foreground jointly. The paper reports state-of-the-art generation quality on the public autonomous-driving benchmark: FVD 122.9 and FID 14.65 at 224×400, improving to FVD 87.23 and FID 13.72 at 512×1024, with downstream detection and BEV segmentation also improved when the generated videos are used as training data.","pith_inferences":["Because the decomposition is driven by box-derived masks, the same scheme should extend to any conditional generator with layout control, such as indoor or object-centric multi-view synthesis; this is an extension the paper does not test.","The shared and residual variance coefficients set an explicit consistency-versus-diversity dial; a natural test is to tune them per downstream task rather than leave them fixed.","If the mask premise is the weak point, a testable improvement is replacing binary box masks with learned or semantic masks so that non-box foreground objects are not silently treated as background."],"forward_implications":["The starting point of denoising, not attention, becomes the main carrier of spatiotemporal consistency, so the framework can be grafted onto existing video diffusion models without changing their attention design.","Longer multi-frame collaboration helps only up to a window of K=5; K=6 hurts under the same training budget, so pushing further needs more training rather than simply a bigger window.","Generated multi-view videos can serve as training data for downstream perception: detection mAP rises from 32.48 to 36.07 and road segmentation IoU from 74.30 to 80.13 compared with no augmentation.","The method transfers: applying it to a tracklet-conditioned video generator improves its FVD and FID as well."],"supporting_citations":[{"why":"Supplies the pretrained video diffusion model the method starts from and the baseline it must beat.","marker":"[5]"},{"why":"Second video diffusion model the method is integrated onto to demonstrate transferability.","marker":"[11]"},{"why":"Introduces shared and residual noise decomposition that the multi-level decomposition extends to multi-view.","marker":"[18]"},{"why":"Provides the noise-prior view that motivates controlling the structure of the initial noise.","marker":"[6]"},{"why":"Dataset used for training, generation evaluation, and downstream perception testing.","marker":"[3]"},{"why":"Detection model used to score generated videos on mAP and NDS.","marker":"[16]"},{"why":"Segmentation model used to score foreground and background control on mIoU.","marker":"[45]"},{"why":"Latent diffusion formulation that supplies the noise-latent structure used for joint denoising.","marker":"[24]"}],"fun_headline_variants":["Noise decomposition slices multi-view video FVD to 87","Shared and residual noise boost multi-view video consistency","Foreground/background noise split improves multi-view FVD to 87","Noise splitting with collaboration cuts multi-view video FVD","Engineer the noise: multi-view video FVD drops to 87"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole scheme assumes that binary masks from 3D object boxes correctly separate foreground from background in the noise representation at every denoising step, so that masked noise behaves like valid input to the diffusion network.","fun_headline_variants_meta":{"raw":{"variants":["Noise decomposition slices multi-view video FVD to 87","Shared and residual noise boost multi-view video consistency","Foreground/background noise split improves multi-view FVD to 87","Noise splitting with collaboration cuts multi-view video FVD","Engineer the noise: multi-view video FVD drops to 87"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000927,"raw_usage":{"total_tokens":3995,"prompt_tokens":995,"completion_tokens":3000,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":2914}},"tokens_in":611,"tokens_out":3000,"duration_ms":22146,"temperature":1.0,"reasoning_tokens":2914,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:16:35.620377+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render the mask premise false in a controlled way: take the trained model and feed it box masks that are shifted, dilated, or replaced by random binary masks, keeping everything else fixed. If FVD and FID stay near 122.9 and 14.65, the gains do not depend on mask accuracy; if quality collapses, the reported consistency gains are contingent on precise foreground and background separation in the latent domain.","supporting_citations":[{"cited_title":"MagicDrive: Street view generation with diverse 3d geometry control","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained video diffusion model the method starts from and the baseline it must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces shared and residual noise decomposition that the multi-level decomposition extends to multi-view."},{"cited_title":"Preserve your own correlation: A noise prior for video diffusion models","cited_arxiv_id":null,"evidence_quote":"Provides the noise-prior view that motivates controlling the structure of the initial noise."},{"cited_title":"Bevfusion: Multi- task multi-sensor fusion with unified bird’s-eye view repre- sentation","cited_arxiv_id":null,"evidence_quote":"Detection model used to score generated videos on mAP and NDS."},{"cited_title":"Cross-view transform- ers for real-time map-view semantic segmentation","cited_arxiv_id":null,"evidence_quote":"Segmentation model used to score foreground and background control on mIoU."},{"cited_title":"High-resolution image 9 synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Latent diffusion formulation that supplies the noise-latent structure used for joint denoising."}],"review_version":1}