{"id":"ab704dc0-0050-499e-bd23-7d5f49e943c3","arxiv_id":"2411.11934","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A self-supervised video diffusion pipeline that generates stereo video from a monocular input by training on depth-based pseudo-stereo pairs and conditioning on a stereo deviation strength signal.","lead":"SpatialDreamer turns ordinary single-view videos into stereo (left-right) videos using a diffusion model trained on synthetic view pairs generated from depth maps and optical flow. The authors report smoother, more consistent 3D video than existing tools, including Apple's Vision Pro converter, though the evaluation uses a private test set and reports no error bars.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No stereo ground-truth evaluation: FVD and Ewarp cannot verify geometric stereo correctness, so the SOTA/AVP-beating claim is unsupported for the property the paper actually claims.","rationale":"The reader's weakest assumption already identifies the same load-bearing point: pseudo-stereo training data from DVG is never validated against real binocular geometry, and the reported video metrics do not measure stereo correctness. My stress-test confirms and sharpens this. The strongest claim in the paper is not merely 'temporally smooth plausible depth video' but 'geometric consistency' and beating AVP. FVD and Ewarp are necessary but not sufficient for that claim; a model could hallucinate smooth parallax with incorrect disparities and still score well. The RealEstate10K numbers, while strong for image-level novel-view synthesis, do not transfer to stereo-video correctness because the t+5 frame is not a stereo right view. The private benchmark creates an additional reproducibility barrier. Because the concern is that the central claim is unsupported rather than disproven, the appropriate verdict remains CONDITIONAL: the authors should supply real-stereo validation and, ideally, code/data. My recommendation is UNCHANGED relative to the reader's CONDITIONAL verdict, with the condition made explicit: release a real stereo video evaluation with disparity-level metrics. I do not see an internal inconsistency or a fatal flaw in the proposed architecture; the weakness is in the evaluation's ability to verify the paper's own claim.","tokens_in":14914,"tokens_out":4582,"duration_ms":50910,"concrete_test":"Run the method on a public real stereo video dataset with known calibration (e.g., KITTI Raw stereo sequences or a synchronized dual-rig capture), using the actual left-right frame pairs as ground truth. Compute stereo-specific errors between the generated left-right pair and ground-truth disparity/depth: bad-pixel disparity error (e.g., threshold 3 px and 5%), average endpoint error, and left-right consistency error, reported separately for visible and occluded pixels. Include the same metrics for the simple depth-warping baseline and for AVP/3D-photography. If SpatialDreamer does not significantly beat the depth-warping baseline on these stereo-geometry errors, the central claim of geometric consistency and superiority over open-source methods fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that SpatialDreamer produces stereo video with geometric consistency and beats AVP and open-source methods. But no reported metric directly measures stereo geometric correctness. FVD measures distributional similarity of generated videos, and Ewarp measures temporal warp error within a single generated sequence; both can be excellent while the two output views violate epipolar and disparity constraints. The RealEstate10K evaluation is image-level only, and the t=5/t=10 condition compares a synthesized view against a future monocular frame, which is a wide-baseline novel-view benchmark, not a true stereo pair. The video benchmark is a private 100-video set with no stereo ground truth. Meanwhile, the DVG training pairs are produced by monocular depth estimation plus single-image inpainting: occluded target-view content is hallucinated, so training on these pairs can learn plausible, temporally smooth parallax rather than true binocular occlusion/disparity structure. The paper's own Limitations section (4.6) concedes that depth accuracy and distribution affect stereoscopic quality, but no experiment quantifies this against real stereo data. Supplementary Table 6 even shows that DepthCrafter yields a better FVD (62.50) than the proposed MiDaS configuration (67.09), illustrating that FVD ranking does not track stereo fidelity. Thus, the headline 'beats AVP and any open-source stereo video synthesis method' is currently supported only by metrics that do not test the claimed geometric property.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SpatialDreamer, a self-supervised framework for synthesizing stereo video from a monocular input. It uses a depth-based video generation module (DVG) to create pseudo-stereo training pairs via forward-backward rendering and inpainting, with optical-flow-based occlusion refinement; a RefinerNet to inject reference-view spatial features into a Stable Video Diffusion denoising U-Net; and a consistency-control module combining a temporal interaction learning module (TIL) with a 'stereo deviation strength' scalar added as a conditioning embedding. The authors report strong image-level results on RealEstate10K (e.g., PSNR 32.26, SSIM 0.916 at t=5) and the best FVD and Ewarp scores on a privately collected 100-video benchmark, concluding that the method beats AVP and open-source stereo video synthesis methods.","tokens_in":15152,"tokens_out":6086,"duration_ms":61263,"significance":"If the claims were fully supported, the contribution would be significant: it offers a plausible way to train stereo-video synthesis without paired ground-truth stereo data, and the RealEstate10K image results are internally consistent with the ablation study and show large margins over several baselines. The DVG idea of using optical-flow-refined occlusion masks is a useful engineering contribution, and the supplementary ablations of depth and motion estimators are a good practice. However, the central video-level claim of geometric stereo correctness and superiority over AVP is not currently backed by a metric that measures stereo geometry; this gap must be closed before the headline conclusion can be accepted.","major_comments":[{"comment":"The video evaluation is not sufficient for the paper's central claim. FVD is computed between the input monocular video and the synthesized target-view video, but those two videos are supposed to differ by camera viewpoint and parallax; a model that produces only tiny deviations from the input could score well on FVD without producing any meaningful stereo effect. Ewarp measures temporal warp consistency within the generated sequence and does not check epipolar geometry, disparity plausibility, or binocular occlusion structure between the left and right outputs. Since the test set is a private 100-video collection with no stereo ground truth, no reported number in Table 2 verifies that the output is geometrically correct stereo video. Please add a stereo-specific evaluation (e.g., left-right disparity consistency, epipolar error, or comparison against real stereo pairs from a public dataset such as KITTI or DrivingStereo), and report error bars or significance tests over the test videos.","section":"Sec. 4.1 / Table 2"},{"comment":"The training procedure is partly self-referential for the video claim. DVG builds pseudo-stereo pairs from monocular depth, forward-backward rendering, and inpainting; occluded content in the target view is hallucinated by the inpainting model, so the training signal can teach the network to produce plausible and temporally smooth parallax effects rather than true binocular geometry. The paper's own Limitations section (Sec. 4.6) concedes that depth accuracy and distribution directly affect stereoscopic quality, but no experiment quantifies this dependence against real stereo data. I would like to see an experiment that measures the geometric error of the synthesized stereo pairs with respect to known camera geometry, or at least an evaluation on a real stereo video benchmark, so that the 'geometric consistency' claim is not evaluated only through the same pipeline that generated the training data.","section":"Sec. 3.2 / Sec. 4.6"},{"comment":"The RealEstate10K image evaluation does not actually use a stereo baseline. The t=5 and t=10 settings compare the current frame with a future monocular frame, which is a wide-baseline novel-view-synthesis benchmark rather than a typical small-baseline stereo pair. This makes the comparison favorable or unfavorable in ways that are not controlled, and it does not validate the method's behavior at the small baselines used in the video demonstration. In addition, Table 1 has missing entries for several methods at t=10 (e.g., NVSVDE-Net, MVSplat), so the per-condition comparisons are incomplete. Please report results at a proper stereo baseline (e.g., adjacent frames or a fixed small pose offset) and complete the table.","section":"Sec. 4.4 / Table 1"},{"comment":"The 'stereo deviation strength' is defined as s(z) = ||z0 - z_ref||_2 in latent space, but this is not a measure of binocular disparity or scene depth; it is an aggregate L1 difference between latent features. The paper states that this metric 'quantitatively assesses the binocular disparity in a scene', but no connection is shown between this scalar and any actual geometric quantity such as disparity range, baseline, or epipolar geometry. The stereo-aware loss in Eq. (5) therefore supervises a latent-difference quantity, not stereo geometry. This weakens the claim that the consistency-control module 'ensures geometric consistency', because the training signal does not explicitly enforce any stereo constraint.","section":"Sec. 3.4.2 / Eq. (4)"},{"comment":"The depth-estimation ablation reports FVD 62.50 for DepthCrafter versus 67.09 for the proposed MiDaS-based configuration, meaning that the final chosen configuration is not the best according to the headline video metric. This is not necessarily wrong, but it undercuts the argument that FVD is the appropriate ranking metric for this task. Please explain the selection of MiDaS and report stereo-specific metrics (e.g., disparity error or left-right consistency) for each depth estimator; otherwise the reader cannot tell whether the chosen configuration is better in the property that actually matters, namely stereo correctness.","section":"Supplementary Table 6"}],"minor_comments":[{"comment":"There is a typo 'ate-of-the-art' in the contributions paragraph, and the styling 'A VP' should be 'AVP' throughout.","section":"Sec. 1"},{"comment":"Figure 3 contains untranslated Chinese placeholder text ('在此处键入公式。') and should be cleaned before publication.","section":"Figure 3"},{"comment":"Equation (2) is ambiguous: the mask m_t(i,j) appears on both sides of the definition, and the threshold '≥ 1' over a sum of values weighted by a confidence map C(i,j) is not clearly motivated; please clarify the notation and the intended merging rule.","section":"Eq. (2)"},{"comment":"Table 4 has empty cells for several baselines, and the cost table mixes 'DVG' and 'Inference' rows in an inconsistent way; please complete the table or report only the entries that were actually measured.","section":"Table 4"},{"comment":"The private 100-video test set is not described in sufficient detail (resolution, duration, scene types, and camera baseline for the expected stereo pairs), which limits reproducibility; the paper should report these statistics or release the test set.","section":"Sec. 4.1 / Sec. 7.4"},{"comment":"The conclusion claims the method 'meets the demand of real-world application', but Table 4 reports 1059 seconds of inference for a 30-frame 1024x1024 clip and the Limitations section acknowledges the large parameter size; this claim should be qualified.","section":"Sec. 5 / Table 4"}],"recommendation":"major_revision","confidential_remarks":"The private 100-video benchmark is a serious reproducibility bottleneck: the headline 'beats AVP and any open-source stereo video synthesis method' claim is computed on a test set that is not released and for which no stereo ground truth is provided. I would ask the editor to require either public release of the test videos and evaluation protocol, or use of an existing public stereo-video dataset, before final acceptance. The image-level RealEstate10K results are credible and could support a revised paper focused on what is actually demonstrated; the current framing overstates the evidence for geometric stereo correctness."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a competent, well-built pipeline with genuinely strong image-level numbers, but the headline video claim is not supported by the reported metrics. The paper deserves a serious referee, but it needs a major revision before the stereo-video claims can be trusted.\n\nWhat is actually new and good: the DVG module is a sensible fix to a real data problem. Using optical-flow-refined occlusion masks to stabilize pseudo-stereo training pairs addresses the temporal flicker that naively rendered DIBR data would inject. RefinerNet is a reasonable alternative to ControlNet for reference-view conditioning, and the ablation table in Table 3 shows each component (RefinerNet, SDS, TIL, DVG) pulling its weight. The RealEstate10K image results are strong and internally consistent: at t=5 the proposed method gets PSNR 32.26 versus AdaMPI's 27.70, and the qualitative figures show cleaner edges and fewer artifacts. The paper is also honest about its depth dependence in Section 4.6.\n\nThe soft spots are real, and they land on the central claim. The video evaluation uses FVD and Ewarp on a private 100-video test set. FVD measures distributional similarity of generated videos; Ewarp measures temporal warp error inside a single generated sequence. Neither checks whether the left and right views satisfy disparity or epipolar constraints. A method could ace both while producing geometrically wrong stereo. The supplementary itself exposes the problem: Table 6 shows DepthCrafter gives a better FVD (62.50) than the MiDaS configuration they actually use (67.09), so FVD ranking clearly does not track stereo fidelity. There are also no error bars or significance tests on any video metric, and the inference-time setting of the stereo deviation strength s(z) is underspecified: at test time there is no target view to compute |z0 - z_ref|, so it functions as a user dial, and the paper never says what setting produced the reported numbers. Finally, Figure 3 literally contains an unfinished placeholder (\"在此处键入公式\") and an incomplete \"confi...\" line, which is sloppy but fixable.\n\nNone of this kills the paper. The image-level evaluation on RealEstate10K provides independent grounding for spatial accuracy, and the ablations are coherent. But the claim that SpatialDreamer \"beats AVP and any open-source stereo video synthesis methods\" is currently supported only by metrics that test temporal smoothness and photorealism, not stereo geometric correctness. The authors should add stereo-specific evaluation: for example, metrics on real stereo video datasets where ground-truth disparity is available, or a user study with 3D viewing, plus error bars and a precise description of the inference-time deviation strength.\n\nWho this is for: researchers working on stereo video synthesis, monocular NVS, or spatial computing content pipelines. It will be useful once the evaluation is tightened. I would send it to a serious referee rather than desk-reject, but with a clear list of required changes and a request for code or data release.","headline":"Solid self-supervised stereo video pipeline with strong image-level results, but the headline 'beats AVP' claim rests on video metrics that never check stereo geometric correctness.","tokens_in":15743,"tokens_out":2169,"would_cite":false,"duration_ms":23859,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SpatialDreamer claims that stereo video can be synthesized from monocular input by a self-supervised video diffusion model trained on depth-rendered pseudo-stereo pairs, and that the result outperforms published novel-view-synthesis and…","keywords":["stereo video synthesis","monocular to stereo","video diffusion model","self-supervised learning","novel view synthesis","temporal consistency","depth-based rendering","optical flow"],"falsifier":"Collect a set of real stereo videos, for example from a dual-lens camera, run SpatialDreamer on the left-eye stream, and measure disparity error or epipolar consistency against the real right-eye stream; if the generated right-eye views violate epipolar geometry or are no closer to the true right view than a simple depth-warping baseline, the central claim of geometric stereo correctness is refuted.","tokens_in":1806,"feed_emoji":"🎥","tokens_out":1725,"duration_ms":80844,"temperature":0.7,"pith_summary":"SpatialDreamer aims to show that real-world stereo video can be synthesized from a single monocular stream without any paired stereo training data. The paper argues that existing novel-view-synthesis methods fail to keep dynamic scenes consistent over time, and that the data bottleneck can be broken by generating training pairs with depth-based forward-backward rendering plus optical-flow-refined occlusion masks. It then trains a latent video diffusion backbone with a reference-image feature refiner and a consistency-control module, and reports state-of-the-art scores on its collected stereo-video benchmark and on RealEstate10K image synthesis. If the claims hold, high-quality stereo content for VR and spatial video would no longer require dual-camera capture or per-scene optimization.","feed_headline":"Self-supervised pipeline turns single-camera video into stereo","feed_subtitle":"Depth-based training pairs let a diffusion model beat published stereo-video baselines without paired ground truth.","key_machinery":"The central machinery is a self-supervised data-generation loop, called DVG, feeding a conditioned video diffusion model. DVG takes monocular video, estimates per-frame depth with MiDaS, renders the reference view to a target viewpoint and back, and refines occlusion masks by propagating them with RAFT optical flow and forward-backward consistency, producing paired training videos that carry geometric and temporal priors. RefinerNet, a U-Net initialized from Stable Diffusion 2.1 without temporal layers, supplies reference-view features through concatenated spatial self-attention. The consistency-control module adds a stereo deviation strength, s(z) = |z0 - zref|, embedded like a timestep and supervised by a stereo-aware loss, plus TIL, which augments the reference feature with a blend of self-attention and cross-view attention over adjacent frames.","core_discovery":"The paper's central claim is that monocular-to-stereo video synthesis can be solved as a self-supervised generation problem rather than a geometric rendering problem. Its core discovery is that depth-based pseudo-stereo video, generated by rendering a monocular video to a target viewpoint and back, with occlusion masks refined across frames using optical flow, is sufficient supervision to train a video diffusion model that produces geometrically and temporally consistent stereo pairs. The trained model, SpatialDreamer, uses RefinerNet to inject reference-view features into the denoising U-Net, a stereo-deviation-strength embedding to control disparity, and a Temporal Interaction Learning module to blend adjacent-frame features. On RealEstate10K the method reports the best image-level SSIM, PSNR, and LPIPS at time offset 5, and on a self-collected 100-video stereo benchmark it reports the best FVD (67.09) and flow-warping error (3.374e-3), beating published open-source baselines and the Apple Vision Pro 3D converter.","pith_inferences":["The paper's metrics never compare the synthesized right-eye view against a real right-eye camera; a stereo-correctness audit using disparity error, epipolar fit, or viewer fusion comfort is the natural next experiment and is left future work.","Because DVG and inference share the same monocular depth prior, a systematic depth bias on some scene type would corrupt both the pseudo-training pairs and the test-time renderings; this could be probed by training on one depth estimator and running inference with another.","The stereo-deviation-strength embedding may carry over to interactive depth editing or variable-baseline stereo generation, though the paper does not explore those uses."],"forward_implications":["Because DVG generates paired videos without annotation, the method removes the need for expensive dual-camera stereo capture during training.","The stereo-deviation-strength embedding gives users a controllable knob for disparity, so the 3D effect of a scene can be tuned instead of being locked to the input viewpoint.","The optical-flow-refined occlusion masks and TIL are designed to eliminate flicker and jitter that plague frame-by-frame novel-view synthesis, and the reported Ewarp improvement is the quantitative expression of that claim.","The framework inherits the 8-frame clip structure of the video diffusion backbone, so long videos would be synthesized in clips and temporal consistency is claimed over that window.","The paper reports that on its collected benchmark the method beats both open-source depth-warping and NVS baselines and the commercial Apple Vision Pro 3D converter, which it takes as evidence that monocular-to-stereo conversion is production-viable without specialized capture."],"supporting_citations":[{"why":"Stable Video Diffusion supplies the pretrained latent video diffusion backbone and temporal layers that SpatialDreamer adapts.","marker":"[7]"},{"why":"MiDaS provides the monocular depth estimates used by DVG to render the target viewpoint.","marker":"[49]"},{"why":"RAFT optical flow establishes pixel correspondences between adjacent frames for occlusion-mask refinement.","marker":"[60]"},{"why":"Forward-backward consistency in optical flow supplies the confidence map used to merge occlusion masks across frames.","marker":"[16]"},{"why":"Stable Diffusion 2.1 weights initialize RefinerNet so it can extract reference-view features in the same latent space as the denoising U-Net.","marker":"[50]"},{"why":"RealEstate10K provides the large-scale pose-annotated monocular video dataset used for image-level evaluation.","marker":"[78]"},{"why":"3D-photography defines the LPIPS, PSNR, and SSIM evaluation protocol and serves as a primary depth-warping baseline.","marker":"[56]"},{"why":"The Apple Vision Pro 3D converter is the commercial spatial-video baseline that the paper claims to outperform.","marker":"[44]"},{"why":"FVD is the video quality metric used to measure perceptual similarity between input and generated stereo videos.","marker":"[62]"},{"why":"Ewarp is the flow-warping error metric used to evaluate temporal consistency of the generated video sequences.","marker":"[35]"}],"fun_headline_variants":["Self-supervised stereo video from single-camera input","Depth-based pseudo-pairs train stereo video diffusion","SpatialDreamer: monocular video to stereo without paired data","Self-supervised method beats baselines on stereo video synthesis","Diffusion model generates consistent stereo video from monocular"],"cache_read_input_tokens":17792,"weakest_assumption_plain":"The paper assumes that pseudo stereo pairs produced by monocular depth plus rendering are good enough substitutes for real stereo pairs during training, and it does not test the resulting videos against true stereo ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised stereo video from single-camera input","Depth-based pseudo-pairs train stereo video diffusion","SpatialDreamer: monocular video to stereo without paired data","Self-supervised method beats baselines on stereo video synthesis","Diffusion model generates consistent stereo video from monocular"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000619,"raw_usage":{"total_tokens":2891,"prompt_tokens":984,"completion_tokens":1907,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":1827}},"tokens_in":600,"tokens_out":1907,"duration_ms":13187,"temperature":1.0,"reasoning_tokens":1827,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:18:00.012876+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a set of real stereo videos, for example from a dual-lens camera, run SpatialDreamer on the left-eye stream, and measure disparity error or epipolar consistency against the real right-eye stream; if the generated right-eye views violate epipolar geometry or are no closer to the true right view than a simple depth-warping baseline, the central claim of geometric stereo correctness is refuted.","supporting_citations":[{"cited_title":"Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer","cited_arxiv_id":null,"evidence_quote":"MiDaS provides the monocular depth estimates used by DVG to render the target viewpoint."},{"cited_title":"Egnal and R.P","cited_arxiv_id":null,"evidence_quote":"Forward-backward consistency in optical flow supplies the confidence map used to merge occlusion masks across frames."},{"cited_title":"3d photography using context-aware layered depth 10 inpainting","cited_arxiv_id":null,"evidence_quote":"3D-photography defines the LPIPS, PSNR, and SSIM evaluation protocol and serves as a primary depth-warping baseline."},{"cited_title":"What is spatial video on iphone 15 pro and vision pro","cited_arxiv_id":null,"evidence_quote":"The Apple Vision Pro 3D converter is the commercial spatial-video baseline that the paper claims to outperform."},{"cited_title":"FVD: A new metric for video generation","cited_arxiv_id":null,"evidence_quote":"FVD is the video quality metric used to measure perceptual similarity between input and generated stereo videos."},{"cited_title":"Learning blind video temporal consistency","cited_arxiv_id":null,"evidence_quote":"Ewarp is the flow-warping error metric used to evaluate temporal consistency of the generated video sequences."}],"review_version":1}