Pith. sign in

REVIEW 5 major objections 6 minor 2 cited by

SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SpatialDreamer claims that stereo video can be synthesized from monocular input by a self-supervised video diffusion model trained on depth-rendered pseudo-stereo pairs, and that the result outperforms published novel-view-synthesis and…

desk verdict Solid self-supervised stereo video pipeline with strong image-level results, but the headline 'beats AVP' claim rests on video metrics that never check stereo geometric correctness. read the letter →

arxiv 2411.11934 v2 pith:PNBGX5BO submitted 2024-11-18 cs.CV cs.AI

Zhen Lv , Yangqi Long , Congzhentao Huang , Cao Li , Chengfei Lv , Hao Ren , Dian Zheng This is my paper · ORCID
classification cs.CVcs.AI
keywords stereovideosynthesismonoculartodiffusionmodelself-supervisedlearningnovelviewtemporalconsistencydepth-basedrenderingopticalflow
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SpatialDreamer aims to show that real-world stereo video can be synthesized from a single monocular stream without any paired stereo training data. The paper argues that existing novel-view-synthesis methods fail to keep dynamic scenes consistent over time, and that the data bottleneck can be broken by generating training pairs with depth-based forward-backward rendering plus optical-flow-refined occlusion masks. It then trains a latent video diffusion backbone with a reference-image feature refiner and a consistency-control module, and reports state-of-the-art scores on its collected stereo-video benchmark and on RealEstate10K image synthesis. If the claims hold, high-quality stereo content for VR and spatial video would no longer require dual-camera capture or per-scene optimization.

What carries the argument

The central machinery is a self-supervised data-generation loop, called DVG, feeding a conditioned video diffusion model. DVG takes monocular video, estimates per-frame depth with MiDaS, renders the reference view to a target viewpoint and back, and refines occlusion masks by propagating them with RAFT optical flow and forward-backward consistency, producing paired training videos that carry geometric and temporal priors. RefinerNet, a U-Net initialized from Stable Diffusion 2.1 without temporal layers, supplies reference-view features through concatenated spatial self-attention. The consistency-control module adds a stereo deviation strength, s(z) = |z0 - zref|, embedded like a timestep and supervised by a stereo-aware loss, plus TIL, which augments the reference feature with a blend of self-attention and cross-view attention over adjacent frames.

What would settle it

Collect a set of real stereo videos, for example from a dual-lens camera, run SpatialDreamer on the left-eye stream, and measure disparity error or epipolar consistency against the real right-eye stream; if the generated right-eye views violate epipolar geometry or are no closer to the true right view than a simple depth-warping baseline, the central claim of geometric stereo correctness is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that monocular-to-stereo video synthesis can be solved as a self-supervised generation problem rather than a geometric rendering problem. Its core discovery is that depth-based pseudo-stereo video, generated by rendering a monocular video to a target viewpoint and back, with occlusion masks refined across frames using optical flow, is sufficient supervision to train a video diffusion model that produces geometrically and temporally consistent stereo pairs. The trained model, SpatialDreamer, uses RefinerNet to inject reference-view features into the denoising U-Net, a stereo-deviation-strength embedding to control disparity, and a Temporal Interaction Learning module to blend adjacent-frame features. On RealEstate10K the method reports the best image-level SSIM, PSNR, and LPIPS at time offset 5, and on a self-collected 100-video stereo benchmark it reports the best FVD (67.09) and flow-warping error (3.374e-3), beating published open-source baselines and the Apple Vision Pro 3D converter.

Load-bearing premise

The paper assumes that pseudo stereo pairs produced by monocular depth plus rendering are good enough substitutes for real stereo pairs during training, and it does not test the resulting videos against true stereo ground truth.

Editorial extensions

If this is right

  • Because DVG generates paired videos without annotation, the method removes the need for expensive dual-camera stereo capture during training.
  • The stereo-deviation-strength embedding gives users a controllable knob for disparity, so the 3D effect of a scene can be tuned instead of being locked to the input viewpoint.
  • The optical-flow-refined occlusion masks and TIL are designed to eliminate flicker and jitter that plague frame-by-frame novel-view synthesis, and the reported Ewarp improvement is the quantitative expression of that claim.
  • The framework inherits the 8-frame clip structure of the video diffusion backbone, so long videos would be synthesized in clips and temporal consistency is claimed over that window.
  • The paper reports that on its collected benchmark the method beats both open-source depth-warping and NVS baselines and the commercial Apple Vision Pro 3D converter, which it takes as evidence that monocular-to-stereo conversion is production-viable without specialized capture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's metrics never compare the synthesized right-eye view against a real right-eye camera; a stereo-correctness audit using disparity error, epipolar fit, or viewer fusion comfort is the natural next experiment and is left future work.
  • Because DVG and inference share the same monocular depth prior, a systematic depth bias on some scene type would corrupt both the pseudo-training pairs and the test-time renderings; this could be probed by training on one depth estimator and running inference with another.
  • The stereo-deviation-strength embedding may carry over to interactive depth editing or variable-baseline stereo generation, though the paper does not explore those uses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes SpatialDreamer, a self-supervised framework for synthesizing stereo video from a monocular input. It uses a depth-based video generation module (DVG) to create pseudo-stereo training pairs via forward-backward rendering and inpainting, with optical-flow-based occlusion refinement; a RefinerNet to inject reference-view spatial features into a Stable Video Diffusion denoising U-Net; and a consistency-control module combining a temporal interaction learning module (TIL) with a 'stereo deviation strength' scalar added as a conditioning embedding. The authors report strong image-level results on RealEstate10K (e.g., PSNR 32.26, SSIM 0.916 at t=5) and the best FVD and Ewarp scores on a privately collected 100-video benchmark, concluding that the method beats AVP and open-source stereo video synthesis methods.

Significance. If the claims were fully supported, the contribution would be significant: it offers a plausible way to train stereo-video synthesis without paired ground-truth stereo data, and the RealEstate10K image results are internally consistent with the ablation study and show large margins over several baselines. The DVG idea of using optical-flow-refined occlusion masks is a useful engineering contribution, and the supplementary ablations of depth and motion estimators are a good practice. However, the central video-level claim of geometric stereo correctness and superiority over AVP is not currently backed by a metric that measures stereo geometry; this gap must be closed before the headline conclusion can be accepted.

major comments (5)
  1. [Sec. 4.1 / Table 2] The video evaluation is not sufficient for the paper's central claim. FVD is computed between the input monocular video and the synthesized target-view video, but those two videos are supposed to differ by camera viewpoint and parallax; a model that produces only tiny deviations from the input could score well on FVD without producing any meaningful stereo effect. Ewarp measures temporal warp consistency within the generated sequence and does not check epipolar geometry, disparity plausibility, or binocular occlusion structure between the left and right outputs. Since the test set is a private 100-video collection with no stereo ground truth, no reported number in Table 2 verifies that the output is geometrically correct stereo video. Please add a stereo-specific evaluation (e.g., left-right disparity consistency, epipolar error, or comparison against real stereo pairs from a public dataset such as KITTI or DrivingStereo), and report error bars or significance tests over the test videos.
  2. [Sec. 3.2 / Sec. 4.6] The training procedure is partly self-referential for the video claim. DVG builds pseudo-stereo pairs from monocular depth, forward-backward rendering, and inpainting; occluded content in the target view is hallucinated by the inpainting model, so the training signal can teach the network to produce plausible and temporally smooth parallax effects rather than true binocular geometry. The paper's own Limitations section (Sec. 4.6) concedes that depth accuracy and distribution directly affect stereoscopic quality, but no experiment quantifies this dependence against real stereo data. I would like to see an experiment that measures the geometric error of the synthesized stereo pairs with respect to known camera geometry, or at least an evaluation on a real stereo video benchmark, so that the 'geometric consistency' claim is not evaluated only through the same pipeline that generated the training data.
  3. [Sec. 4.4 / Table 1] The RealEstate10K image evaluation does not actually use a stereo baseline. The t=5 and t=10 settings compare the current frame with a future monocular frame, which is a wide-baseline novel-view-synthesis benchmark rather than a typical small-baseline stereo pair. This makes the comparison favorable or unfavorable in ways that are not controlled, and it does not validate the method's behavior at the small baselines used in the video demonstration. In addition, Table 1 has missing entries for several methods at t=10 (e.g., NVSVDE-Net, MVSplat), so the per-condition comparisons are incomplete. Please report results at a proper stereo baseline (e.g., adjacent frames or a fixed small pose offset) and complete the table.
  4. [Sec. 3.4.2 / Eq. (4)] The 'stereo deviation strength' is defined as s(z) = ||z0 - z_ref||_2 in latent space, but this is not a measure of binocular disparity or scene depth; it is an aggregate L1 difference between latent features. The paper states that this metric 'quantitatively assesses the binocular disparity in a scene', but no connection is shown between this scalar and any actual geometric quantity such as disparity range, baseline, or epipolar geometry. The stereo-aware loss in Eq. (5) therefore supervises a latent-difference quantity, not stereo geometry. This weakens the claim that the consistency-control module 'ensures geometric consistency', because the training signal does not explicitly enforce any stereo constraint.
  5. [Supplementary Table 6] The depth-estimation ablation reports FVD 62.50 for DepthCrafter versus 67.09 for the proposed MiDaS-based configuration, meaning that the final chosen configuration is not the best according to the headline video metric. This is not necessarily wrong, but it undercuts the argument that FVD is the appropriate ranking metric for this task. Please explain the selection of MiDaS and report stereo-specific metrics (e.g., disparity error or left-right consistency) for each depth estimator; otherwise the reader cannot tell whether the chosen configuration is better in the property that actually matters, namely stereo correctness.
minor comments (6)
  1. [Sec. 1] There is a typo 'ate-of-the-art' in the contributions paragraph, and the styling 'A VP' should be 'AVP' throughout.
  2. [Figure 3] Figure 3 contains untranslated Chinese placeholder text ('在此处键入公式。') and should be cleaned before publication.
  3. [Eq. (2)] Equation (2) is ambiguous: the mask m_t(i,j) appears on both sides of the definition, and the threshold '≥ 1' over a sum of values weighted by a confidence map C(i,j) is not clearly motivated; please clarify the notation and the intended merging rule.
  4. [Table 4] Table 4 has empty cells for several baselines, and the cost table mixes 'DVG' and 'Inference' rows in an inconsistent way; please complete the table or report only the entries that were actually measured.
  5. [Sec. 4.1 / Sec. 7.4] The private 100-video test set is not described in sufficient detail (resolution, duration, scene types, and camera baseline for the expected stereo pairs), which limits reproducibility; the paper should report these statistics or release the test set.
  6. [Sec. 5 / Table 4] The conclusion claims the method 'meets the demand of real-world application', but Table 4 reports 1059 seconds of inference for a 30-frame 1024x1024 clip and the Limitations section acknowledges the large parameter size; this claim should be qualified.

Circularity Check

2 steps flagged · score 6.0 of 10

Stereo-video claims are self-referential: the stereo deviation condition is defined from the target latent it must predict, and the video benchmark measures FVD/Ewarp, not stereo geometry, so 'beats AVP' is not independently grounded.

  1. self definitional [Section 3.4.2, Eqs. (4)-(7), Figure 4]
    "Consequently, we introduce a metric called the stereo deviation strength, which quantitatively assesses the binocular disparity in a scene and facilitates the creation of controllable stereo vision: s(z) = |z0 − zref| (4) ... a stereo-aware loss function is proposed to directly supervise the disparity difference: ld = ||s(z0) − s(ˆz0)||2 2 (5) ... where ˆz0 represents the estimated clean video latent z0, which can be obtained by [Eq. 6]."

    The quantity used as the geometric condition is defined as the latent distance between the target latent z0 and the reference latent zref. Eq. (6) defines hat_z0 as the estimated clean latent z0, so the conditioning signal is extracted from the very target the denoising U-Net must predict. The stereo-aware loss then only penalizes a mismatch between the predicted latent's deviation and the ground-truth target's deviation, which is already entailed by the reconstruction loss l_epsilon. No independent binocular, disparity, or depth signal enters the supervision, and at inference the paper does not estimate s(z0) from the monocular input; it is added like a time-step embedding.

  2. other [Section 3.2 and Section 4.1, Table 2; Supplementary Table 6]
    "Secondly, the reference view image x1 is rendered into a masked one under the target viewpoint P2, and then the trained inpainting model [58] is used to fill these occluded regions to obtain the novel viewpoint x2 [43]. ... We employ the FVD [62] score to measure the perceptual similarity between input videos and outputs and report the flow warping error Ewarp [35] to assess the temporal consistency of the resulting video sequences."

    The pseudo ground-truth target view used for training is generated by monocular depth estimation plus DIBR and single-image inpainting, so occluded target-view content is hallucinated rather than measured. The self-collected video test set has no stereo ground truth, and neither FVD nor Ewarp checks epipolar or disparity correctness: FVD is a distributional video metric and Ewarp is a monocular temporal-warp error. Consequently, the central claim of geometric consistency and the claim of beating AVP are validated against the same kind of pseudo-stereo signal the model was fitted to reproduce. Supplementary Table 6 illustrates the disconnect: DepthCrafter gives FVD 62.50 versus MiDaS 67.09, so the FVD ranking does not track stereo fidelity.

full rationale

The paper's strongest claim is that SpatialDreamer achieves geometrically and temporally consistent stereo video and outperforms AVP and open-source stereo video synthesis methods. That specific claim is supported by a self-referential evidence chain. First, the stereo deviation strength used as the geometric condition and supervised by Eq. (5) is defined as the latent distance to the target latent z0, and Eq. (6) identifies z0 as the very latent being estimated; the geometric supervision is therefore an auxiliary restatement of the reconstruction objective rather than an independent constraint. Second, the DVG training pairs are produced by depth estimation, forward-backward rendering, and inpainting, meaning occluded target content is hallucinated, and the video benchmark contains no stereo ground truth. FVD and Ewarp measure distributional realism and temporal smoothness, not binocular geometry, so the 'beats AVP' conclusion is not independently grounded for the property actually claimed. The paper's own Section 4.6 concedes that depth accuracy and distribution affect stereoscopic quality, yet no experiment quantifies this against real stereo data. There is no self-citation chain or imported uniqueness theorem here, and the RealEstate10K image-level evaluation against real frames provides genuine independent evidence for single-view spatial synthesis. Nevertheless, the headline stereo-video result reduces in part to reproducing pseudo-stereo targets and to metrics that do not measure stereo correctness, so the overall circularity score is 6.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The method rests on a chain of domain assumptions about monocular depth, rendered pseudo ground truth, optical-flow occlusions, and the link between latent distance and disparity. Three hand-set hyperparameters are used without sensitivity analysis; the stereo deviation strength is an invented conditioning variable with no external validation.

free parameters (3)
  • lambda_ld (stereo-aware loss weight) = 0.001
    Set by hand in Eq. (7); scaling factor combining noise prediction loss and stereo-aware loss; no sensitivity analysis.
  • lambda_TIL (temporal blending weight) = 0.6
    Set by hand in Eq. (3) to blend self-attention and cross-view attention in TIL; no sensitivity analysis.
  • mask refinement threshold = 1
    Threshold in Eq. (2) for merging occlusion masks from adjacent frames; chosen by the authors, no sensitivity analysis.
assumptions (5)
  • domain assumption Monocular depth estimation (MiDaS) provides a disparity signal accurate enough for novel-view rendering.
    DVG renders the target view by warping pixels with the estimated depth; depth errors directly corrupt the pseudo ground truth (Section 3.2, Figure 3).
  • domain assumption Forward-backward rendering plus inpainting produces a valid stereo target view that can serve as ground truth for training.
    The self-supervised framework trains the model to reproduce the inpainted target view; if the inpainted content is geometrically wrong, the model learns wrong geometry (Section 3.2, Section 3.3).
  • domain assumption Optical flow forward-backward consistency identifies occlusion boundaries correctly in dynamic scenes.
    Eq. (2) propagates occlusion masks from neighboring frames using flow confidence C; incorrect flow confidence leads to incorrect mask refinement (Section 3.2).
  • domain assumption L1 distance in VAE latent space correlates with binocular disparity/viewpoint difference.
    Stereo deviation strength s(z)=|z0-zref| is used as a conditioning signal and supervised by Eq. (5); no evidence links latent L1 distance to perceived disparity (Section 3.4.2).
  • domain assumption Pretrained SVD, SD2.1 and CLIP weights provide suitable priors for video diffusion.
    The architecture inherits pretrained weights from SVD and SD2.1; the method assumes these priors transfer to stereo video synthesis (Section 3.5).
invented entities (1)
  • Stereo deviation strength s(z)
    purpose: Conditioning scalar controlling binocular disparity strength in the diffusion model
    Defined as latent L1 difference between reference and target latents (Eq. 4); its link to real-world disparity is untested and no external measurement validates it; the ablation gain is internal.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input." pith.science (2026). https://pith.science/paper/PNBGX5BO

@misc{pith2026241111934,
  author       = {Pith},
  title        = {Pith review of: SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PNBGX5BO}},
  note         = {Machine review of arXiv:2411.11934}
}
read the original abstract

Stereo video synthesis from a monocular input is a demanding task in the fields of spatial computing and virtual reality. The main challenges of this task lie on the insufficiency of high-quality paired stereo videos for training and the difficulty of maintaining the spatio-temporal consistency between frames. Existing methods primarily address these issues by directly applying novel view synthesis (NVS) techniques to video, while facing limitations such as the inability to effectively represent dynamic scenes and the requirement for large amounts of training data. In this paper, we introduce a novel self-supervised stereo video synthesis paradigm via a video diffusion model, termed SpatialDreamer, which meets the challenges head-on. Firstly, to address the stereo video data insufficiency, we propose a Depth based Video Generation module DVG, which employs a forward-backward rendering mechanism to generate paired videos with geometric and temporal priors. Leveraging data generated by DVG, we propose RefinerNet along with a self-supervised synthetic framework designed to facilitate efficient and dedicated training. More importantly, we devise a consistency control module, which consists of a metric of stereo deviation strength and a Temporal Interaction Learning module TIL for geometric and temporal consistency ensurance respectively. We evaluated the proposed method against various benchmark methods, with the results showcasing its superior performance.

Figures

Figures reproduced from arXiv: 2411.11934 by the authors.

Figure 1
Figure 1. Visualization of the temporal consistency in stereo video [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed method, given left view video, the target view video is rendered, encoded, and concatenated with [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. DVG. The temporal motion can be utilized to refine the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Stereo deviation strength guidance examples. Augment [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results on the RealEstate10K dataset. The proposed method generates better-quality information and maintains the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visual comparisons in stereo video benchmark. We show the scanline (yellow line in original frame) slice through the spatial [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Ablation study of different designs. Only the proposed [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: Visual comparisons for the improvement of stereo oc [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Visual result on the RealEstate10K dataset. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Visual result of ablation study. 3 [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Visual comparison of different depth estimation methods. The zoomed image represents occluded regions. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Quantitative comparison with the other methods.The first three columns are adjacent frames with t, t+1, t+2, and the last column [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SpatialMe: Stereo Video Conversion Using Depth-Warping and Blend-Inpainting

    cs.CV 2024-12 conditional novelty 6.0 of 10

    A depth-warp and blend-inpainting framework with a learned multi-branch refiner, plus a 1000-video real-world stereo dataset, converts monocular video into stereo.

  2. DECAF: De-Clustering for Adaptive Representational Unlearning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    DECAF is a forget-only unlearning method that adds input noise, suppresses the forget-class probability, and diversifies outputs, achieving 0.10% forget accuracy and 79.4% retain accuracy on CIFAR-10/ResNet-18 while d...

Reference graph

Works this paper leans on

80 extracted references · 48 canonical work pages · cited by 2 Pith papers

  1. [1]

    Occlusion refinement for stereo video us- ing optical flow

    Dmitry Akimov, Alexey Shestov, Alexander V oronov, and Dmitriy Vatolin. Occlusion refinement for stereo video us- ing optical flow. In 2012 International Conference on 3D Imaging (IC3D), pages 1–8, 2012

    work page 2012

  2. [2]

    visionos 2 brings new spatial computing experiences to apple vision pro

    Andrea Schubert. visionos 2 brings new spatial computing experiences to apple vision pro. https://www.apple. com/newsroom/2024/06/visionos- 2- brings- new - spatial - computing - experiences - to - apple-vision-pro, 2024

    work page 2024

  3. [3]

    Text2live: Text-driven layered image and video editing

    Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kas- ten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In European conference on computer vi- sion, pages 707–723. Springer, 2022

    2022

  4. [4]

    Novel view synthesis with view-dependent effects from a single image

    Juan Luis Gonzalez Bello and Munchurl Kim. Novel view synthesis with view-dependent effects from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024

    work page 2024

  5. [5]

    Zoedepth: Zero-shot transfer by com- bining relative and metric depth, 2023

    Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias M ¨uller. Zoedepth: Zero-shot transfer by com- bining relative and metric depth, 2023

    work page 2023

  6. [6]

    A review of stereo-photogrammetry method for 3-d reconstruction in computer vision

    Phuong Ngoc Binh Do and Quoc Chi Nguyen. A review of stereo-photogrammetry method for 3-d reconstruction in computer vision. In 2019 19th International Symposium on Communications and Information Technologies (ISCIT) , pages 138–143, 2019

    work page 2019

  7. [7]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  8. [8]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023

    2023

Show all 80 references
  1. [9]

    Large occlusion stereo

    Aaron Bobick and Stephen Intille. Large occlusion stereo. International Journal of Computer Vision , 33:181–200, 1999

    1999

  2. [10]

    Video generation models as world simulators

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luh- man, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024

    2024

  3. [11]

    Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing, 2023

    Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xi- aohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mu- tual self-attention control for consistent image synthesis and editing, 2023

    2023

  4. [12]

    pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction

    David Charatan, Sizhe Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In arXiv, 2023

    2023

  5. [13]

    Control-a-video: Controllable text-to-video generation with diffusion models

    Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840, 2023

    2023 arXiv

  6. [14]

    Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images

    Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Marc Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. arXiv preprint arXiv:2403.14627, 2024

    2024 arXiv

  7. [15]

    Animateanything: Fine- grained open domain image animation with motion guid- ance

    Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. Animateanything: Fine- grained open domain image animation with motion guid- ance. arXiv e-prints, pages arXiv–2311, 2023

    2023

  8. [16]

    Egnal and R.P

    G. Egnal and R.P. Wildes. Detecting binocular half- occlusions: empirical comparisons of five approaches. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(8):1127–1133, 2002

    2002

  9. [17]

    Structure and content-guided video synthesis with diffusion models

    Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023

    2023

  10. [18]

    Dynamic view synthesis from dynamic monocular video, 2021

    Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video, 2021

    2021

  11. [19]

    Animatediff: Animate your personalized text-to- image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to- image diffusion models without specific tuning. Interna- tional Conference on Learning Representations, 2024

    2024

  12. [20]

    Photorealistic video generation with diffusion models

    Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and Jos ´e Lezama. Photorealistic video generation with diffusion models. arXiv preprint arXiv:2312.06662, 2023

    2023 arXiv

  13. [21]

    Single- view view synthesis in the wild with learned adaptive mul- tiplane images

    Yuxuan Han, Ruicheng Wang, and Jiaolong Yang. Single- view view synthesis in the wild with learned adaptive mul- tiplane images. In ACM SIGGRAPH 2022 Conference Pro- ceedings, pages 1–8, 2022

    2022

  14. [22]

    Prompt-to-prompt image editing with cross attention control, 2022

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control, 2022

    2022

  15. [23]

    Video diffu- sion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffu- sion models. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2022

    2022

  16. [24]

    Unifying corre- spondence, pose and nerf for pose-free novel view synthesis from stereo pairs, 2024

    Sunghwan Hong, Jaewoo Jung, Heeseong Shin, Jiaolong Yang, Seungryong Kim, and Chong Luo. Unifying corre- spondence, pose and nerf for pose-free novel view synthesis from stereo pairs, 2024

    2024

  17. [25]

    Animate anyone: Consistent and controllable image-to-video synthesis for character animation

    Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.17117, 2023

    2023 arXiv

  18. [26]

    Depthcrafter: Generating consistent long depth sequences for open-world videos

    Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. arXiv preprint arXiv:2409.02095, 2024

    2024 arXiv

  19. [27]

    Slide: Single image 3d photography with soft layering and depth-aware inpainting

    Varun Jampani, Huiwen Chang, Kyle Sargent, Abhishek Kar, Richard Tucker, Michael Krainin, Dominik Kaeser, William T Freeman, David Salesin, Brian Curless, et al. Slide: Single image 3d photography with soft layering and depth-aware inpainting. In Proceedings of the IEEE/CVF In...

    2021

  20. [28]

    Nvist: In the wild new view synthesis from a single image with transformers, 2024

    Wonbong Jang and Lourdes Agapito. Nvist: In the wild new view synthesis from a single image with transformers, 2024

    2024

  21. [29]

    Ocai: Improving optical flow estimation by occlusion and consistency aware interpolation, 2024

    Jisoo Jeong, Hong Cai, Risheek Garrepalli, Jamie Menjay Lin, Munawar Hayat, and Fatih Porikli. Ocai: Improving optical flow estimation by occlusion and consistency aware interpolation, 2024

    2024

  22. [30]

    Multi-view stereo reconstruction of dense shape and complex appear- ance

    Hailin Jin, Stefano Soatto, and Anthony J Yezzi. Multi-view stereo reconstruction of dense shape and complex appear- ance. International Journal of Computer Vision , 63:175– 189, 2005

    2005

  23. [31]

    Repurpos- ing diffusion-based image generators for monocular depth estimation, 2024

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Met- zger, Rodrigo Caye Daudt, and Konrad Schindler. Repurpos- ing diffusion-based image generators for monocular depth estimation, 2024

    2024

  24. [32]

    3d gaussian splatting for real-time radiance field rendering, 2023

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk ¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering, 2023

    2023

  25. [33]

    Text2video-zero: Text-to- image diffusion models are zero-shot video generators.arXiv preprint arXiv:2303.13439, 2023

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to- image diffusion models are zero-shot video generators.arXiv preprint arXiv:2303.13439, 2023

    2023 arXiv

  26. [34]

    Videopoet: A large language model for zero-shot video generation

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos ´e Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023

    2023 arXiv

  27. [35]

    Learning blind video temporal consistency

    Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. Learning blind video temporal consistency. In Proceedings of the European con- ference on computer vision (ECCV), pages 170–185, 2018

    2018

  28. [36]

    Wonderland: Nav- igating 3d scenes from a single image

    Hanwen Liang, Junli Cao, Vidit Goel, Guocheng Qian, Sergei Korolev, Demetri Terzopoulos, Konstantinos Platan- iotis, Sergey Tulyakov, and Jian Ren. Wonderland: Nav- igating 3d scenes from a single image. arXiv preprint arXiv:2412.12091, 2024

    2024 arXiv

  29. [37]

    Infinite na- ture: Perpetual view generation of natural scenes from a sin- gle image

    Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite na- ture: Perpetual view generation of natural scenes from a sin- gle image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021

    2021

  30. [38]

    Sora: A review on background, technology, limitations, and opportunities of large vision models, 2024

    Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jian- feng Gao, Lifang He, and Lichao Sun. Sora: A review on background, technology, limitations, and opportunities of large vision models, 2024

    2024

  31. [39]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  32. [40]

    Cohen, Bryce Evans, Johannes Kopf, and Richard Szeliski

    Kevin Matzen, Michael F. Cohen, Bryce Evans, Johannes Kopf, and Richard Szeliski. Low-cost 360 stereo photogra- phy and video capture. ACM Trans. Graph., 36(4), 2017

    2017

  33. [41]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis, 2020

    2020

  34. [42]

    Multidiff: Consistent novel view synthesis from a single image

    Norman M ¨uller, Katja Schwarz, Barbara R ¨ossle, Lorenzo Porzi, Samuel Rota Bul `o, Matthias Nießner, and Peter Kontschieder. Multidiff: Consistent novel view synthesis from a single image. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition...

    2024

  35. [43]

    Softmax splatting for video frame interpolation, 2020

    Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation, 2020

    2020

  36. [44]

    What is spatial video on iphone 15 pro and vision pro

    Onee. What is spatial video on iphone 15 pro and vision pro. https://xreality.zone/en/posts/what- is- spatial- video- on- iphone- 15- pro- and- apple-vision-pro, 2024

    2024

  37. [45]

    Codef: Content deformation fields for temporally consistent video processing

    Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Jun- tao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. Codef: Content deformation fields for temporally consistent video processing. arXiv preprint arXiv:2308.07926, 2023

    2023 arXiv

  38. [46]

    A survey of structure from motion*

    Onur ¨Ozyes ¸il, Vladislav V oroninski, Ronen Basri, and Amit Singer. A survey of structure from motion*. Acta Numerica, 26:305–364, 2017

    2017

  39. [47]

    Sinmpi: Novel view synthesis from a single image with expanded multiplane images, 2023

    Guo Pu, Peng-Shuai Wang, and Zhouhui Lian. Sinmpi: Novel view synthesis from a single image with expanded multiplane images, 2023

    2023

  40. [48]

    Fatezero: Fus- ing attentions for zero-shot text-based video editing

    Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fus- ing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09535, 2023

    2023 arXiv

  41. [49]

    Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer

    Ren ´e Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 44(3), 2022

    2022

  42. [50]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  43. [51]

    A survey of image labelling for computer vision applications

    Christoph Sager, Christian Janiesch, and Patrick Zschech. A survey of image labelling for computer vision applications. Journal of Business Analytics, 4(2):91–110, 2021

    2021

  44. [52]

    Scharstein

    D. Scharstein. View Synthesis Using Stereo Vision. Springer Berlin Heidelberg, 1999

    1999

  45. [53]

    Structure- from-motion revisited

    Johannes L Schonberger and Jan-Michael Frahm. Structure- from-motion revisited. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 4104–4113, 2016

    2016

  46. [54]

    Seitz, B

    S.M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski. A comparison and evaluation of multi-view stereo reconstruction algorithms. In 2006 IEEE Computer Soci- ety Conference on Computer Vision and Pattern Recognition (CVPR’06), pages 519–528, 2006

    2006

  47. [55]

    Self-supervised vis- ibility learning for novel view synthesis.*

    Yujiao Shi, Hongdong Li, and Xin Yu. Self-supervised vis- ibility learning for novel view synthesis.*. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021

    2021

  48. [56]

    3d photography using context-aware layered depth 10 inpainting

    Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang. 3d photography using context-aware layered depth 10 inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8028– 8038, 2020

    2020

  49. [57]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022

    2022 arXiv

  50. [58]

    Train stable diffusion for inpainting, 2023

    Lorenzo Stacchio. Train stable diffusion for inpainting, 2023

    2023

  51. [59]

    An overview of free view-point depth- image-based rendering (dibr)

    Wenxiu Sun, Lingfeng Xu, Oscar C Au, Sung Him Chui, and Chun Wing Kwok. An overview of free view-point depth- image-based rendering (dibr). In APSIPA Annual Summit and Conference, pages 1023–1030, 2010

    2010

  52. [60]

    Raft: Recurrent all-pairs field transforms for optical flow, 2020

    Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow, 2020

    2020

  53. [61]

    Single-view view synthe- sis with multiplane images

    Richard Tucker and Noah Snavely. Single-view view synthe- sis with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 551–560, 2020

    2020

  54. [62]

    FVD: A new metric for video generation

    Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Rapha¨el Marinier, Marcin Michalski, and Sylvain Gelly. FVD: A new metric for video generation. In Deep Gener- ative Models for Highly Structured Data, ICLR 2019 Work- shop, New Orleans, Louisiana, United States, May 6, ...

    2019

  55. [63]

    Generative camera dolly: Ex- treme monocular dynamic novel view synthesis

    Basile Van Hoorick, Rundi Wu, Ege Ozguroglu, Kyle Sar- gent, Ruoshi Liu, Pavel Tokmakov, Achal Dave, Changxi Zheng, and Carl V ondrick. Generative camera dolly: Ex- treme monocular dynamic novel view synthesis. European Conference on Computer Vision (ECCV), 2024

    2024

  56. [64]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  57. [65]

    Zero-shot video editing using off-the-shelf image diffusion models

    Wen Wang, Yan Jiang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. Zero-shot video editing using off-the-shelf image diffusion models. arXiv preprint arXiv:2303.17599, 2023

    2023 arXiv

  58. [66]

    Learning 3d photography videos via self-supervised diffusion on single images

    Xiaodong Wang, Chenfei Wu, Shengming Yin, Minheng Ni, Jianfeng Wang, Linjie Li, Zhengyuan Yang, Fan Yang, Li- juan Wang, Zicheng Liu, et al. Learning 3d photography videos via self-supervised diffusion on single images. arXiv preprint arXiv:2302.10781, 2023

    2023 arXiv

  59. [67]

    Videocomposer: Compositional video synthesis with motion controllability

    Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Ji- uniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jin- gren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36, 2024

    2024

  60. [68]

    Photo wake-up: 3d character animation from a single photo

    Chung-Yi Weng, Brian Curless, and Ira Kemelmacher- Shlizerman. Photo wake-up: 3d character animation from a single photo. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5908–5917, 2019

    2019

  61. [69]

    Synsin: End-to-end view synthesis from a sin- gle image

    Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. Synsin: End-to-end view synthesis from a sin- gle image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020

    2020

  62. [70]

    Image dis- tortions in stereoscopic video systems

    Andrew Woods, Tom Docherty, and Rolf Koch. Image dis- tortions in stereoscopic video systems. Proc SPIE, 1915, 2002

    1915

  63. [71]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  64. [72]

    Sinnerf: Training neural radiance fields on complex scenes from a single image, 2022

    Dejia Xu, Yifan Jiang, Peihao Wang, Zhiwen Fan, Humphrey Shi, and Zhangyang Wang. Sinnerf: Training neural radiance fields on complex scenes from a single image, 2022

    2022

  65. [73]

    Depth anything: Unleashing the power of large-scale unlabeled data, 2024

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data, 2024

    2024

  66. [74]

    Nvs-solver: Video diffusion model as zero-shot novel view synthesizer

    Meng You, Zhiyu Zhu, Hui Liu, and Junhui Hou. Nvs-solver: Video diffusion model as zero-shot novel view synthesizer. arXiv preprint arXiv:2405.15364, 2024

    2024 arXiv

  67. [75]

    Yu, Fereshteh Forghani, Konstantinos G

    Jason J. Yu, Fereshteh Forghani, Konstantinos G. Derpanis, and Marcus A. Brubaker. Long-term photometric consistent novel view synthesis with diffusion models, 2023

    2023

  68. [76]

    Stereoscopic video synthesis from a monoc- ular video

    Guofeng Zhang, Wei Hua, Xueying Qin, Tien-Tsin Wong, and Hujun Bao. Stereoscopic video synthesis from a monoc- ular video. IEEE Transactions on Visualization and Com- puter Graphics, 13(4):686–696, 2007

    2007

  69. [77]

    Controlvideo: Adding conditional control for one shot text-to-video editing

    Min Zhao, Rongzhen Wang, Fan Bao, Chongxuan Li, and Jun Zhu. Controlvideo: Adding conditional control for one shot text-to-video editing. arXiv preprint arXiv:2305.17098, 2023

    2023 arXiv

  70. [78]

    Stereo magnification: Learning view synthesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. CoRR, abs/1805.09817, 2018. 11 SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input Supplementary Material

    2018 arXiv

  71. [79]

    Visual Comparison with Other Methods More results of comparison with other methods are shown in Figure 9

    Image Synthesis 6.1. Visual Comparison with Other Methods More results of comparison with other methods are shown in Figure 9. 6.2. Visual Result of Ablation Study More results of ablation study are shown in Figure 10. By zooming in on the images, one can more clearly discern ...

  72. [80]

    compare to others

    Video Synthesis 7.1. Computational Cost As shown in Table 4 with our 2.485 billion model. 7.2. Quantitative Impact of Depth and Motion Es- timation Table 5 and Table 6 show the impact of motion estimation methods and depth estimation methods, respectively. 7.3. Quantitative co...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.