Pith. sign in

REVIEW 4 major objections 4 minor 54 references

IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read IllumiCraft claims that fusing HDR environment maps, synthetically relit frames, and 3D point tracks into a single diffusion transformer yields temporally coherent, prompt-controlled video relighting with better fidelity than prior…

desk verdict Good dataset and architecture, but Eq. 1 inverts the relighting direction, so the paper's central claim is unsupported as written. read the letter →

arxiv 2506.03150 v1 pith:AEZHHOJ2 submitted 2025-06-03 cs.CV cs.AIcs.LGcs.MM

Yuanze Lin , Yi-Wen Chen , Yi-Hsuan Tsai , Ronald Clark , Ming-Hsuan Yang This is my paper
classification cs.CVcs.AIcs.LGcs.MM
keywords videorelightingdiffusionmodelsHDRenvironmentmaps3DpointtrackingcontrollablegenerationDiTtransformergeometryguidanceeditingdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video relighting is hard because shadows and highlights must move with the scene's geometry, yet most generative video models treat illumination as an implicit, uncontrollable factor. IllumiCraft proposes to make light controllable by feeding three kinds of signals into one diffusion transformer: HDR environment maps that record incoming light from every direction, synthetically relit frames optionally paired with a static background reference, and 3D point tracks for geometry. The paper claims that merging these cues teaches the model the joint evolution of appearance and motion under dynamic illumination, yielding more temporally coherent, prompt-aligned relit videos. On its 20,170-video dataset it reports lower FVD (Fréchet Video Distance), better text alignment, and better temporal consistency than IC-Light, AnyV2V, Light-A-Video, and RelightVid.

What carries the argument

IllumiCraft is built on Wan2.1, a transformer-based video diffusion model (DiT). Illumination enters through three learned tokens: a lighting encoder maps the HDR environment video into token embeddings that are added to the text-prompt embedding. Geometry enters through a ControlNet-style condition DiT, a clone of the first four DiT blocks whose outputs pass through zero-initialized linear layers and are added to the main stream, fed with latent codes of 3D tracking videos. Appearance is injected by concatenating latent codes of the relit foreground video and an optional zero-padded background reference frame with the noisy latent of the video being edited. The whole network is trained with the standard denoising loss while the VAE and text encoder stay frozen, and each conditioning stream is randomly dropped during training so all controls remain optional at inference.

What would settle it

Take a validation video that has already been relit under a known prompt, run IllumiCraft on it with a different prompt, and compare the output to both the original video and the model's output on the original input; the claimed relighting role predicts the output should change toward the new prompt, whereas a reconstruction learned from the training loss would tend to preserve the input.

Watch

Extended reading notes

Core claim

The paper's central claim is that explicit geometry is the missing ingredient in video relighting: a diffusion model that sees 3D point tracks alongside illumination cues learns to keep shadows, specular highlights, and reflections attached to scene surfaces as the camera and objects move. It reports that this joint conditioning produces sharper, more prompt-aligned, and more temporally stable relit videos than existing methods, and that the improvement is not merely additive: ablations show that adding geometry guidance to illumination-only training lowers FVD from 1305.45 to 1072.38 in the background-conditioned setting. On the full 49-frame evaluation, the method achieves an FVD of 1072.38 in the background-conditioned setting and 2186.40 in the text-only setting, against 1704.63 and 3946.71 for the strongest compared baselines, with higher text alignment and temporal consistency in both cases.

Load-bearing premise

The load-bearing premise is that the denoising network learns the relighting mapping from the training data even though its training loss is defined as reconstructing the original appearance video, while at inference the user's input video is expected to play the role of the relit condition.

Editorial extensions

If this is right

  • Text-only relighting works without HDR input at inference: training drops the HDR tokens 50% of the time so the prompt embedding alone learns to steer lighting.
  • The same model handles background-conditioned editing by concatenating a static background reference, with the reference dropped 10% of the time during training.
  • On an A6000, a full 49-frame 720x480 video relights in about 105 seconds, faster than the frame-wise and progressive-fusion baselines reported.
  • The 20,170-pair dataset, including original videos, relit videos, HDR maps, and 3D tracking sequences, is itself a reusable resource for geometry-guided video editing.
  • Geometry guidance improves every reported metric over illumination-only training, suggesting 3D point tracks can serve as a general conditioning signal for controllable video generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, one can test whether the learned mapping is genuinely relighting rather than reconstruction by feeding a pre-relit video and a different prompt and checking whether the output changes accordingly.
  • Because the relit training frames are produced by the very baseline method the model is compared against, the upper bound of training-signal quality is set by that baseline; stronger physical relighters or captured relit pairs would likely raise the fidelity ceiling without architectural change.
  • The HDR maps are warped from one first-frame chrome ball, so they encode a static lighting condition; a natural extension is to estimate per-frame HDR maps and see whether dynamic lighting changes, such as a moving light source, become controllable.
  • Stratifying evaluation by scene motion could reveal where geometry guidance matters most: large parallax and articulated motion should benefit more than near-static clips because the 3D tracks carry the most information there.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes IllumiCraft, a unified diffusion model for controllable video relighting. The method accepts three complementary inputs during training: temporally coherent HDR environment maps, synthetically relit foreground videos (optionally paired with a static background reference image), and 3D point tracks. The architecture is built on the Wan2.1 video DiT, with illumination tokens injected into the text embedding and a ControlNet-style cloned branch for 3D geometry conditioning. A data pipeline, IllumiPipe, constructs a 20,170-video dataset from Pexels, augmenting each video with DiffusionLight-based HDR maps, Light-A-Video-generated relit foregrounds, DiffEraser-inpainted backgrounds, SpatialTracker 3D tracks, and CogVLM2-generated captions. Experiments compare against IC-Light, AnyV2V + IC-Light, Light-A-Video, and RelightVid on a 50-video set, reporting FVD, LPIPS, PSNR, CLIP text alignment, and temporal consistency. The paper also reports ablations over the drop rates for HDR, 3D tracking, and reference-image conditioning.

Significance. If the method worked as described, the paper would introduce a useful dataset and a technically interesting architecture for video relighting: the curation of paired videos with HDR maps, masks, backgrounds, and 3D tracks is a potentially valuable resource, and the combination of illumination tokens with ControlNet-style geometry injection into a video DiT is a reasonable design direction. However, the paper's central claim is undermined by a fundamental training/inference mismatch: the model is trained to reconstruct the original appearance video conditioned on the relit foreground, while inference is described as relighting an input appearance video. In addition, the quantitative evaluation lacks a well-defined ground-truth reference, and several design choices are selected on the evaluation set itself. These issues affect the main contribution as stated, not merely its presentation.

major comments (4)
  1. [Section 3.2, Eq. (1), Section 3.4] The training objective in Eq. (1) minimizes denoising error on the appearance latent z = E_VAE(V_appr), conditioned on E = {z_g, z_c, P'} with z_c = Concat(z_rf, z_ref). Thus the model learns the conditional distribution p(V_appr | V_rf, V_ref, V_geo, prompt). However, the inference protocol in Section 3.4 takes an input video (the original appearance) and a text prompt and expects a relit video as output, which would require sampling from p(V_rf | V_appr, V_ref, prompt). The paper never explains how the roles of V_appr and V_rf are swapped at inference, and Figure 1 shows the original video as input to the relighting process. Unless 'input video' is redefined to be the relit foreground—which contradicts both Figure 1 and Section 3.4—the model learns the inverse of the claimed relighting task, and Tables 1 and 2 do not measure what they claim to measure.
  2. [Section 4.1, Tables 1 and 2] The evaluation protocol is not well defined. FVD, LPIPS, and PSNR require either a ground-truth reference video or a reference distribution, but for the 50 test videos no ground-truth relit version exists. The sentence stating that these scores are computed 'against the results of the existing methods' does not specify the reference set. If the metrics are computed between the output and the original input video, then a method that leaves the input unchanged would trivially score well, which is not the intended notion of relighting fidelity. If they are computed by comparing outputs of different methods against each other, they measure distributional differences rather than correctness of relighting. As reported, the numeric advantages in Tables 1 and 2 cannot be interpreted as improvements in relighting fidelity.
  3. [Section 3.1 and Tables 1-2] The relit foreground videos V_rf used as training targets are generated by Light-A-Video, which is also one of the comparison baselines in Tables 1 and 2. Training on Light-A-Video's outputs and then evaluating against Light-A-Video makes the claimed 'better fidelity than Light-A-Video' partially an evaluation against the model's own training signal. The paper should either ablate this effect by using relit annotations from a different source, such as synthetic relighting with known ground truth or a human study, or explicitly acknowledge and analyze this circularity. As it stands, the comparison is not independent.
  4. [Section 4.4 and Appendix D.2, Tables 4-9] The drop rates for HDR maps, 3D tracking videos, and reference images are selected by evaluating FVD, text alignment, and temporal consistency on the same 50-video evaluation set used for the final comparisons in Tables 1 and 2. Selecting hyperparameters on the evaluation set optimizes the reported metrics and makes the selected drop rates (50%, 30%, and 10%, respectively) and the associated numbers optimistically biased. The claim that these drop rates are 'best' is therefore not a valid model-selection result, and the reported ablations do not provide reliable evidence about the contribution of each conditioning signal.
minor comments (4)
  1. [Section 3.3, Eq. (1)] Equation (1) states that ε_θ is a '3D UNet', but Section 3.2 describes the backbone as a DiT/transformer based on Wan2.1. The terminology should be corrected to be consistent.
  2. [Figure 4 and Figure 5 captions] The captions abbreviate Light-A-Video as 'LA V' and cite it as [8], but [8] is RelightVid; Light-A-Video is reference [1]. Please fix these citations to avoid confusion.
  3. [Section 3.2, Latent Feature Extraction] The control latent is defined as z_c = Concat(z_rf, z_ref), but in the text-only inference setting no relit foreground is available. The paper should clarify how z_rf is obtained or replaced at inference, or explicitly explain the intended role assignment between V_appr and V_rf at test time.
  4. [Section 3.1, Appendix B] The HDR environment maps are estimated by warping the first-frame chrome ball with a damped affine camera-motion estimate. The paper does not provide any quantitative validation of temporal coherence or accuracy for these HDR maps, which is important given that they are a core training signal. A simple comparison against per-frame DiffusionLight HDR estimates would strengthen the claim of temporal coherence.

Circularity Check

1 steps flagged · score 6.0 of 10

Training objective and inference protocol invert the relighting direction: the claimed relit output is exactly the conditioning latent z_rf, and the user's input video is the training target V_appr.

  1. self definitional [Section 3.2 (Latent Feature Extraction), Section 3.3 (Eq. 1), Section 3.4 (Inference)]
    "We concatenate the relit foreground latent and reference latent along the channel dimension to form the control latent: zc = Concat(zrf, zref). ... Here ϵθ is the 3D UNet and x denotes the appearance video Vappr. ... At inference time, users provide a text prompt and an input video to relight the scene."

    By Eq. (1), the denoising target is z = E_VAE(V_appr) while the control latent is z_c = Concat(z_rf, z_ref); the model is trained to reconstruct the appearance video from the relit video. Figure 1 and Section 3.4 instead take the appearance video as the user input and call the relit video the output. Therefore the claimed relit output is exactly the conditioning tensor z_rf used in training, and the reported input video is the training target. The claimed prediction p(V_rf | V_appr, prompt) is the inverse of the trained conditional p(V_appr | V_rf, prompt); no redefinition of the inference input as V_rf is stated. The headline relighting result is thus the model's input by construction, not a quantity the stated objective ever learns to produce.

full rationale

The central, load-bearing circularity is the train/inference role inversion. Eq. (1) minimizes noise prediction on z = E_VAE(V_appr) with conditioning set E = {z_g, Concat(z_rf, z_ref), P'}, so the model learns p(V_appr | V_rf, V_ref, V_geo, prompt). Section 3.4 says users give an input video (the appearance video in Figure 1) and expect a relit video, i.e., p(V_rf | V_appr, prompt). The relit output claimed by the paper is therefore the conditioning latent z_rf, and the input video is the training target—an exact swap. Unless the paper redefines the user-supplied 'input video' as V_rf and the output as V_appr (which would be de-relighting, not relighting), the architecture as written cannot produce the claimed predictions; the metrics in Tables 1 and 2 then measure a different task. I did not score the use of Light-A-Video-generated relit targets as a separate circular step because the paper does not explicitly state that the evaluation reference is V_rf; however, it remains a benchmark-validity concern. Hyperparameter drop rates selected on the 50-video evaluation set are a test-set-selection concern, not circularity. Apart from these, the ablation of geometry guidance (I vs I+G) is an honest, non-circular comparison.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The central results depend on tool-generated pseudo-ground truth and several hand-tuned hyperparameters. The most important unverified inputs are Light-A-Video relit outputs as training targets, the DiffusionLight-based HDR warp, and the SpatialTracker geometry. No independent physical ground truth is used.

free parameters (5)
  • X_hdr drop rate = 0.50
    Selected from {0.4, 0.5, 0.6, 0.7} using FVD, text alignment, and temporal consistency on the 50-video evaluation set (Tables 4 and 5), so the final model is tuned on the benchmark used for claims.
  • 3D tracking drop rate = 0.30
    Selected from {0.1, 0.2, 0.3, 0.4} using the same evaluation set (Tables 6 and 7).
  • reference image drop rate = 0.10
    Selected from {0.05, 0.1, 0.2, 0.3} using the same evaluation set (Tables 8 and 9).
  • number of illumination tokens N = 3
    Chosen by hand in Sec. 3.2 with no ablation; all reported results depend on N=3.
  • HDR warp damping alpha = 0.05
    Hand-set in Appendix B.2 to smooth camera motion estimates; no ablation reported.
assumptions (5)
  • domain assumption Light-A-Video relit outputs are valid training targets for video relighting.
    Section 3.1 uses these outputs as the 'relit foreground' supervision. If Light-A-Video has systematic lighting errors, IllumiCraft inherits them, and comparisons against Light-A-Video become partly self-referential.
  • ad hoc to paper Warping the first-frame chrome ball from DiffusionLight with estimated camera motion yields temporally coherent HDR maps.
    Appendix B assumes distant illumination and that affine camera motion from sparse features plus average depth is sufficient; no validation against true HDR captures.
  • domain assumption SpatialTracker 3D point tracks approximate ground-truth scene geometry.
    Section 3.1; no error analysis linking track accuracy to relighting quality, but the method claims geometry guidance helps.
  • domain assumption Wan2.1 pretrained weights provide a valid initialization and frozen VAE and CLIP encoders preserve priors.
    Section 3.3; standard transfer-learning assumption, not independently validated here.
  • standard math Diffusion denoising objective with Gaussian noise is a valid training signal for this conditioning setup.
    Eq. 1; standard DDPM-style loss, no issue itself.
invented entities (1)
  • Learned illumination token matrix X (N=3 tokens)
    purpose: Carries HDR lighting information into the text-prompt embedding of the DiT.
    Section 3.2 introduces X and updates it with encoded HDR features; it is an internal learned representation with no external falsifiable handle, and N=3 is chosen without ablation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation." pith.science (2026). https://pith.science/paper/AEZHHOJ2

@misc{pith2026250603150,
  author       = {Pith},
  title        = {Pith review of: IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AEZHHOJ2}},
  note         = {Machine review of arXiv:2506.03150}
}
read the original abstract

Although diffusion-based models can generate high-quality and high-resolution video sequences from textual or image inputs, they lack explicit integration of geometric cues when controlling scene lighting and visual appearance across frames. To address this limitation, we propose IllumiCraft, an end-to-end diffusion framework accepting three complementary inputs: (1) high-dynamic-range (HDR) video maps for detailed lighting control; (2) synthetically relit frames with randomized illumination changes (optionally paired with a static background reference image) to provide appearance cues; and (3) 3D point tracks that capture precise 3D geometry information. By integrating the lighting, appearance, and geometry cues within a unified diffusion architecture, IllumiCraft generates temporally coherent videos aligned with user-defined prompts. It supports background-conditioned and text-conditioned video relighting and provides better fidelity than existing controllable video generation methods. Project Page: https://yuanze-lin.me/IllumiCraft_page

Figures

Figures reproduced from arXiv: 2506.03150 by the authors.

Figure 1
Figure 1. Given a prompt and input video, IllumiCraft edits scene illumination conditioned on the static background image. It handles a variety of illumination scenarios, including spotlight effects. Abstract Although diffusion-based models can generate high-quality and high-resolution video sequences from textual or image inputs, they lack explicit integration of geo￾metric cues when controlling scene lighting and visual app… view at source ↗
Figure 2
Figure 2. Data collection mechanism of our proposed IllumiPipe. For each input video, our proposed IllumiPipe extracts various types of data: (1) HDR maps, (2) foreground video and mask video, (3) relit video, (4) background video, and (5) 3D tracking video. The details and collection process for each data type are described in Section 3.1. where Vrf ∈ R T ×H×W×3 represents the relit foreground video, Vbg ∈ R T ×H×W×3 is the … view at source ↗
Figure 3
Figure 3. Famework of IllumiCraft. It uses HDR maps, relit foreground video, 3D tracking, and an optional background image to jointly model illumination, appearance, and geometry, then generates videos from an illumination-aware text prompt. The figure shows an example with 3 illumination tokens; HDR maps, background images, and 3D tracking videos are all optional during training. temporally coherent set of 3D point tracks th… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Visual results under the text-conditioned setting. We compare IC-Light [9], AnyV2V [33], Light-A-Video [8] (abbreviated LAV in the figure), and our proposed method, IllumiCraft [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Visual results under the background-conditioned setting. We compare IC-Light [9], AnyV2V [33], Light-A-Video [8], RelightVid [8] and our proposed method, IllumiCraft [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visual results under the text-conditioned setting. We compare IC-Light [9], AnyV2V [33], Light-A-Video [8] (abbreviated LAV in the figure), and our proposed method, IllumiCraft. scenarios, ranging from simple indoor scenes to dramatic, otherworldly effects. This variet…
Figure 7
Figure 7. Figure 7: Visual results under the background-conditioned setting. We compare IC-Light [9], AnyV2V [33], Light-A-Video [8], RelightVid [8] and our proposed method, IllumiCraft [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Visual results of IllumiCraft. Our method produces high-fidelity, prompt-aligned videos that adapt to diverse lighting conditions, including dramatic spotlight effects. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10 [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13 [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14 [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 32 canonical work pages

  1. [1]

    Light-a-video: Training-free video relighting via progressive light fusion.arXiv:2502.08590, 2025

    Yujie Zhou, Jiazi Bu, Pengyang Ling, Pan Zhang, Tong Wu, Qidong Huang, Jinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, et al. Light-a-video: Training-free video relighting via progressive light fusion.arXiv:2502.08590, 2025. 2, 3, 4, 6, 7, 8, 9, 14

    arXiv 2025

  2. [2]

    Wan: Open and advanced large-scale video generative models.arXiv:2503.20314, 2025

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models.arXiv:2503.20314, 2025. 2, 5, 6

    arXiv 2025

  3. [3]

    CogVideoX: Text-to-video diffusion models with an expert transformer.arXiv:2408.06072, 2024

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. CogVideoX: Text-to-video diffusion models with an expert transformer.arXiv:2408.06072, 2024. 2, 3

    arXiv 2024

  4. [4]

    Recovering shape and spatially-varying surface reflectance under unknown illumination.ACM Transactions on Graphics, 35(6):1–12, 2016

    Rui Xia, Yue Dong, Pieter Peers, and Xin Tong. Recovering shape and spatially-varying surface reflectance under unknown illumination.ACM Transactions on Graphics, 35(6):1–12, 2016. 2

    work page 2016

  5. [5]

    Practical svbrdf acquisition of 3d objects with unstructured flash photography.ACM Transactions on Graphics, 37(6):1–12, 2018

    Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. Practical svbrdf acquisition of 3d objects with unstructured flash photography.ACM Transactions on Graphics, 37(6):1–12, 2018. 2

    work page 2018

  6. [6]

    PhySG: Inverse rendering with spherical gaussians for physics-based material editing and relighting

    Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical gaussians for physics-based material editing and relighting. InCVPR,

    work page

  7. [7]

    Real-time 3d-aware portrait video relighting

    Ziqi Cai, Kaiwen Jiang, Shu-Yu Chen, Yu-Kun Lai, Hongbo Fu, Boxin Shi, and Lin Gao. Real-time 3d-aware portrait video relighting. InCVPR, 2024. 2

    work page 2024

  8. [8]

    RelightVid: Temporal-consistent diffusion model for video relighting

    Ye Fang, Zeyi Sun, Shangzhan Zhang, Tong Wu, Yinghao Xu, Pan Zhang, Jiaqi Wang, Gordon Wetzstein, and Dahua Lin. RelightVid: Temporal-consistent diffusion model for video relighting. arXiv:2501.16330, 2025. 2, 3, 6, 7, 8, 14, 15

    arXiv 2025

Show all 54 references
  1. [9]

    IC-Light GitHub Page, 2024

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. IC-Light GitHub Page, 2024. 2, 3, 6, 7, 8, 9, 14, 15

    2024

  2. [10]

    Relightful harmonization: Lighting-aware portrait background replacement

    Mengwei Ren, Wei Xiong, Jae Shin Yoon, Zhixin Shu, Jianming Zhang, HyunJoon Jung, Guido Gerig, and He Zhang. Relightful harmonization: Lighting-aware portrait background replacement. InCVPR, 2024. 2, 3

    2024

  3. [11]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InICCV, 2023. 2, 3, 5, 6

    2023

  4. [12]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InCVPR, 2022. 3

    2022

  5. [13]

    Diffusion models beat GANs on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In NeurIPS, 2021. 3

    2021

  6. [14]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020. 3

    2020

  7. [15]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.02502, 2020. 3

    2010 arXiv

  8. [16]

    InstructPix2Pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. InstructPix2Pix: Learning to follow image editing instructions. InCVPR, 2023. 3

    2023

  9. [17]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. InICCV, 2023. 3, 6

    2023

  10. [18]

    Text-driven image editing via learnable regions

    Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Lu Jiang, and Ming-Hsuan Yang. Text-driven image editing via learnable regions. InCVPR, 2024. 3

    2024

  11. [19]

    DreamFusion: Text-to-3d using 2d diffusion.arXiv:2209.14988, 2022

    Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. DreamFusion: Text-to-3d using 2d diffusion.arXiv:2209.14988, 2022. 3 10

    2022 arXiv

  12. [20]

    DreamPolisher: Towards high-quality text-to-3d generation via geometric diffusion.arXiv:2403.17237, 2024

    Yuanze Lin, Ronald Clark, and Philip Torr. DreamPolisher: Towards high-quality text-to-3d generation via geometric diffusion.arXiv:2403.17237, 2024. 3

    2024 arXiv

  13. [21]

    RenderDiffusion: Image diffusion for 3d reconstruction, inpainting and generation

    Titas Anciukeviˇcius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy J Mitra, and Paul Guerrero. RenderDiffusion: Image diffusion for 3d reconstruction, inpainting and generation. InCVPR, 2023. 3

    2023

  14. [22]

    Bootstrap3D: Improving 3d content creation with synthetic data.arXiv:2406.00093,

    Zeyi Sun, Tong Wu, Pan Zhang, Yuhang Zang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Bootstrap3D: Improving 3d content creation with synthetic data.arXiv:2406.00093,

    arXiv

  15. [23]

    Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction

    Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction. InICCV,

  16. [24]

    Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior

    Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. InICCV, 2023. 3

    2023

  17. [25]

    FlashTex: Fast relightable mesh texturing with lightcontrolnet

    Kangle Deng, Timothy Omernick, Alexander Weiss, Deva Ramanan, Jun-Yan Zhu, Tinghui Zhou, and Maneesh Agrawala. FlashTex: Fast relightable mesh texturing with lightcontrolnet. InECCV, 2025. 3

    2025

  18. [26]

    DiLightNet: Fine-grained lighting control for diffusion-based image generation

    Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. DiLightNet: Fine-grained lighting control for diffusion-based image generation. InSIGGRAPH, 2024. 3

    2024

  19. [27]

    LightIt: Illumination modeling and control for diffusion models

    Peter Kocsis, Julien Philip, Kalyan Sunkavalli, Matthias Nießner, and Yannick Hold-Geoffroy. LightIt: Illumination modeling and control for diffusion models. InCVPR, 2024. 3

    2024

  20. [28]

    Align your latents: High-resolution video synthesis with latent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. InCVPR, 2023. 3

    2023

  21. [29]

    Make-a-video: Text-to-video generation without text-video data.arXiv:2209.14792, 2022

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data.arXiv:2209.14792, 2022. 3

    2022 arXiv

  22. [30]

    MotionClone: Training-free motion cloning for controllable video generation

    Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. MotionClone: Training-free motion cloning for controllable video generation. arXiv:2406.05338, 2024. 3

    2024 arXiv

  23. [31]

    Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv:2311.15127, 2023

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets.arXiv:2311.15127, 2023. 3, 6

    2023 arXiv

  24. [32]

    Olympus: A universal task router for computer vision tasks.arXiv preprint arXiv:2412.09612,

    Yuanze Lin, Yunsheng Li, Dongdong Chen, Weijian Xu, Ronald Clark, and Philip HS Torr. Olympus: A universal task router for computer vision tasks.arXiv preprint arXiv:2412.09612,

    arXiv

  25. [33]

    AnyV2V: A tuning-free framework for any video-to-video editing tasks.arXiv:2403.14468, 2024

    Max Ku, Cong Wei, Weiming Ren, Harry Yang, and Wenhu Chen. AnyV2V: A tuning-free framework for any video-to-video editing tasks.arXiv:2403.14468, 2024. 3, 6, 7, 8, 9, 14, 15

    2024 arXiv

  26. [34]

    BroadWay: Boost your text-to-video generation model in a training-free way.arXiv:2410.06241, 2024

    Jiazi Bu, Pengyang Ling, Pan Zhang, Tong Wu, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. BroadWay: Boost your text-to-video generation model in a training-free way.arXiv:2410.06241, 2024. 3

    2024 arXiv

  27. [35]

    Consistent video-to-video transfer using synthetic dataset.arXiv:2311.00213, 2023

    Jiaxin Cheng, Tianjun Xiao, and Tong He. Consistent video-to-video transfer using synthetic dataset.arXiv:2311.00213, 2023. 3

    2023 arXiv

  28. [36]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. InICCV, 2023. 3 11

    2023

  29. [37]

    Diffusion as shader: 3d-aware video diffusion for versatile video generation control.arXiv:2501.03847, 2025

    Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, et al. Diffusion as shader: 3d-aware video diffusion for versatile video generation control.arXiv:2501.03847, 2025. 3

    2025 arXiv

  30. [38]

    SunStage: Portrait reconstruction and relighting using the sun as a light stage

    Yifan Wang, Aleksander Holynski, Xiuming Zhang, and Xuaner Zhang. SunStage: Portrait reconstruction and relighting using the sun as a light stage. InCVPR, 2023. 3

    2023

  31. [39]

    DifFRelight: Diffusion-based facial performance relighting

    Mingming He, Pascal Clausen, Ahmet Levent Ta¸ sel, Li Ma, Oliver Pilarski, Wenqi Xian, Laszlo Rikker, Xueming Yu, Ryan Burgert, Ning Yu, et al. DifFRelight: Diffusion-based facial performance relighting. InSIGGRAPH Asia, 2024. 3

    2024

  32. [40]

    Switch- Light: Co-design of physics-driven architecture and pre-training framework for human portrait relighting

    Hoon Kim, Minje Jang, Wonjun Yoon, Jisoo Lee, Donghyun Na, and Sanghyun Woo. Switch- Light: Co-design of physics-driven architecture and pre-training framework for human portrait relighting. InCVPR, 2024. 3

    2024

  33. [41]

    DiffusionLight: Light probes for free by painting a chrome ball

    Pakkapon Phongthawee, Worameth Chinchuthakun, Nontaphat Sinsunthithet, Varun Jampani, Amit Raj, Pramook Khungurn, and Supasorn Suwajanakorn. DiffusionLight: Light probes for free by painting a chrome ball. InCVPR, 2024. 4, 9, 13

    2024

  34. [42]

    Video depth anything: Consistent depth estimation for super-long videos

    Sili Chen, Hengkai Guo, Shengnan Zhu, Feihu Zhang, Zilong Huang, Jiashi Feng, and Bingyi Kang. Video depth anything: Consistent depth estimation for super-long videos. arXiv:2501.12375, 2025. 4, 13

    2025 arXiv

  35. [43]

    Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159, 2024

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks.arXiv preprint arXiv:2401.14159, 2024. 4

    2024 arXiv

  36. [44]

    MatAnyone: Stable video matting with consistent memory propagation.arXiv:2501.14677, 2025

    Peiqing Yang, Shangchen Zhou, Jixin Zhao, Qingyi Tao, and Chen Change Loy. MatAnyone: Stable video matting with consistent memory propagation.arXiv:2501.14677, 2025. 4

    2025 arXiv

  37. [45]

    DiffuEraser: A diffusion model for video inpainting.arXiv:2501.10018, 2025

    Xiaowen Li, Haolan Xue, Peiran Ren, and Liefeng Bo. DiffuEraser: A diffusion model for video inpainting.arXiv:2501.10018, 2025. 4

    2025 arXiv

  38. [46]

    SpatialTracker: Tracking any 2D pixels in 3D space

    Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. SpatialTracker: Tracking any 2D pixels in 3D space. InCVPR, 2024. 4

    2024

  39. [47]

    CogVLM2: Visual language models for image and video understanding.arXiv:2408.16500, 2024

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. CogVLM2: Visual language models for image and video understanding.arXiv:2408.16500, 2024. 5

    2024 arXiv

  40. [48]

    Towards accurate generative models of video: A new metric & challenges

    Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv:1812.01717, 2018. 6

    2018 arXiv

  41. [49]

    The unreason- able effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. InCVPR, 2018. 6

    2018

  42. [50]

    Weighted-to-spherically-uniform quality evaluation for omnidi- rectional video.IEEE Signal Processing Letters, 24(9):1408–1412, 2017

    Yule Sun, Ang Lu, and Lu Yu. Weighted-to-spherically-uniform quality evaluation for omnidi- rectional video.IEEE Signal Processing Letters, 24(9):1408–1412, 2017. 6

    2017

  43. [51]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, 2021. 6

    2021

  44. [52]

    Decoupled weight decay regularization.arXiv:1711.05101,

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv:1711.05101,

    arXiv

  45. [53]

    Turquoise waves crash basalt rocks, dull stormy light

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. InCVPR, 2023. 9 12 6 Appendix A Overview In the Appendix, we provide the following content: (a) Details of HDR environment map tra...

    2023

  46. [2048]

    A young man with a beard, …, diffuse daylight

    yielding Y3 ∈R 3×49×1536, and then passed through a depth-wise Conv1d followed by LeakyReLU and squeeze, to collapse the temporal axis into the final outputZ∈R 3×1536. D Additional Experimental Results D.1 Comparison with Existing Methods Text-Conditioned Video Relighting.Figu...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.