Pith. sign in

REVIEW 3 major objections 6 minor 6 cited by

UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Video relighting improves when albedo and relit appearance are predicted together in a single diffusion denoising pass, rather than through separate inverse and forward rendering stages.

desk verdict A genuinely new joint-decomposition-and-synthesis formulation for video relighting, with a strong external benchmark win on MIT; the auto-labeled real-data generalization claim is the weak link. read the letter →

arxiv 2506.15673 v1 pith:XNHJKZA6 submitted 2025-06-18 cs.CV

Kai He , Ruofan Liang , Jacob Munkberg , Jon Hasselgren , Nandita Vijaykumar , Alexander Keller , Sanja Fidler , Igor Gilitschenski
show 2 more authors
Zan Gojcic Zian Wang
This is my paper
classification cs.CV
keywords videorelightingintrinsicdecompositionalbedoestimationdiffusiontransformerjointdenoisingtemporalconsistencyHDRenvironmentlightingsingle-passsynthesis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that relighting a video is better learned as one generative step than as two: rather than first inverse-rendering G-buffers (albedo, normals, depth) and then forward-rendering the relit image, the proposed model denoises the relit video and its albedo together in a single diffusion pass. The central claim is that this joint formulation gives the model an implicit understanding of scene intrinsics, which improves generalization to real-world scenes and temporal consistency. The paper supports the claim with a synthetic dataset of 108k rendered clips and 150k auto-labeled real-world clips, reporting higher PSNR/SSIM/LPIPS than two-stage baselines on the multi-illumination benchmark and preferred by users 84–96% of the time. A skeptical reader should note that the real-world labels come from the authors' own inverse renderer, so the generalization result rests partly on the quality of that labeler.

What carries the argument

The load-bearing mechanism is the concatenated-latent joint denoising pass: the diffusion transformer denoises a single token sequence formed by stacking the latent of the relit video (with HDR lighting features concatenated along the channel dimension) and the latent of the albedo along the temporal/frame dimension, with the input video as conditioning. Type embeddings and binary condition masks tell the transformer which tokens are input, albedo, or relit output, and the training objective sums an $\ell^2$ loss on the relit latent with a ten-times-smaller weight $\lambda_a = 0.1$ on the albedo latent. This single-pass cross-modal self-attention is what lets albedo demodulation act as a prior for relighting.

What would settle it

Re-train the same pipeline on the same 150k real clips but with albedo labels produced by an independent inverse renderer or by averaging multi-illumination captures; if the multi-illumination benchmark scores and the street-scene user preference revert to parity with the two-stage baseline, then the reported generalization gain is carried by the auto-labeler's bias rather than by joint denoising.

Watch

Extended reading notes

Core claim

The paper's central discovery is that relighting and albedo demodulation can be solved as one joint denoising problem rather than sequentially. The model concatenates the latent codes of the input video, the albedo, and the relit video along the temporal dimension, adds learnable type embeddings and condition masks, and fine-tunes a video diffusion transformer to simultaneously predict the relit video and the albedo. On the multi-illumination benchmark the method reaches PSNR 20.76 (SSIM 0.749, LPIPS 0.251), beating the strongest two-stage baseline at 17.29 (0.622, 0.355); on held-out synthetic scenes it reaches 26.97 (0.847, 0.190) versus 26.61 (0.841, 0.222). The paper interprets this as evidence that joint prediction makes the model learn an internal representation of scene structure, reducing the error accumulation that plagues inverse-plus-forward pipelines.

Load-bearing premise

The claim that the model generalizes to real-world scenes rests on the 150k auto-labeled real videos, whose albedo labels come from the authors' own inverse-rendering model fine-tuned on their synthetic data; if that labeler bakes in synthetic shading or shadows, the training signal reinforces rather than corrects domain bias.

Editorial extensions

If this is right

  • Relighting can be done from a single image or video in one generative pass, so the model no longer needs explicit G-buffer estimates and avoids inverse-to-forward error accumulation.
  • Because albedo demodulation is trained jointly, the model transfers to out-of-domain scenes without baking input shadows into the relit output, as shown on urban street scenes.
  • Adding 150k auto-labeled real-world RGB–albedo clips improves perceptual quality on natural scenes beyond what synthetic and multi-illumination data alone provide.
  • The same joint-trained model can be used without its albedo output at inference time, so the albedo head is a training-time prior rather than a runtime requirement.
  • On a 57-frame video the single pass runs in 445.5 seconds, less than the 566.6–780.0 seconds reported for two-stage baselines, because it replaces five inverse passes plus one forward pass.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: apply the same concatenated-latent joint denoising to other coupled inverse/synthesis pairs, such as depth or normals with novel-view synthesis, to see whether the generalization gain is specific to albedo or general to joint intrinsics.
  • The paper's own limitation section concedes that emitting objects, such as lights toggled inside a scene, are out of scope; that boundary follows from conditioning only on environment maps and marks the edge of the joint-denosing claim.
  • Because the real-world albedo labels come from the authors' own inverse renderer fine-tuned on their synthetic data, the generalization story is only as strong as that labeler; an independent albedo ground-truth check on a small real set would be a cheap decisive test.
  • The reported preference for the auto-labeled variant (55% vs 45%, within ±8%) sits inside the noise band, so the perceptual benefit of real-world data may be smaller than the qualitative figures suggest.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents UniRelight, a video diffusion framework that jointly denoises the latent of a relit video and its albedo in a single pass, conditioned on the input video and a target HDR environment map. Training combines a new 108k-clip synthetic dataset, the MIT multi-illumination dataset, and 150k automatically labeled real-world clips. The method is evaluated on a held-out synthetic set and the MIT test set, reporting higher PSNR/SSIM/LPIPS than DiLightNet, NeuralGaffer, DiffusionRenderer, and a Cosmos-backed re-implementation of DiffusionRenderer, together with user-study preferences.

Significance. The headline relighting result is externally grounded on the MIT multi-illumination benchmark with the light-probe masking protocol and on a held-out synthetic set whose assets are disjoint from training. The comparison against a re-implemented DiffusionRenderer on the same Cosmos backbone is a reasonable attempt to isolate the algorithmic contribution. The paper also demonstrates the practical value of joint albedo prediction for avoiding shadow baking. However, the evidence for the auto-labeled real-data contribution is not yet statistically supported, and the claimed temporal-consistency advantage is not measured. If the identified gaps are filled, the contribution would be a solid advance for video relighting.

major comments (3)
  1. [4.2, 4.3, Table 4] The claim that auto-labeled real-world data improves generalization is not supported by the reported statistics. The only quantitative evidence is the StreetScenes user study, where the full model is preferred over the no-auto-labeled variant in 55%±8% of samples (Table 4); this is within roughly one standard deviation of chance. No PSNR/SSIM/LPIPS comparison between these two variants is given on the MIT test set or any other held-out multi-illumination benchmark. Because the pseudo-albedo labels are produced by an inverse renderer that was fine-tuned on the authors' own synthetic data, the auto-labeled training signal may reinforce synthetic-render biases rather than correct them. Please provide metric-based ablation evidence, or weaken the generalization claim accordingly.
  2. [Abstract, Section 5] The abstract and Section 5 claim that UniRelight surpasses previous methods in both visual fidelity and temporal consistency, but no temporal consistency metric is reported anywhere in Tables 1-4 or the appendix. The reported metrics (PSNR, SSIM, LPIPS) are per-frame, and the MIT user study is image-based; the StreetScenes user study asks about shadows and reflections, not temporal coherence. Please add a temporal consistency evaluation (e.g., warping error, temporal flicker metric, or a user study targeting temporal artifacts) or remove the temporal-consistency claim.
  3. [5.2, Table 3] The joint-modeling ablation is evaluated only on SyntheticScenes, where the gain over the no-joint-modeling variant is small (26.97 vs 26.42 PSNR). The qualitative claim of improved out-of-distribution generalization (Figure 5) would be strengthened by a quantitative comparison on MIT or another real-world test set with ground truth. Without this, the paper's central claim that joint decomposition improves generalization across domains rests on a single qualitative figure.
minor comments (6)
  1. [4.3] There is a duplicated article in 'Since the the original input video and the environment map information are unavailable'; it should read 'Since the original input video...'.
  2. [5.1] The phrase 'performs on or par with previous work' contains a typo; it should be 'on par with previous work'.
  3. [Table 2] The two-column layout of Table 2 merges entries such as 'IntrinsicImageDiffusion [32]16.41 0.543 0.395'; adding explicit line breaks or column separators would improve readability.
  4. [4.2] The pseudo-albedo labeler is described as producing 'high-quality albedo estimates', but no quantitative validation of this labeler on independent albedo ground truth is provided; reporting such validation would strengthen the auto-labeling argument.
  5. [References] References [1] and [47] both cite the Cosmos world foundation model platform preprint; they should be consolidated to a single citation to avoid duplicate entries.
  6. [B.1] The text states that classifier-free guidance is not used at inference, while Section 4.3 describes a 10% condition-dropout probability; the relationship between these choices should be explained in one sentence for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the headline relighting results are validated against independent ground truth; the auto-labeling data loop is a data-quality concern, not a circular derivation.

full rationale

The paper's central claim is that jointly denoising albedo and relit video latents (Eqs. 1-2) improves relighting. This is tested on SyntheticScenes (held-out assets) and the MIT multi-illumination benchmark, both with ground-truth relit images, so the PSNR/SSIM/LPIPS numbers do not reduce to the model's own fitted values or to the authors' prior work. The real-world auto-labeling pipeline (Sec. 4.2) uses a re-implemented inverse renderer from the authors' DiffusionRenderer, but the pseudo-albedo labels are used as training targets, not as the evaluation metric; the only ablation for their benefit is a user study on StreetScenes (Table 4) with 45%±8% preference for the base version, which is statistically inconclusive. While this is a legitimate weakness in evidence, it is not circularity: the model is not predicting its own labels, and no equation is equivalent to its inputs by construction. The citations of DiffusionRenderer and Cosmos are methodological rather than load-bearing proofs; no uniqueness theorem or ansatz is imported from self-citations to force the result. Therefore the derivation chain is self-contained against external benchmarks.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on five assumptions: correctness of the diffusion/EDM framework (standard and carried by the pretrained Cosmos model), transfer from procedural synthetic scenes to real scenes, sufficiency of HDR environment-map lighting (emitters explicitly out of scope), accuracy of the authors' own auto-labeler for 150k real clips, and the joint-modeling hypothesis that albedo demodulation helps relighting. The hand-chosen hyperparameters (lambda_a=0.1, data ratios, conditioning drop probabilities, CFG off, 35 steps) are free parameters in the sense that the reported results are not shown to be robust to them. The paper introduces no new physical entities; the three lighting buffers and type embeddings are conditioning representations, not postulates with external falsifiability requirements.

free parameters (5)
  • Albedo loss weight lambda_a = 0.1
    Eq. 2; chosen by hand, no sweep reported, and it directly controls the balance between the two tasks the central claim depends on.
  • Training data mixing ratio (synthetic video : synthetic image : real auto-labeled : MIT) = 8 : 1 : 3 : 2
    Appendix B.1; chosen by hand, never ablated, yet it sets how much of the generalization signal comes from each data source.
  • Conditioning strategy probabilities = 70% joint, 18% with GT albedo, 12% albedo-only; 10% CFG drop on real data
    Section 4.3; chosen by hand, not ablated, and they define what the model learns to condition on.
  • Inference denoising steps and classifier-free guidance setting = 35 steps, CFG disabled
    Appendix B.1; CFG was disabled because it was empirically less accurate, but no quantitative comparison is shown.
  • Two-stage training schedule lengths = 15,000 then 12,000 iterations
    Appendix B.1; stopping points are compute-driven and not validated as sufficient or necessary.
assumptions (5)
  • standard math The EDM-style diffusion framework and the pretrained Cosmos-Predict1 DiT and Cosmos VAE correctly model video latents as used here.
    Section 3 adopts Karras et al. [27] and Cosmos [1, 47] as background; correctness of these pretrained components is assumed, not verified in this paper.
  • domain assumption Procedurally generated synthetic scenes (ground plane, up to three objects plus primitives, environment lighting, path length three) are a sufficient training distribution for real-world relighting.
    Section 4.2 describes the entire synthetic dataset as simple randomized scenes; the claim of generalization to natural videos rests on this transfer.
  • domain assumption Target illumination is fully specified by a distant HDR environment map, with scene-internal emitters out of scope.
    Section 6 explicitly lists emitting objects, such as toggling lights, as a limitation the method cannot handle.
  • domain assumption The re-implemented inverse rendering model (a DiffusionRenderer-style pipeline fine-tuned on the authors' synthetic data) produces albedo pseudo-labels accurate enough to train on 150k real clips.
    Section 4.2, the Real-world auto-labeling paragraph; label accuracy is not validated against any independent albedo ground truth.
  • domain assumption Albedo demodulation is a useful auxiliary objective that improves relighting, so joint denoising helps generalization (the joint-modeling hypothesis).
    Section 1 and Section 4.3 motivate this hypothesis; quantitative support is a 0.55 dB PSNR gain on synthetic data (Table 3) plus qualitative shadow-demodulation results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting." pith.science (2026). https://pith.science/paper/XNHJKZA6

@misc{pith2026250615673,
  author       = {Pith},
  title        = {Pith review of: UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XNHJKZA6}},
  note         = {Machine review of arXiv:2506.15673}
}
read the original abstract

We address the challenge of relighting a single image or video, a task that demands precise scene intrinsic understanding and high-quality light transport synthesis. Existing end-to-end relighting models are often limited by the scarcity of paired multi-illumination data, restricting their ability to generalize across diverse scenes. Conversely, two-stage pipelines that combine inverse and forward rendering can mitigate data requirements but are susceptible to error accumulation and often fail to produce realistic outputs under complex lighting conditions or with sophisticated materials. In this work, we introduce a general-purpose approach that jointly estimates albedo and synthesizes relit outputs in a single pass, harnessing the generative capabilities of video diffusion models. This joint formulation enhances implicit scene comprehension and facilitates the creation of realistic lighting effects and intricate material interactions, such as shadows, reflections, and transparency. Trained on synthetic multi-illumination data and extensive automatically labeled real-world videos, our model demonstrates strong generalization across diverse domains and surpasses previous methods in both visual fidelity and temporal consistency.

Figures

Figures reproduced from arXiv: 2506.15673 by the authors.

Figure 1
Figure 1. Given an input image (top left) or video, our method [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Method overview. Given an input video I and a target lighting configuration (Eldr, Elog, Edir), our method jointly predicts a relit video ˆIE and its corresponding albedo aˆ. We use a pretrained VAE encoder￾decoder pair (E, D) to map input and output videos to a latent space. The latents for the target relit video and albedo are concatenated along the temporal (frame) dimension with the encoded input video. Lighting… view at source ↗
Figure 3
Figure 3. Qualitative comparison on the synthetic dataset and MIT multi-illumination dataset. Our method produces high-quality inter-reflections and shadows in synthetic scenes (top rows). Crucially, on the MIT multi-illumination dataset (bottom rows), it delivers relighting results with higher accuracy than baselines, which fail when faced with complex materials. conditioning strategies. In 12% of training steps, the input v… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison on in-the-wild data. Our method generates more plausible results than the baselines, with higher quality and more realistic appearance [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Ablation on joint modeling. Relighting results on urban street scenes. The orange and green crops highlight regions where the pure relighting model (w/o joint modeling) clearly bakes shadows from the input image into the relit result. Our joint model correctly demodula…
Figure 7
Figure 7. Figure 7: Application for data augmentation. The top left image with green outline is the input image [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Qualitative comparison with IC-Light [70]. We provide the environmental background used for IC-Light conditioning, with the reference environment ball on the left. Our method produces higher-quality and more accurate relighting results. As each of the baselines generat…
Figure 9
Figure 9. Figure 9: Additional qualitative comparison on MIT multi-illumination dataset. Our method consistently achieves more accurate relighting results than all baselines on the MIT multi-illumination dataset, demonstrating strong capability in relighting complex materials. Additional …
Figure 10
Figure 10. Figure 10: Our method produces high-quality albedo and relighting results with realistic specular [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 10
Figure 10. Figure 10: Additional qualitative results on real scenes. Our method provides high-quality albedo estimation and realistic relighting results. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Do Image Editing Models Understand Lighting?

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    New 3DLP benchmark with real-world 1K HDR pairs shows state-of-the-art image editing models vary in physical lighting consistency, with best models close to reality but error-prone in low-light regions.

  2. EvoHIL: Self-Evolving Reward and Flow-Matched Policy Optimization for Robust Human-in-the-Loop Reinforcement Learning

    cs.RO 2026-08 conditional novelty 6.0 of 10

    EvoHIL adapts a success classifier, flow-matched action chunks, and relit replay to keep manipulation policies robust under illumination shift, beating HIL and imitation baselines.

  3. ID-V2V: Identity-Preserving Video Restylization

    cs.CV 2026-07 conditional novelty 6.0 of 10

    ID-V2V restyles video by conditioning a diffusion model on edited keyframes, depth, relit faces, and face normals, so scene edits propagate while facial identity and performance are preserved.

  4. Decoupled Illumination Priors for Spatially Controllable Multi-View Indoor Scene Relighting

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Lume-Palette decouples multi-view indoor relighting into diffusion-based distillation of canonical illumination palettes and casting under receiver-centric 3D lighting maps with asymmetric multi-view conditioning.

  5. Relighting as a Probe of Visual Priors via Augmented Latent Intrinsics

    cs.CV 2026-02 conditional novelty 6.0 of 10

    Semantic encoders can harm relighting, and ALI—fusing dense visual features with latent intrinsics—improves relighting on glossy and specular materials.

  6. RelightAnyone: A Generalized Relightable 3D Gaussian Head Model

    cs.CV 2026-01 conditional novelty 6.0 of 10

    A two-stage model turns flat-lit photos of a new head into a relightable 3D Gaussian avatar, predicting reflectance parameters without needing one-light-at-a-time captures of that person.

Reference graph

Works this paper leans on

74 extracted references · 39 canonical work pages · cited by 6 Pith papers

  1. [2]

    Barron and Jitendra Malik

    Jonathan T. Barron and Jitendra Malik. Shape, illumination, and reflectance from shading. IEEE transactions on pattern analysis and machine intelligence, 37(8):1670–1687, 2014

    work page 2014

  2. [3]

    Harry Barrow, J. M. Tenenbaum, A. Hanson, and E. Riseman. Recovering intrinsic scene characteristics. Comput. Vis. Syst, 2:3–26, 1978

    work page 1978

  3. [4]

    Intrinsic images in the wild

    Sean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in the wild. ACM Transactions on Graphics (TOG), 33(4):159, 2014

    work page 2014

  4. [5]

    Shrisha Bharadwaj, Haiwen Feng, Victoria Abrevaya, and Michael J. Black. GenLit: Reformulating Single-Image Relighting as Video Generation, 2024

    work page 2024

  5. [6]

    Anand Bhattad, Daniel McKee, Derek Hoiem, and D. A. Forsyth. StyleGAN knows normal, depth, albedo, and more, 2023

    work page 2023

  6. [7]

    Anand Bhattad, James Soole, and D.A. Forsyth. Stylitgan: Image-based relighting via latent control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    work page 2024

  7. [8]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023

    arXiv 2023

  8. [9]

    Barron, Ce Liu, and Hendrik P.A

    Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. NeRD: neural reflectance decomposition from image collections. In ICCV, 2021

    2021

Show all 74 references
  1. [10]

    Lensch, and Jan Kautz

    Mark Boss, Varun Jampani, Kihwan Kim, Hendrik P.A. Lensch, and Jan Kautz. Two-shot spatially-varying BRDF and shape estimation. In CVPR, 2020

    2020

  2. [11]

    User-assisted intrinsic images

    Adrien Bousseau, Sylvain Paris, and Frédo Durand. User-assisted intrinsic images. ACM Transactions on Graphics (TOG), 28(5):130, 2009

    2009

  3. [12]

    VideoJAM: Joint appearance-motion representations for enhanced motion generation in video models

    Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin. VideoJAM: Joint appearance-motion representations for enhanced motion generation in video models. arXiv: 2502.02492, 2025

    2025 arXiv

  4. [13]

    DIB-R++: Learning to predict lighting and material with a hybrid differentiable renderer

    Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khalis, Or Litany, and Sanja Fidler. DIB-R++: Learning to predict lighting and material with a hybrid differentiable renderer. In NeurIPS, 2021

    2021

  5. [14]

    Uni-Renderer: Unifying Rendering and Inverse Rendering Via Dual Stream Diffusion, 2025

    Zhifei Chen, Tianshuo Xu, Wenhang Ge, Leyi Wu, Dongyu Yan, Jing He, Luozhou Wang, Lu Zeng, Shunsi Zhang, and Yingcong Chen. Uni-Renderer: Unifying Rendering and Inverse Rendering Via Dual Stream Diffusion, 2025

    2025

  6. [15]

    Acquiring the reflectance field of a human face

    Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, Westley Sarokin, and Mark Sagar. Acquiring the reflectance field of a human face. InProceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 145–156, 2000

    2000

  7. [16]

    Objaverse: A universe of annotated 3D objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3D objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Rec...

    2023

  8. [17]

    Diffusion models beat GANs on image synthesis

    Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 2021. 10

    2021

  9. [18]

    Generative models: What do they know? Do they know things? Let’s find out!, 2024

    Xiaodan Du, Nicholas Kolkin, Greg Shakhnarovich, and Anand Bhattad. Generative models: What do they know? Do they know things? Let’s find out!, 2024

    2024

  10. [19]

    GeoWizard: unleashing the diffusion priors for 3D geometry estimation from a single image

    Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. GeoWizard: unleashing the diffusion priors for 3D geometry estimation from a single image. In ECCV, 2024

    2024

  11. [20]

    Outcast: Outdoor single-image relighting with cast shadows

    David Griffiths, Tobias Ritschel, and Julien Philip. Outcast: Outdoor single-image relighting with cast shadows. Computer Graphics Forum, 41(2):179–193, 2022

    2022

  12. [21]

    Johnson, Edward H

    Roger Grosse, Micah K. Johnson, Edward H. Adelson, and William T. Freeman. Ground truth dataset and baseline evaluations for intrinsic image algorithms. In ICCV, pages 2335–2342. IEEE, 2009

    2009

  13. [22]

    Shape, light, and material decomposition from images using Monte Carlo rendering and denoising

    Jon Hasselgren, Nikolai Hofmann, and Jacob Munkberg. Shape, light, and material decomposition from images using Monte Carlo rendering and denoising. arXiv:2206.03380, 2022

    2022 arXiv

  14. [23]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020

    2020

  15. [24]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  16. [25]

    GaussianShader: 3D Gaussian splatting with shading functions for reflective surfaces

    Yingwenqi Jiang, Jiadong Tu, Yuan Liu, Xifeng Gao, Xiaoxiao Long, Wenping Wang, and Yuexin Ma. GaussianShader: 3D Gaussian splatting with shading functions for reflective surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5322...

    2024

  17. [26]

    Neural Gaffer: Relighting any object via diffusion

    Haian Jin, Yuan Li, Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, and Noah Snavely. Neural Gaffer: Relighting any object via diffusion. InAdvances in Neural Information Processing Systems, 2024

    2024

  18. [27]

    Elucidating the design space of diffusion-based generative models

    Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS, 2022

    2022

  19. [28]

    Repurposing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  20. [29]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything. arXiv:2304.02643, 2023

    2023 arXiv

  21. [30]

    IntrinsiX: High-Quality PBR Generation using Image Priors, 2025

    Peter Kocsis, Lukas Höllein, and Matthias Nießner. IntrinsiX: High-Quality PBR Generation using Image Priors, 2025

    2025

  22. [31]

    LightIt: illumination modeling and control for diffusion models

    Peter Kocsis, Julien Philip, Kalyan Sunkavalli, Matthias Nießner, and Yannick Hold-Geoffroy. LightIt: illumination modeling and control for diffusion models. In CVPR, 2024

    2024

  23. [32]

    Intrinsic image diffusion for single-view material estimation

    Peter Kocsis, Vincent Sitzmann, and Matthias Nießner. Intrinsic image diffusion for single-view material estimation. In arxiv, 2023

    2023

  24. [33]

    Shading annotations in the wild

    Balazs Kovacs, Sean Bell, Noah Snavely, and Kavita Bala. Shading annotations in the wild. In CVPR, pages 6998–7007, 2017

    2017

  25. [34]

    Land and John J

    Edwin H. Land and John J. McCann. Lightness and retinex theory. Josa, 61(1):1–11, 1971

    1971

  26. [35]

    CGintrinsics: Better intrinsic image decomposition through physically-based rendering

    Zhengqi Li and Noah Snavely. CGintrinsics: Better intrinsic image decomposition through physically-based rendering. In ECCV, pages 371–387, 2018

    2018

  27. [36]

    Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image

    Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and svbrdf from a single image. In CVPR, pages 2475–2484, 2020

    2020

  28. [37]

    OpenRooms: an end-to-end open framework for photorealistic indoor scene datasets

    Zhengqin Li, Ting-Wei Yu, Shen Sang, Sarah Wang, Sai Bi, Zexiang Xu, Hong-Xing Yu, Kalyan Sunkavalli, Miloš Hašan, Ravi Ramamoorthi, et al. OpenRooms: an end-to-end open framework for photorealistic indoor scene datasets. arXiv preprint arXiv:2007.12868, 2020

    2007 arXiv

  29. [38]

    EN- VIDR: Implicit differentiable renderer with neural environment lighting

    Ruofan Liang, Huiting Chen, Chunlin Li, Fan Chen, Selvakumar Panneer, and Nandita Vijaykumar. EN- VIDR: Implicit differentiable renderer with neural environment lighting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 79–89, 2023. 11

    2023

  30. [39]

    DiffusionRenderer: Neural inverse and forward rendering with video diffusion models

    Ruofan Liang, Zan Gojcic, Huan Ling, Jacob Munkberg, Jon Hasselgren, Zhi-Hao Lin, Jun Gao, Alexander Keller, Nandita Vijaykumar, Sanja Fidler, and Zian Wang. DiffusionRenderer: Neural inverse and forward rendering with video diffusion models. arXiv: 2501.18590, 2025

    2025 arXiv

  31. [40]

    Photorealistic object insertion with diffusion-guided inverse rendering

    Ruofan Liang, Zan Gojcic, Merlin Nimier-David, David Acuna, Nandita Vijaykumar, Sanja Fidler, and Zian Wang. Photorealistic object insertion with diffusion-guided inverse rendering. In ECCV, 2024

    2024

  32. [41]

    GS-IR: 3D Gaussian splatting for inverse rendering

    Zhihao Liang, Qi Zhang, Ying Feng, Ying Shan, and Kui Jia. GS-IR: 3D Gaussian splatting for inverse rendering. arXiv preprint arXiv:2311.16473, 2023

    2023 arXiv

  33. [42]

    Efros, and Noah Snavely

    Andrew Liu, Shiry Ginosar, Tinghui Zhou, Alexei A. Efros, and Noah Snavely. Learning to factorize and relight a city. In ECCV, 2020

    2020

  34. [43]

    Matrix3D: Large Photogrammetry Model All-in-One, 2025

    Yuanxun Lu, Jingyang Zhang, Tian Fang, Jean-Daniel Nahmias, Yanghai Tsin, Long Quan, Xun Cao, Yao Yao, and Shiwei Li. Matrix3D: Large Photogrammetry Model All-in-One, 2025

    2025

  35. [44]

    Extracting triangular 3D models, materials, and lighting from images

    Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3D models, materials, and lighting from images. arXiv:2111.12503, 2021

    2021 arXiv

  36. [45]

    VideoMat: Extracting PBR Materials from Video Diffusion Models

    Jacob Munkberg, Zian Wang, Ruofan Liang, Tianchang Shen, and Jon Hasselgren. VideoMat: Extracting PBR Materials from Video Diffusion Models. In Eurographics Symposium on Rendering - CGF Track, 2025

    2025

  37. [46]

    A multi-illumination dataset of indoor object appearance

    Lukas Murmann, Michael Gharbi, Miika Aittala, and Fredo Durand. A multi-illumination dataset of indoor object appearance. In 2019 IEEE International Conference on Computer Vision (ICCV), Oct 2019

    2019

  38. [47]

    Cosmos world foundation model platform for physical AI

    NVIDIA. Cosmos world foundation model platform for physical AI. arXiv preprint arXiv:2501.03575, 2025

    2025 arXiv

  39. [48]

    Total relighting: Learning to relight portraits for background replacement

    Rohit Pandey, Sergio Orts-Escolano, Chloe LeGendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul Debevec, and Sean Fanello. Total relighting: Learning to relight portraits for background replacement. ACM Transactions on Graphics (Proceedings SIGGRAPH), 40(4), August 2021

    2021

  40. [49]

    Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, and Martin Stich

    Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, and Martin Stich. OptiX: a general purpose ray tracing engine. ACM Trans. Graph., 29(4), jul 2010

    2010

  41. [50]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022

    2022 arXiv

  42. [51]

    Multi-view relighting using a geometry-aware network

    Julien Philip, Michaël Gharbi, Tinghui Zhou, Alexei A Efros, and George Drettakis. Multi-view relighting using a geometry-aware network. ACM Trans. Graph., 38(4):78–1, 2019

    2019

  43. [52]

    DiffusionLight: light probes for free by painting a chrome ball

    Pakkapon Phongthawee, Worameth Chinchuthakun, Nontaphat Sinsunthithet, Amit Raj, Varun Jampani, Pramook Khungurn, and Supasorn Suwajanakorn. DiffusionLight: light probes for free by painting a chrome ball. In ArXiv, 2023

    2023

  44. [53]

    A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis

    Yohan Poirier-Ginter, Alban Gauthier, Julien Philip, Jean-François Lalonde, and George Drettakis. A Diffusion Approach to Radiance Field Relighting using Multi-Illumination Synthesis. Computer Graphics Forum, 2024

    2024

  45. [54]

    NeRF for outdoor scene relighting

    Viktor Rudnev, Mohamed Elgharib, William Smith, Lingjie Liu, Vladislav Golyanik, and Christian Theobalt. NeRF for outdoor scene relighting. In ECCV, 2022

    2022

  46. [55]

    Jacobs, and Jan Kautz

    Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W. Jacobs, and Jan Kautz. Neural inverse rendering of an indoor scene from a single image. In ICCV, 2019

    2019

  47. [56]

    GIR: 3D Gaussian inverse rendering for relightable scene factorization

    Yahao Shi, Yanmin Wu, Chenming Wu, Xing Liu, Chen Zhao, Haocheng Feng, Jingtuo Liu, Liangjun Zhang, Jian Zhang, Bin Zhou, et al. GIR: 3D Gaussian inverse rendering for relightable scene factorization. arXiv preprint arXiv:2312.05133, 2023

    2023 arXiv

  48. [57]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, 2015

    2015

  49. [58]

    Neural light field estimation for street scenes with differentiable virtual object insertion

    Zian Wang, Wenzheng Chen, David Acuna, Jan Kautz, and Sanja Fidler. Neural light field estimation for street scenes with differentiable virtual object insertion. In ECCV, 2022

    2022

  50. [59]

    Learning indoor inverse rendering with 3D spatially-varying lighting

    Zian Wang, Jonah Philion, Sanja Fidler, and Jan Kautz. Learning indoor inverse rendering with 3D spatially-varying lighting. In ICCV, 2021. 12

    2021

  51. [60]

    Neural fields meet explicit geometric representations for inverse rendering of urban scenes

    Zian Wang, Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojcic, Wenzheng Chen, and Sanja Fidler. Neural fields meet explicit geometric representations for inverse rendering of urban scenes. In The IEEE Conference on Computer Vision and Pattern Re...

    2023

  52. [61]

    De-rendering 3D objects in the wild

    Felix Wimbauer, Shangzhe Wu, and Christian Rupprecht. De-rendering 3D objects in the wild. In CVPR, 2022

    2022

  53. [62]

    Chen Xi, Peng Sida, Yang Dongchen, Liu Yuan, Pan Bowen, Lv Chengfei, and Zhou. Xiaowei. Intrinsi- cAnything: learning diffusion priors for inverse rendering under unknown illumination. arxiv: 2404.11593, 2024

    2024 arXiv

  54. [63]

    Luminet: Latent intrinsics meets diffusion models for indoor scene relighting, 2024

    Xiaoyan Xing, Konrad Groh, Sezer Karaoglu, Theo Gevers, and Anand Bhattad. Luminet: Latent intrinsics meets diffusion models for indoor scene relighting, 2024

    2024

  55. [64]

    Ye Yu and William A. P. Smith. InverseRenderNet: learning single image inverse rendering. In CVPR, 2019

    2019

  56. [65]

    Poly Haven - The Public 3D Asset Library, 2024

    Greg Zaal and et al. Poly Haven - The Public 3D Asset Library, 2024

    2024

  57. [66]

    DiLightNet: fine- grained lighting control for diffusion-based image generation

    Chong Zeng, Yue Dong, Pieter Peers, Youkang Kong, Hongzhi Wu, and Xin Tong. DiLightNet: fine- grained lighting control for diffusion-based image generation. In ACM SIGGRAPH 2024 Conference Papers, 2024

    2024

  58. [67]

    RGB↔X: image decomposition and synthesis using material-and lighting-aware diffusion models

    Zheng Zeng, Valentin Deschaintre, Iliyan Georgiev, Yannick Hold-Geoffroy, Yiwei Hu, Fujun Luan, Ling- Qi Yan, and Miloš Hašan. RGB↔X: image decomposition and synthesis using material-and lighting-aware diffusion models. In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024

    2024

  59. [68]

    IRON: inverse rendering by optimizing neural SDFs and materials from photometric images

    Kai Zhang, Fujun Luan, Zhengqi Li, and Noah Snavely. IRON: inverse rendering by optimizing neural SDFs and materials from photometric images. In CVPR, 2022

    2022

  60. [69]

    PhySG: Inverse rendering with spherical Gaussians for physics-based material editing and relighting

    Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. PhySG: Inverse rendering with spherical Gaussians for physics-based material editing and relighting. In CVPR, 2021

    2021

  61. [70]

    Scaling in-the-wild training for diffusion-based illumi- nation harmonization and editing by imposing consistent light transport

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Scaling in-the-wild training for diffusion-based illumi- nation harmonization and editing by imposing consistent light transport. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  62. [71]

    Efros, Eli Shechtman, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018

    2018

  63. [72]

    Freeman, Kai Zhang, and Fujun Luan

    Tianyuan Zhang, Zhengfei Kuang, Haian Jin, Zexiang Xu, Sai Bi, Hao Tan, He Zhang, Yiwei Hu, Milos Hasan, William T. Freeman, Kai Zhang, and Fujun Luan. RelitLRM: Generative relightable radiance for large reconstruction models. In The Thirteenth International Conference on Lear...

    2025

  64. [73]

    Srinivasan, Boyang Deng, Paul Debevec, William T

    Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec, William T. Freeman, and Jonathan T. Barron. NeRFactor: neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (TOG), 40(6):1–18, 2021

    2021

  65. [74]

    A closed-form solution to retinex with nonlocal texture constraints

    Qi Zhao, Ping Tan, Qiang Dai, Li Shen, Enhua Wu, and Stephen Lin. A closed-form solution to retinex with nonlocal texture constraints. TPAMI, 34(7):1437–1444, 2012

    2012

  66. [75]

    What is in this image? Describe the materials. Be concise and produce an answer with a few sentences, no more than 50 words

    Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W. Jacobs. Deep single-image portrait relighting. In ICCV, October 2019. 13 Appendix In this Appendix, we first discuss the broader impact of our project (Sec. A). We then provide additional implementation details of our mode...

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.