Pith. sign in

REVIEW 9 cited by

DiffusionRenderer: Neural Inverse and Forward Rendering with Video Diffusion Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2501.18590 v2 pith:UNW5EBMS submitted 2025-01-30 cs.CV cs.GR

Ruofan Liang , Zan Gojcic , Huan Ling , Jacob Munkberg , Jon Hasselgren , Zhi-Hao Lin , Jun Gao , Alexander Keller
show 3 more authors
Nandita Vijaykumar Sanja Fidler Zian Wang
This is my paper · ORCID
classification cs.CVcs.GR
keywords renderingmodelinversediffusionrenderervideoaccuratelydiffusionediting
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Understanding and modeling lighting effects are fundamental tasks in computer vision and graphics. Classic physically-based rendering (PBR) accurately simulates the light transport, but relies on precise scene representations--explicit 3D geometry, high-quality material properties, and lighting conditions--that are often impractical to obtain in real-world scenarios. Therefore, we introduce DiffusionRenderer, a neural approach that addresses the dual problem of inverse and forward rendering within a holistic framework. Leveraging powerful video diffusion model priors, the inverse rendering model accurately estimates G-buffers from real-world videos, providing an interface for image editing tasks, and training data for the rendering model. Conversely, our rendering model generates photorealistic images from G-buffers without explicit light transport simulation. Experiments demonstrate that DiffusionRenderer effectively approximates inverse and forwards rendering, consistently outperforming the state-of-the-art. Our model enables practical applications from a single video input--including relighting, material editing, and realistic object insertion.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Video Models as Native 4D Renderers: World-Grounded Conditioning from Animated Mesh

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Tracking plus world-position maps in a neural G-buffer outperform depth as a geometric condition for reference-guided video diffusion rendering on a 68-clip synthetic benchmark.

  2. Video Generation Models are General-Purpose Vision Learners

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A video-diffusion backbone fine-tuned as a single-step multi-task perceiver matches or beats specialists on depth, normals, pose and segmentation, with high data efficiency and sim-to-real transfer.

  3. LuxDiT: Lighting Estimation with Video Diffusion Transformer

    cs.GR 2025-09 conditional novelty 6.0 of 10

    A video diffusion transformer fine-tuned on synthetic and real data predicts HDR environment maps from images/videos, cutting peak light-direction error by roughly 45% on sunny outdoor scenes versus DiffusionLight.

  4. InvRGB+L: Inverse Rendering of Complex Scenes with Unified Color and LiDAR Reflectance Modeling

    cs.CV 2025-07 conditional novelty 6.0 of 10

    InvRGB+L jointly estimates visible and LiDAR albedo with a physics-based specular LiDAR model and cross-modal consistency losses, improving inverse rendering and LiDAR intensity simulation for urban and indoor scenes.

  5. UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Jointly predicting albedo and relit appearance with one video-diffusion pass improves relighting fidelity and generalization over two-stage inverse-plus-forward pipelines.

  6. Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Post-trained Cosmos world models generate controllable multi-view driving videos and LiDAR; augmenting real AV training data with these synthetic clips improves downstream perception and policy metrics, especially in ...

  7. MV-CoLight: Efficient Object Compositing with Consistent Lighting and Shadow Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A feed-forward two-stage compositing framework that harmonizes inserted objects across views using a Hilbert-ordered Gaussian color mapping, trained and evaluated on a new 480k-scene synthetic dataset.

  8. Bridging Rendering and Generative Modeling with Monte Carlo Transport Scheduling

    cs.CV 2026-02 reject novelty 5.0 of 10

    A common variance-time SDE aligns Monte Carlo rendering noise with diffusion-model denoising, enabling low-spp render refinement and stage-ordered material control.

  9. StableIntrinsic: Detail-preserving One-step Diffusion Model for Multi-view Material Estimation

    cs.CV 2025-08 conditional novelty 5.0 of 10

    StableIntrinsic estimates albedo, roughness, and metallic maps from multi-view RGB images in a single diffusion step, achieving higher PSNR and lower MSE than prior multi-step diffusion methods.

Pith tools