pith. sign in

arxiv: 2604.23094 · v1 · submitted 2026-04-25 · 💻 cs.CV · cs.GR· cs.LG

Toward Real-World Adoption of Portrait Relighting via Hybrid Domain Knowledge Fusion

Qian Huang , Mayoore Selvarasa Jaiswal , Zhen Zhong , Rochelle Pereira , Jianyuan Min This is my paper

Pith reviewed 2026-05-08 08:39 UTC · model grok-4.3

classification 💻 cs.CV cs.GRcs.LG
keywords portrait relightingknowledge distillationdomain adaptationsynthetic datasethybrid fusionlightweight modelreal-world adoptionone-light-at-a-time
0
0 comments X

The pith

Hybrid Domain Knowledge Fusion transfers multi-domain expertise into a compact portrait relighting model for real-world use.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Portrait relighting struggles to move from labs to everyday use because of mismatches between synthetic training data and real camera captures, plus the high compute cost of accurate models. This paper proposes fusing knowledge from synthetic, one-light-at-a-time, and real-world datasets through domain-adapted prior models and then distilling that knowledge into a small student network. If successful, the resulting model runs dramatically faster while preserving high visual quality, making real-time relighting practical on consumer devices. The work also releases a large synthetic dataset with detailed ground-truth lighting information to aid further progress.

Core claim

The central claim is that specialized prior models trained on different domains can be adapted and their knowledge distilled into a lightweight student model that inherits multi-domain capabilities, achieving substantial inference speedups without sacrificing state-of-the-art quality on real inputs. This is enabled by the Hybrid Domain Knowledge Fusion paradigm and supported by a new large-scale synthetic dataset with diverse intrinsics.

What carries the argument

Hybrid Domain Knowledge Fusion, a two-stage process of domain-aware adaptation of prior models followed by augmented knowledge distillation to a compact student.

If this is right

  • The compact model enables real-time portrait relighting on edge devices.
  • Quality remains comparable to larger specialized models across domains.
  • The new synthetic dataset provides better training signals for intrinsic decomposition tasks.
  • Inference costs drop by factors of 6 to 240 times compared to prior approaches.
  • This fusion approach generalizes to other image synthesis tasks facing domain gaps.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Applications in mobile photography apps could become feasible without cloud processing.
  • Future work might test the method's robustness to extreme lighting conditions not covered in the datasets.
  • The speedup could allow integration into video relighting pipelines for live streaming.
  • One might explore whether the same fusion technique reduces the need for large real-world capture setups in other vision tasks.

Load-bearing premise

The domain-aware adaptation and augmented knowledge distillation successfully pass on expertise from multiple domains to the student model with no significant drop in quality or introduction of artifacts when tested on real-world images.

What would settle it

Running the lightweight model on a held-out set of real-world portraits captured under varied camera conditions and lighting, and observing if its relit outputs match or exceed the visual fidelity of the original prior models without new artifacts.

Figures

Figures reproduced from arXiv: 2604.23094 by Jianyuan Min, Mayoore Selvarasa Jaiswal, Qian Huang, Rochelle Pereira, Zhen Zhong.

Figure 1
Figure 1. Figure 1: Our method relights portrait images (top) and videos (bottom) under arbitrary HDR environment maps in real time on consumer GPUs. Given inputs from different cameras, our model faithfully reproduces the directionality, color temperature, and complex illumination of diverse target environments (shown as insets) while preserving subject identity. Video relighting is temporally stable across frames. details. … view at source ↗
Figure 2
Figure 2. Figure 2: Overview of Hybrid Domain Knowledge Fusion. Stage 1: Three specialized prior models are trained on heterogeneous data sources—Pphys on synthetic data with unsupervised real-data adaptation via Lconsist, Pref l on synthetic and OLAT data with optical augmentations, and Preal initialized from Pref l and fine-tuned on real data. Stage 2: The frozen priors (P ∗ ) generate domain-routed pseudo-ground truth to s… view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison on the OLAT test set. From left to right: (a) input, (b) ground truth, (c) LUMOS [35], (d) DiffusionRenderer [17], (e) UniRelight [12], (f) IC-Light [37], (g) SwitchLight v3 [15], (h) 3D-Aware Relighting [6] (face crop only), and (i) Ours. Our results most closely match the ground truth in brightness, direc￾tionality, and skin tone across the various in-the-wild HDRs. LPIPS. Notably,… view at source ↗
Figure 4
Figure 4. Figure 4: Relighting fidelity comparison on portraits from cameras of varying quality, relit under a common target HDR environment. From left to right: input, target environment map, and results from DiffusionRenderer [17], UniRelight [12], SwitchLight v3 [15], and Ours. DiffusionRenderer introduces excessive specularity; SwitchLight v3 over￾smooths skin and hair, yielding a synthetic appearance. Our method best pre… view at source ↗
Figure 5
Figure 5. Figure 5: Ablation of prior model components. Each row shows (left to right) the input, the output without the indicated component, and the full model output. Top: without physical augmentations, the model hallucinates lens glare as facial texture. Middle: without albedo consistency losses, ambient light leaks into the relit output. Bottom: without domain-aware normalization, global contrast and textural realism degrade view at source ↗
Figure 6
Figure 6. Figure 6: Ablation of distillation components. From left to right: input, teacher output, baseline distillation (without augmentation or multi-teacher fusion), distillation with augmentation, and the full pipeline. Asymmetric augmentation improves disentangle￾ment of colored illumination; multi-teacher fusion preserves subject identity. targeted augmentations effectively decouple the relighting signal from imaging￾d… view at source ↗
Figure 7
Figure 7. Figure 7: Interactive Interface. The demonstration is recorded from our interactive interface running on a standard laptop with a mobile graphics card (NVIDIA RTX PRO 3000 GPU). A visualization of this interface is provided in view at source ↗
Figure 8
Figure 8. Figure 8: First row: input, target environment lighting. Second row: DiffusionRen￾derer [17], UniRelight [12], IC-Light [37]. Third row: 3D-Aware Relighting [6] (face crop only), SwitchLight v3 [15], Ours view at source ↗
Figure 9
Figure 9. Figure 9: First row: input, target environment lighting. Second row: DiffusionRen￾derer [17], UniRelight [12], IC-Light [37]. Third row: 3D-Aware Relighting [6] (face crop only), SwitchLight v3 [15], Ours. C Additional Details on the Synthetic Dataset Our dataset is rendered utilizing the NVIDIA Omniverse Replicator extension. To ensure the model learns a highly generalized representation, the dataset com￾prises an … view at source ↗
Figure 10
Figure 10. Figure 10: First row: input, target environment lighting. Second row: DiffusionRen￾derer [17], UniRelight [12], IC-Light [37]. Third row: 3D-Aware Relighting [6] (face crop only), SwitchLight v3 [15], Ours view at source ↗
Figure 11
Figure 11. Figure 11: First row: input, target environment lighting. Second row: DiffusionRen￾derer [17], UniRelight [12], IC-Light [37]. Third row: 3D-Aware Relighting [6] (face crop only), SwitchLight v3 [15], Ours view at source ↗
Figure 12
Figure 12. Figure 12: First row: input, target environment lighting. Second row: DiffusionRen￾derer [17], UniRelight [12], IC-Light [37]. Third row: 3D-Aware Relighting [6] (face crop only), SwitchLight v3 [15], Ours view at source ↗
Figure 13
Figure 13. Figure 13: Examples of our synthetic dataset using Digital Human avatars. (a) and (b) are paired images under two different HDRIs, along with the corresponding (c) diffuse albedo map, and (d) camera-space normal maps view at source ↗
read the original abstract

The real-world adoption of portrait relighting is hindered by dataset domain gaps, camera sensitivity, and computational costs. We address these challenges with Hybrid Domain Knowledge Fusion, a paradigm that fuses the specialized strengths of synthetic, One-Light-at-A-Time (OLAT), and real-world datasets into a compact model. Our approach features specialized prior models hardened by domain-aware adaptation, followed by augmented knowledge distillation into a lightweight student model with multi-domain expertise. Our method demonstrates a 6x to 240x inference speedup while maintaining state-of-the-art (SOTA) visual quality in the experiments. Additionally, we construct a massive, high-fidelity synthetic dataset with diverse ground-truth intrinsics to support our training pipeline.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper introduces the Hybrid Domain Knowledge Fusion paradigm for portrait relighting, which fuses specialized strengths from synthetic, OLAT, and real-world datasets. It does so via domain-aware adaptation of prior models followed by augmented knowledge distillation into a lightweight student model that acquires multi-domain expertise. The authors also construct a large-scale synthetic dataset with diverse ground-truth intrinsics. The central claim is that the resulting model achieves 6x to 240x inference speedup while preserving state-of-the-art visual quality.

Significance. If the performance claims are substantiated, the work could meaningfully advance practical deployment of portrait relighting by mitigating domain gaps and computational overhead, with potential benefits for mobile photography, AR/VR, and content creation pipelines. The new synthetic dataset with intrinsics would be a reusable resource for the community.

major comments (1)
  1. Abstract: the assertion of 'state-of-the-art (SOTA) visual quality' and '6x to 240x inference speedup' is the load-bearing claim, yet the abstract supplies no quantitative metrics (e.g., PSNR/SSIM/LPIPS values), baseline comparisons, ablation tables, or error analysis. Without these, it is impossible to determine whether the results support the claim or reflect post-hoc evaluation choices.
minor comments (1)
  1. The phrase 'Hybrid Domain Knowledge Fusion paradigm' is introduced without a concise formal definition or overview diagram; a schematic in §3 or §4 would clarify the flow from prior-model adaptation to student distillation.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. The single major comment highlights an opportunity to strengthen the abstract, and we address it directly below with a commitment to revision.

read point-by-point responses

  1. Referee: Abstract: the assertion of 'state-of-the-art (SOTA) visual quality' and '6x to 240x inference speedup' is the load-bearing claim, yet the abstract supplies no quantitative metrics (e.g., PSNR/SSIM/LPIPS values), baseline comparisons, ablation tables, or error analysis. Without these, it is impossible to determine whether the results support the claim or reflect post-hoc evaluation choices.

    Authors: We agree that the abstract would be more informative with explicit quantitative anchors. The full manuscript reports standard metrics (PSNR, SSIM, LPIPS) and inference timings against multiple baselines in the Experiments section, with ablations and cross-dataset evaluations that follow established protocols in the portrait relighting literature. To address the concern, we will revise the abstract to include concise key results (e.g., average PSNR/LPIPS gains and the observed speedup range relative to prior methods) while preserving brevity. This change will make the central claims immediately verifiable from the abstract itself. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper presents a standard machine-learning pipeline: domain-aware adaptation of specialized prior models trained on synthetic/OLAT/real-world data, followed by augmented knowledge distillation into a compact student model. Claims of 6x–240x speedup and SOTA quality are positioned as empirical outcomes measured on held-out data, not as algebraic identities or fitted parameters renamed as predictions. No equations, self-referential definitions, or load-bearing self-citations appear in the provided text; the construction of an auxiliary synthetic dataset is an input step, not a circular output. The derivation chain therefore remains self-contained and externally falsifiable.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 1 invented entities

The central claim rests primarily on domain assumptions about successful cross-domain adaptation and distillation rather than new mathematical derivations, fitted parameters, or invented physical entities.

axioms (2)
  • domain assumption Specialized prior models trained on individual data domains can be hardened by domain-aware adaptation to contribute useful knowledge across domains.
    Invoked as the first step of the fusion process.
  • domain assumption Augmented knowledge distillation can transfer combined multi-domain expertise into a single lightweight student model while preserving visual quality.
    Central mechanism for achieving the reported inference speedup.
invented entities (1)
  • Hybrid Domain Knowledge Fusion paradigm no independent evidence
    purpose: Framework for fusing strengths of synthetic, OLAT, and real-world datasets into a compact model
    Newly introduced method name and pipeline; no independent evidence outside the paper is provided.

pith-pipeline@v0.9.0 · 5429 in / 1538 out tokens · 60598 ms · 2026-05-08T08:39:38.805482+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 38 canonical work pages · 1 internal anchor

  1. [1]

    In: CVPR

    Afifi, M., Brubaker, M.A., Brown, M.S.: HistoGAN: Controlling colors of GAN- generated and real images via color histograms. In: CVPR. pp. 7941–7950 (2021)

    work page 2021

  2. [2]

    In: IEEE ICSIP

    Alam, M.Z., Giuliani, N., Chen, H., Mantiuk, R.K.: Reduction of glare in images with saturated pixels. In: IEEE ICSIP. pp. 498–502 (2021)

    work page 2021

  3. [3]

    In: ICCVW (2025)

    Anonymous: DNF-Avatar: Distilling neural fields for real-time animatable avatar relighting. In: ICCVW (2025)

    work page 2025

  4. [4]

    In: NeurIPS (2023)

    Bashkirova, D., Ray, A., Mallick, R., Bargal, S.A., Zhang, J., Krishna, R., Saenko, K.: Lasagna: Layered score distillation for disentangled image editing. In: NeurIPS (2023)

    work page 2023

  5. [5]

    Information11(2) (2020)

    Buslaev, A., Iglovikov, V.I., Khvedchenya, E., Parinov, A., Druzhinin, M., Kalinin, A.A.: Albumentations: Fast and flexible image augmentations. Information11(2) (2020)

    work page 2020

  6. [6]

    In: CVPR

    Cai, Z., Jiang, K., Chen, S.Y., Lai, Y.K., Fu, H., Shi, B., Gao, L.: Real-time 3D- aware portrait video relighting. In: CVPR. pp. 6221–6231 (2024)

    work page 2024

  7. [7]

    In: CVPR (2025)

    Chaturvedi, S., Ren, M., Hold-Geoffroy, Y., Liu, J., Dorsey, J., Shu, Z.: SynthLight: Portrait relighting with diffusion model by learning to re-render synthetic faces. In: CVPR (2025)

    work page 2025

  8. [8]

    In: ACM SIGGRAPH

    Debevec, P., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W., Sagar, M.: Ac- quiring the reflectance field of a human face. In: ACM SIGGRAPH. pp. 145–156 (2000)

    work page 2000

  9. [9]

    In: Smart Tools and Apps for Graphics (STAG) (2024)

    Dulecha, T.G., et al.: Optimized NeuralRTI relighting through knowledge distilla- tion. In: Smart Tools and Apps for Graphics (STAG) (2024)

    work page 2024

  10. [10]

    arXiv preprint arXiv:2501.16330 (2025)

    Fang, Y., Sun, Z., Zhang, S., Wu, T., Xu, Y., Zhang, P., Wang, J., Wetzstein, G., Lin, D.: RelightVid: Temporal-consistent diffusion model for video relighting. arXiv preprint arXiv:2501.16330 (2025)

    work page arXiv 2025

  11. [11]

    Communications of the ACM63(11), 139–144 (2020)

    Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. Communications of the ACM63(11), 139–144 (2020)

    work page 2020

  12. [12]

    arXiv preprint arXiv:2506.15673 , year=

    He, K., Liang, R., Munkberg, J., Hasselgren, J., Vijaykumar, N., Keller, A., Fidler, S., Gilitschenski, I., Gojcic, Z., Wang, Z.: UniRelight: Learning joint decomposition and synthesis for video relighting. arXiv preprint arXiv:2506.15673 (2025)

    work page arXiv 2025

  13. [13]

    In: NIPS Deep Learning and Representation Learning Workshop (2015)

    Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NIPS Deep Learning and Representation Learning Workshop (2015)

    work page 2015

  14. [14]

    arXiv preprint arXiv:2510.23494 (2025)

    Jüttner, E., Pfeifer, J., Krath, L., Korfhage, S., Dröge, H., Hullin, M.B., Stam- minger, M., Thies, J.: Yesnt: Are diffusion relighting models ready for cap- ture stage compositing? A hybrid alternative to bridge the gap. arXiv preprint arXiv:2510.23494 (2025)

    work page arXiv 2025

  15. [15]

    In: CVPR (2024)

    Kim, K., et al.: SwitchLight: Co-design of physics-driven architecture and pre- training framework for human portrait relighting. In: CVPR (2024)

    work page 2024

  16. [16]

    Adam: A Method for Stochastic Optimization

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

    work page internal anchor Pith review arXiv 2014

  17. [17]

    Huang et al

    Liang, R., Gojcic, Z., Ling, H., Munkberg, J., Hasselgren, J., Lin, Z.H., Gao, J., Keller, A., Vijaykumar, N., Fidler, S., Wang, Z.: Diffusionrenderer: Neural inverse 16 Q. Huang et al. and forward rendering with video diffusion models. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2025)

    work page 2025

  18. [18]

    arXiv preprint arXiv:2601.22135 (2026)

    Liang, Z., Chen, Z., Chen, Y., Wei, T., Wang, T., Pan, X.: PI-Light: Physics- inspired diffusion for full-image relighting. arXiv preprint arXiv:2601.22135 (2026)

    work page arXiv 2026

  19. [19]

    In: CVPR

    Lin, H.: DreamSalon: A staged diffusion framework for preserving identity-context in editable face generation. In: CVPR. pp. 8589–8598 (2024)

    work page 2024

  20. [20]

    In: CVPR

    Lin, M.H., Reddy, M., Berger, G., Sarkis, M., Porikli, F., Bi, N.: EdgeRelight360: Text-conditioned 360-degree HDR image generation for real-time on-device video portrait relighting. In: CVPR. pp. 831–840 (2024)

    work page 2024

  21. [21]

    In: CVPR

    Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A ConvNet for the 2020s. In: CVPR. pp. 11976–11986 (2022)

    work page 2022

  22. [22]

    In: CVPR

    Mei, Y., He, M., Ma, L., Philip, J., Xian, W., George, D.M., Yu, X., Dedic, G., Taşel, A.L., Yu, N., et al.: Lux post facto: Learning portrait performance relighting with conditional video diffusion and a hybrid dataset. In: CVPR. pp. 5510–5522 (2025)

    work page 2025

  23. [23]

    In: SIGGRAPH Advanced Computer Graphics Animation Course Notes (1984)

    Miller, G.S., Hoffman, C.R.: Illumination and reflection maps: Simulated objects in simulated and real environments. In: SIGGRAPH Advanced Computer Graphics Animation Course Notes (1984)

    work page 1984

  24. [24]

    ACM TOG40(4) (2021)

    Pandey, R., Orts-Escolano, S., Legendre, C., Haene, C., Bouaziz, S., Rhemann, C., Debevec, P., Fanello, S.: Total relighting: Learning to relight portraits for back- ground replacement. ACM TOG40(4) (2021)

    work page 2021

  25. [25]

    In: ACM SIGGRAPH (2024)

    Rao, Y., et al.: Lite2Relight: 3D-aware single image portrait relighting. In: ACM SIGGRAPH (2024)

    work page 2024

  26. [26]

    Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses

    Risser, E., Wilmot, P., Barnes, C.: Stable and controllable neural texture synthesis and style transfer using histogram losses. arXiv preprint arXiv:1701.08893 (2017)

    work page Pith review arXiv 2017

  27. [27]

    In: MICCAI

    Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomed- ical image segmentation. In: MICCAI. pp. 234–241 (2015)

    work page 2015

  28. [28]

    In: ISMAR

    Song, G., Cham, T.J., Cai, J., Zheng, J.: Real-time shadow-aware portrait re- lighting in virtual backgrounds for realistic telepresence. In: ISMAR. pp. 729–738 (2022)

    work page 2022

  29. [29]

    ACM TOG26(3), 37 (2007)

    Talvala, E.V., Adams, A., Horowitz, M., Levoy, M.: Veiling glare in high dynamic range imaging. ACM TOG26(3), 37 (2007)

    work page 2007

  30. [30]

    In: CVPR

    Wang, T.C., Mallya, A., Liu, M.Y.: One-shot free-view neural talking-head syn- thesis for video conferencing. In: CVPR. pp. 10039–10049 (2021)

    work page 2021

  31. [31]

    IEEE TIP (2004)

    Wang, Z., et al.: Image quality assessment: from error visibility to structural sim- ilarity. IEEE TIP (2004)

    work page 2004

  32. [32]

    Optical engineering19(1), 139–144 (1980)

    Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Optical engineering19(1), 139–144 (1980)

    work page 1980

  33. [33]

    In: ICCV

    Wu, Y., He, Q., Xue, T., Garg, R., Chen, J., Veeraraghavan, A., Barron, J.T.: How to train neural networks for flare removal. In: ICCV. pp. 2239–2247 (2021)

    work page 2021

  34. [34]

    In: CVPR (2022)

    Xue, H., Hang, T., Zeng, Y., Sun, Y., Liu, B., Yang, H., Fu, J., Guo, B.: Advancing high-resolution video-language representation with large-scale video transcriptions. In: CVPR (2022)

    work page 2022

  35. [35]

    ACM TOG41(6), 1–15 (2022)

    Yeh, Y.Y., Nagano, K., Khamis, S., Kautz, J., Liu, M.Y., Wang, T.C.: Learning to relight portrait images via a virtual light stage and synthetic-to-real adaptation. ACM TOG41(6), 1–15 (2022)

    work page 2022

  36. [36]

    In: ICCV

    Zhang, L., Zhang, Q., Wu, M., Yu, J., Xu, L.: Neural video portrait relighting in real-time via consistency modeling. In: ICCV. pp. 802–812 (2021)

    work page 2021

  37. [37]

    In: ICLR

    Zhang, L., Rao, A., Agrawala, M.: Scaling in-the-wild training for diffusion-based illumination harmonization and editing by imposing consistent light transport. In: ICLR. pp. 10728–10745 (2025) Toward Real-World Adoption of Portrait Relighting 17

    work page 2025

  38. [38]

    In: CVPR (2018) 18 Q

    Zhang, R., et al.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018) 18 Q. Huang et al. A Video Demonstration Wehighlyencouragereaderstoviewthesupplementaryvideodemonstration.mp4, provided as an ancillary file with this arXiv submission. This video provides comprehensive visual evidence of the relighting fidelity, tem...

    work page 2018