REVIEW 3 major objections 5 minor 14 references
IntrinsicEdit: Precise generative image manipulation in intrinsic space
T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A diffusion-based editing workflow that operates on albedo, normals, roughness, and irradiance instead of pixels or prompts can perform material edits, object insertion and removal, and relighting with identity preservation.
desk verdict Solid inference-time fix for RGB↔X editing, but the quantitative evaluation is thinner than the claims and the VAE ceiling is never quantified. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the X→RGB latent diffusion model, treated as a neural renderer that maps an initial noise, a set of intrinsic channel latents, and a text embedding to an image latent. The argument is carried by three mechanisms: exact DDIM inversion, which finds the $z_T$ that reproduces the input under fixed conditions; prompt tuning with the loss $\mathcal{L}_{\mathrm{tune}} = \mathbb{E}_{t,\boldsymbol{\varepsilon}} \|\boldsymbol{\varepsilon} - \boldsymbol{\varepsilon}_\theta(z_t, t, c_i, c_p)\|^2$, which absorbs residual identity into $c_p$; and channel-to-prompt transfer with a loss $\mathcal{L}_{\mathrm{transfer}}$, which matches prompt-conditioned predictions to the full-channel prediction so dropped channels (null conditions $\emptyset_i$) are still represented. The key property they exploit is that the recomposer was trained with channel dropout, so it accepts any subset of valid and null intrinsic conditions; this is what makes it possible to edit one channel while the prompt silently carries the others.
What would settle it
Run the full no-edit pipeline on a high-frequency texture (hair, fur, fabric) and compare the reconstruction to the input and to the base VAE's encode–decode reconstruction; if the identity shift is not clearly smaller than the encode–decode baseline, the central claim is falsified. A second check: edit only the albedo in a scene with a conflicting normal channel and see whether the output ghosts; the paper's Figure 11 shows this is exactly where channel-to-prompt transfer must prevent artifacts.
Extended reading notes
Core claim
On its own terms, the central claim is that the intrinsic-image decompose–recompose idea, which was previously blocked by random identity shifts and by the need to update all channels in sync, becomes a usable editor once the recomposer is anchored. The anchor is exact DDIM inversion: treating the recomposition model as a neural renderer $z_0 = \mathrm{X\to RGB}(z_T, c_i, c_p)$, the method solves for the initial noise $z_T$ that reproduces the input latent $z_0$ under the given intrinsic conditions and a tuned prompt. Because such inversion tends to bake image-specific detail into the noise, a prompt-tuning step first moves that detail into the text embedding $c_p$; because the intrinsic channels are entangled, a channel-to-prompt transfer step encodes the dropped channels into $c_p$ as well. The result is a single, deterministic noise anchor plus a prompt that carries all non-edited information, so the user can edit only the albedo (for color, texture, and object removal), albedo with normal (for insertion), roughness, or irradiance (for relighting), and the recomposer resolves shadows, reflections, and indirect lighting automatically.
Load-bearing premise
The load-bearing premise is that the compressed latent representation of the input image retains enough detail for the inverted noise and tuned prompt to reconstruct identity; the paper's own Figure 10 shows that plain encode–decode already loses high-frequency detail, so if that compression loss exceeds the edit tolerance the identity-preservation claim fails regardless of inversion.
Editorial extensions
If this is right
- A single untrained pipeline covers material color, texture, normal, and roughness edits, object insertion and removal, and full-scene relighting, so an artist no longer needs a different specialized model per task.
- Editing albedo alone removes an object together with its cast shadow and reflection, and editing roughness or normal alone propagates the change to reflections and lighting; global illumination is resolved by the recomposer rather than hand-painted.
- Relighting reduces to supplying a new irradiance channel, whether generated by prompting the recomposer, by a shading model such as OutCast, or by decomposing another image, while scene identity stays anchored by the inverted noise.
- The pipeline is deterministic after decomposition, so edits can be composed sequentially on the same noise anchor (remove, insert, re-texture, relight) with repeatable results.
- Performance is bounded by the underlying RGB↔X models: the method targets indoor scenes and inherits their weak spots (people, garments, metals, transparent objects, imperfect decompositions), which the paper documents as limitations.
Reading between the lines
- Editorial inference: if the VAE compression loss is the true identity bottleneck, replacing the base autoencoder with a higher-fidelity or invertible encoder should improve every downstream task; the paper does not test this, but it is directly measurable with the same no-edit reconstruction comparison.
- Editorial inference: the channel-to-prompt transfer recipe is generic—any generative model with dropout-trained conditioning could borrow it to disentangle entangled condition spaces (e.g., depth, normal, segmentation) without retraining.
- Editorial inference: the paper's relighting bootstrapping amounts to a light-transfer operator: decompose a scene, sample a lit version, decompose that, and reuse its irradiance. This suggests a copy-lighting-from-one-photo-to-another tool that the paper only demonstrates implicitly.
- Editorial inference: because the user-facing effort is now a single-channel inpainting or editing step, a dedicated intrinsic-channel inpainting model would make object removal fully automatic and remove the manual mask requirement.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces IntrinsicEdit, a training-free editing framework that operates in the intrinsic-image latent space of the RGB↔X diffusion pipeline. The method couples exact DDIM inversion of the X→RGB renderer with two prompt-embedding optimizations: prompt tuning, which absorbs image-specific identity information before inversion, and channel-to-prompt transfer, which encodes dropped intrinsic channels into the prompt so that only a small selected subset of channels needs to be edited. This is applied to material (color, normal, roughness) editing, object removal and insertion, and relighting, with qualitative comparisons to RGB↔X and several specialized baselines, plus small quantitative evaluations on synthetic material-editing pairs and real-world object-removal pairs. The paper also includes ablations of the inversion method and prompt-optimization components.
Significance. This is a timely and well-motivated extension of the RGB↔X framework, addressing two known deficiencies—identity preservation and intrinsic-channel entanglement—using only inference-time optimization, without fine-tuning or data collection. The qualitative results across diverse editing tasks are compelling, and the method is presented honestly, including a discussion section that attributes residual identity shifts to the base Stable Diffusion VAE. The paper's strengths include the clear formulation of the prompt-tuning and channel-transfer losses, the systematic ablations, and the explicit acknowledgment of the external VAE ceiling. However, the quantitative evaluation is thin (n=10, 4, and 12), and the VAE ceiling is not separated from the method's own edit error, so the central identity-preservation claim is not yet fully quantified.
major comments (3)
- [§5, Fig. 10, §4.2] The paper explicitly states that "the vast majority of identity shifts are caused by the latent-space encoding of the base Stable Diffusion (SD) 2.1 model" and shows in Fig. 10 that both D(E(I)) and even D(D^{-1}(I)) lose high-frequency detail, yet it never quantifies this ceiling on the datasets used for the reported PSNR/LPIPS numbers. In particular, the paper does not state whether the quantitative experiments in §4.2 use z0 = E(I) or z0 = D^{-1}(I), and it does not report a no-edit reconstruction error (e.g., D(E(I)) or D(D^{-1}(I)) against the input) for the same 10, 4, and 12 pairs. Without this decomposition, the reported edit errors conflate the external VAE compression loss with the errors introduced by the inversion, prompt optimization, and editing steps. This is load-bearing for the central claim of pixel-precise, identity-preserving editing. I request that the authors (i) state explicitly how z0 is computed in all experiments, (ii) report no-edit reconstruction PSNR/LPIPS on the quantitative datasets, and (iii) if possible, express edit errors relative to that reconstruction ceiling.
- [§4.2, Fig. 13, Fig. 14] The quantitative evaluation uses only 10 before/after pairs for texture editing, 4 pairs for roughness editing, and 12 pairs for real-world object removal, reporting only mean PSNR and LPIPS with no variance, confidence intervals, or significance tests. With n=4, a single outlier can determine the ranking, and the paper's claim of "state-of-the-art performance" across tasks is not statistically supported. I ask the authors to report per-pair results (or scatter plots), provide standard deviations or confidence intervals, and run a simple significance test (e.g., paired bootstrap or Wilcoxon) for the comparisons where the method is claimed to be best. If the datasets cannot be expanded, the claims in the abstract and conclusion should be tempered to reflect the small scale.
- [§4.1, Relighting] The relighting evaluation is qualitative only, and the target irradiance maps are generated by a bootstrapping procedure using the same X→RGB model that is then evaluated: the authors sample X→RGB with a prompt and all conditions except irradiance, decompose the result with RGB→X, and use the extracted irradiance as the edit target. This makes the relighting demonstration partly self-confirming, since the model is asked to re-render a lighting condition it produced itself. The paper should either add a quantitative relighting comparison with an independent ground truth (e.g., multi-illumination captured data) or explicitly state in the limitations that relighting results are validated only for plausibility and are generated with the same model used for evaluation.
minor comments (5)
- [Fig. 8] The caption says "Naive DIM inversion" but this appears to be a typo for "Naive DDIM inversion."
- [§3.3, Eq. (7)] The channel-to-prompt transfer loss optimizes the prompt to imitate the network's own behavior with the full set of channels; it may be helpful to state explicitly that the transferred prompt is therefore optimal for this particular X→RGB renderer and that physical plausibility is inherited from the model's prior rather than from any independent physical supervision.
- [§4, Implementation details] The hyperparameters λ (in [0.1,10] and [1,10] for roughness) and ω (=1.5, or 6 for roughness) are given without any sensitivity analysis; a small ablation or at least a statement of how these values were chosen would help reproducibility.
- [§5] There is a typo: "identity shifts" appears as "identify shifts" in the first sentence of the "Identity preservation" subsection.
- [Table in §4, Channel organization] The roughness row in the channel table contains a confusing "✔ ✔ / ✘ ✦ ✘" entry; this should be reformatted for readability.
Circularity Check
Partial circularity in relighting bootstrapping and prompt-transfer self-consistency; central quantitative claims rest on independent ground truth.
-
other
[Section 4.1, Relighting, Fig. 5]
"Since the irradiance channel can be challenging to modify manually, we use a bootstrapping approach: We sample X→RGB with all intrinsic conditions except irradiance, using text description of the lighting, until we obtain the desired effect. ... To use the new lighting with our method, we simply decompose the obtained image using RGB→X and use the extracted irradiance channel to relight the original input image using our pipeline, after potential stylistic manipulations to the channel."
The target irradiance is generated by sampling the same X→RGB model that Eq. (9) invokes at synthesis, then decomposed by RGB→X from the model's own output. Re-rendering the original image with that irradiance closes the loop X→RGB → RGB→X → X→RGB; if the two models are approximate inverses, success largely measures the model pair's self-consistency rather than an externally grounded relighting prediction. The third row of Fig. 5 uses OutCast as an external shading model, so the circularity is partial rather than total.
-
other
[Section 3.3, Eq. (7); Section 3.4, Eq. (9)]
"we transfer the information from the dropped channels to the prompt embedding. ... Ltransfer(cp)=E_{t,ε}||ε_θ(z_t, t,{c_i+, c_i−}, ∅_p)−ε_θ(z_t, t,{c_i+, ∅_i}, c_p)||^2 ... zedited_0 = X→RGB(z_T,{cedited_i+, ∅_i}, c_p)."
The prompt c_p is optimized so that the same network ε_θ, given kept channels, reproduces the model's own noise predictions for the dropped channels. The final synthesis then uses that same network with the fitted prompt. 'Preserving dropped-channel information' is thus defined by the model's internal consistency, not by an independent measure. The quantitative object-removal and synthetic material-editing benchmarks provide external checks, so this is a partial fitted-input construction rather than a complete reduction of the central claim.
full rationale
The central editing claims are not circular overall. Exact DDIM inversion (Eq. 5) and prompt optimization (Eqs. 6-8) are indeed fits to the input image and to the same X→RGB model used at inference, but the paper's principal quantitative evidence is independent: object removal is evaluated against real before/after photographs, and material editing against rendered ground-truth pairs. These external benchmarks prevent the identity-preservation claim from reducing to a reconstruction tautology. The self-citation to RGB↔X [Zeng et al. 2024a] is transparent and non-load-bearing for the verdict: the public code and models are used as a component and as a baseline, and the paper also compares against other intrinsic methods (Kocsis et al.), prompt-based editors, and specialized inpainting/compositing tools. No uniqueness theorem is imported from the authors' prior work. The two flagged steps are genuine but secondary. The relighting bootstrapping protocol generates its target irradiance with the same model family used for synthesis, so part of the relighting demonstration is a self-consistency check; the OutCast row provides external grounding. The channel-to-prompt transfer loss Eq. (7) fits a prompt to the same ε_θ used in Eq. (9), making 'preservation' of dropped channels an internal consistency property; the external quantitative benchmarks limit the damage. The paper's own Section 5 and Fig. 10 acknowledge that the SD 2.1 VAE imposes an irrecoverable detail-loss ceiling, which is a correctness/robustness limitation rather than a circularity. Overall, partial circularity exists in the relighting bootstrap and in the self-referential prompt-transfer construction, but the central derivation chain retains independent empirical content, giving a score of 3.
Assumptions & free parameters
free parameters (6)
- lambda (prompt loss balance) =
range [0.1, 10]; lambda=1 for roughness in Fig. 4
- guidance scale omega =
1.5 default; 6 for roughness editing
- number of prompt optimization iterations =
200
- optimization iterations per inversion step =
2-3
- learning rate for prompt optimization =
0.1
- diffusion inference steps =
50
assumptions (4)
- standard math The DDIM sampling and exact DDIM inversion recurrences (Eqs. 3 and 5) correctly describe the X to RGB diffusion model.
- domain assumption The X to RGB model supports arbitrary combinations of valid and null intrinsic conditions because it was trained with channel dropout.
- domain assumption Optimizing the prompt embedding with the noise-prediction loss (Eq. 6) makes the prompt absorb image-identity information missing from the intrinsic channels.
- domain assumption The RGB to X decomposition produces intrinsic channels accurate enough that editing them and re-rendering yields plausible images.
Cite this review
Pith. "Pith review of IntrinsicEdit: Precise generative image manipulation in intrinsic space." pith.science (2026). https://pith.science/paper/KVRHIGLI
@misc{pith2026250508889,
author = {Pith},
title = {Pith review of: IntrinsicEdit: Precise generative image manipulation in intrinsic space},
year = {2026},
howpublished = {\url{https://pith.science/paper/KVRHIGLI}},
note = {Machine review of arXiv:2505.08889}
}
read the original abstract
Generative diffusion models have advanced image editing with high-quality results and intuitive interfaces such as prompts and semantic drawing. However, these interfaces lack precise control, and the associated methods typically specialize on a single editing task. We introduce a versatile, generative workflow that operates in an intrinsic-image latent space, enabling semantic, local manipulation with pixel precision for a range of editing operations. Building atop the RGB-X diffusion framework, we address key challenges of identity preservation and intrinsic-channel entanglement. By incorporating exact diffusion inversion and disentangled channel manipulation, we enable precise, efficient editing with automatic resolution of global illumination effects -- all without additional data collection or model fine-tuning. We demonstrate state-of-the-art performance across a variety of tasks on complex images, including color and texture adjustments, object insertion and removal, global relighting, and their combinations.
Figures
Figures from the paper (23 more)
Reference graph
Works this paper leans on
-
[2]
arXiv preprint arXiv:2307.09481 (2023)
Anydoor: Zero-shot object-level image customization. arXiv preprint arXiv:2307.09481 (2023). Peter Kocsis, Vincent Sitzmann, and Matthias Nießner
arXiv 2023
-
[3]
We compare against two intrinsic-image methods: original RGB→X→RGB [Zeng et al
Color editing B. We compare against two intrinsic-image methods: original RGB→X→RGB [Zeng et al. 2024] and intrinsic image diffusion [Kocsis et al. 2024]. Our method enables precise manipulation of individual material properties while preserving identity and achieving seamless illumination harmonization. In the botom row, we observe that if the texture ed...
work page 2024
-
[5]
arXiv preprint arXiv:2410.08168 (2024)
Zerocomp: Zero-shot object compositing from image intrin- sics via diffusion. arXiv preprint arXiv:2410.08168 (2024). ACM Trans. Graph., Vol. 44, No. 4, Article . Publication date: August
arXiv 2024
-
[6]
We compare against original RGB→X→RGB [Zeng et al
Object removal A. We compare against original RGB→X→RGB [Zeng et al. 2024], Photoshop generative fill [Adobe Inc. 2024], and Stable Diffusion XL inpainting [Stability AI 2023]. Without being specialized for this task, our method performs on par with or better than prior work. It demonstrates a capability to automatically remove shadows and reflections in ...
work page 2024
-
[7]
We compare against two intrinsic-image methods: original RGB→X→RGB [Zeng et al
Color editing A. We compare against two intrinsic-image methods: original RGB→X→RGB [Zeng et al. 2024] and intrinsic image diffusion [Kocsis et al. 2024]. Our method enables precise manipulation of individual material properties while preserving identity and achieving seamless illumination harmonization. ACM Trans. Graph., Vol. 44, No. 4, Article . Public...
work page 2024
-
[10]
We compare against original RGB→X→RGB [Zeng et al
Object removal B. We compare against original RGB→X→RGB [Zeng et al. 2024], Photoshop generative fill [Adobe Inc. 2024], and Stable Diffusion XL inpainting [Stability AI 2023]. Without being specialized for this task, our method performs on par with or better than prior work. It demonstrates a capability to automatically remove shadows and reflections in ...
work page 2024
-
[11]
We compare against original RGB→X→RGB [Zeng et al
Object insertion A. We compare against original RGB→X→RGB [Zeng et al. 2024] and existing object-insertion and intrinsic-based methods: IntrinsicComp [Careaga et al. 2023], ZeroComp [Zhang et al. 2024], Anydoor [Chen et al. 2023], and Poisson cloning [Pérez et al. 2003]. For intrinsic-based methods, we insert the object into the albedo channel. Despite no...
work page 2024
-
[12]
We compare against original RGB→X→RGB [Zeng et al
Object insertion B. We compare against original RGB→X→RGB [Zeng et al. 2024] and existing object-insertion and intrinsic-based methods: IntrinsicComp [Careaga et al. 2023], ZeroComp [Zhang et al. 2024], Anydoor [Chen et al. 2023], and Poisson cloning [Pérez et al. 2003]. For intrinsic-based methods, we insert the object into the albedo channel. Despite no...
work page 2024
Show all 14 references
-
[13]
We compare against original RGB→X→RGB [Zeng et al
Relighting A. We compare against original RGB→X→RGB [Zeng et al. 2024] relighting by changing the input irradiance. Our relighting handles the new lighting condition more naturally and better preserves the identity of the scene content. ACM Trans. Graph., Vol. 44, No. 4, Artic...
2024
-
[14]
We compare against original RGB→X→RGB [Zeng et al
Relighting B. We compare against original RGB→X→RGB [Zeng et al. 2024] relighting by changing the input irradiance. Our relighting handles the new lighting condition more naturally and better preserves the identity of the scene content. ACM Trans. Graph., Vol. 44, No. 4, Artic...
2024
-
[2003]
In ACM SIGGRAPH 2003 Papers
Poisson image editing. In ACM SIGGRAPH 2003 Papers. 313–318. Stability AI
2003
-
[2023]
Intrinsic Harmonization for Illumination-Aware Compositing Supplementary Material. (2023). Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao
2023
-
[2024]
In ACM SIGGRAPH 2024 Conference Papers
Rgb↔ x: Image decomposition and synthesis using material-and lighting-aware diffusion models. In ACM SIGGRAPH 2024 Conference Papers. 1–11. Zitian Zhang, Frédéric Fortier-Chouinard, Mathieu Garon, Anand Bhattad, and Jean- François Lalonde
2024
-
[2025]
arXiv:2505.08889v2 [cs.GR] 15 May 2025 2 • Linjie Lyu, Valentin Deschaintre, Yannick Hold-Geoffroy, Miloš Hašan, Jae Shin Yoon, Thomas Leimkühler, Christian Theobalt, and Iliyan Georgiev Input image Original RGB→X→RGBIntrinsicEdit (ours) Intrinsic image diffusion Fig
2025 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.