{"id":"5073d7fb-2866-4a49-b2fc-ee923696c9c7","arxiv_id":"2412.02322","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A residual-generation latent diffusion model with a ControlNet-style controller and a detail-preserving decoder achieves state-of-the-art shadow removal on the ISTD+ and SRD datasets.","lead":"The authors present a shadow removal method that fine-tunes a pre-trained latent diffusion model to predict the residual between shadowed and shadow-free images, rather than generating the shadow-free image from noise. The approach reports state-of-the-art results on two standard benchmarks while better preserving fine image details such as text and textures.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Inference as written has a sign error at the final reconstruction: Eq. 10 sets r̂_t = z_s − ẑ_t^0, but the text says to add r̂_1 to z_s, yielding 2z_s − ẑ_1^0 instead of z_0; no code is provided.","rationale":"The paper's central claim is conditional on the described inference being executable, and the most distinctive component is the residual generation/refinement procedure. The sign inconsistency between Eq. 10 and the final reconstruction step is therefore load-bearing: a literal implementation of the text cannot produce a shadow-free latent, so the reported quantitative results cannot be reproduced from the paper alone. This is a stronger and more concrete defect than the reader's identified weakest assumption about the pretrained backbone's behavior on residual-mixed latents, because even a perfectly transferring backbone would produce the wrong final output under the literal instruction to add r̂_1 to z_s. The evidence in the experimental section, including the ablations, suggests the authors likely implemented a sign convention in code that differs from the written one; the issue is therefore more consistent with a critical manuscript error than with fabricated results. The appropriate verdict remains conditional, contingent on correcting the sign, clarifying the final reconstruction formula, and ideally releasing code so the inference can be checked. The reader's concern about the noised residual-latent distribution shift is reasonable but secondary; our disagreement is on emphasis, not on the general correctness risk of the paper.","tokens_in":15599,"tokens_out":12261,"duration_ms":132882,"concrete_test":"Implement the inference loop from Section III-A exactly as written, using an oracle network that outputs ε + β̄_t√ᾱ_t r/√(1−ᾱ_t) and ẑ_t^0 = z_0, on any ISTD+ test image. Run the final step both ways: (A) z_s + r̂_1 as the text instructs, and (B) z_s − r̂_1 (equivalently z′_0 from Eq. 11). If (A) does not reproduce z_0 while (B) does, the manuscript contains a sign error that must be corrected, and the reported results should be attributed to the corrected version.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In the proposed residual schedule, r = z_s − z_0 (Eq. 5), and the final shadow-free latent must be z_0 = z_s − r. The paper's Eq. 10 defines the estimated residual as r̂_t = z_s − ẑ_t^0. At the final step, Eq. 11 with β̄_0 = 0 gives z′_0 = z_s − r̂_1 = ẑ_1^0, which is the correct prediction if ẑ_1^0 ≈ z_0. However, Section III-A then states 'we directly add the estimated shadow residual r̂_1 to z_s to yield a shadow-free latent', i.e. z_s + r̂_1 = 2z_s − ẑ_1^0. This is not z_0 unless z_s = z_0. The discrepancy is not cosmetic: if a reader implements the equations literally, the final output is a contrast-inverted sum of the shadow and predicted shadow-free latents, so the reported PSNR/LPIPS/FID numbers cannot be reproduced from the described inference. Since no code is released, the exact final step — subtract residual (Eq. 11) vs. add residual (text) — cannot be disambiguated. This is a more basic threat to the central claim than the transferability of the pretrained noise estimator: the sampling algorithm as written is internally inconsistent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a latent-diffusion shadow-removal framework that generates and refines image residuals instead of regenerating shadow-free images from scratch. It keeps a pretrained Paint-by-Example inpainting backbone, adds a ControlNet-style branch, introduces a shadow residual schedule into the forward diffusion process, trains with an EMA-based cross-timestep self-enhancement scheme, and augments the VQGAN decoder with a control branch and multi-scale skip connections. Experiments on ISTD+ and SRD report improved LPIPS and FID scores over prior mask-free methods, competitive PSNR/SSIM, and an extension to mask-available shadow removal.","tokens_in":15913,"tokens_out":6199,"duration_ms":68638,"significance":"If the described algorithm is reproducible as written, the paper makes a useful contribution: the residual-generation formulation is a principled way to retain the generative prior of a large pretrained diffusion model while preserving input fidelity, and the cross-timestep self-enhancement strategy plus the detail-preserving decoder are supported by the ablation study. The reported LPIPS/FID gains on ISTD+ and SRD are substantial, and the mask-available variant is competitive with current state of the art. However, the manuscript as written contains an internal sign inconsistency in the final sampling reconstruction, inconsistent noise-schedule notation, and no released code, so the empirical claims cannot currently be reproduced from the text. These issues are fixable but require a careful revision.","major_comments":[{"comment":"There is a sign inconsistency in the final reconstruction. Equation (10) defines the estimated residual as r_hat_t = z_s - z_hat_t^0. Substituting this into Eq. (11) at the final step with beta_bar_0 = 0 gives z'_0 = z_s + (beta_bar_0 - 1) r_hat_1 = z_hat_1^0, which is the predicted shadow-free latent. However, the final paragraph of Section III-A states that the estimated shadow residual r_hat_1 is directly added to z_s, which would yield z_s + r_hat_1 = 2 z_s - z_hat_1^0. This is not a shadow-free latent unless z_s equals z_0. Since no code is provided, a reader implementing the equations literally and a reader implementing the prose description will obtain different outputs, and only one of them could have produced the reported PSNR/LPIPS/FID numbers. Please state the exact final reconstruction unambiguously and release code or pseudocode for the sampling loop.","section":"Section III-A, Eq. (10)-(11) and final paragraph"},{"comment":"The noise schedule notation is internally inconsistent. Equation (6) writes z_t = sqrt(alpha_t) z'_t + sqrt(1 - alpha_t) epsilon, while Eq. (8) divides by sqrt(alpha_bar_t) and Eq. (9) rearranges the terms as if z_t were sqrt(alpha_bar_t) z'_t + sqrt(1 - alpha_bar_t) epsilon. In standard DDPM notation, alpha_bar_t is the cumulative product of the alpha coefficients, not the sum described in the text. As written, Eq. (9) does not follow from Eq. (6). Please define alpha_t and alpha_bar_t consistently and correct Eq. (6) and Eq. (13) so that the derivation is valid.","section":"Section III-A, Eqs. (6), (8), (9)"},{"comment":"The paper reports only single-run numbers with no error bars or statistical significance tests, and it omits key sampling details such as the number of inference steps, any classifier-free guidance setting, and random seeds. Since the central claim is that the method improves over state-of-the-art methods, please provide uncertainty estimates for at least the main comparisons and state all inference hyperparameters. The absence of released code makes these details particularly important for reproducibility.","section":"Section IV, Tables I-III"},{"comment":"The derivation of Eq. (9) assumes that the pretrained noise estimator remains accurate when its input latent is a mixture of Gaussian noise and a shadow residual. This transferability assumption is plausible but is not directly validated anywhere in the paper. The authors should provide a per-timestep analysis, for example by measuring the error between the predicted noise/residual components and their ground-truth values during the backward process, or by comparing results when the noise estimator is fine-tuned versus kept frozen. Such an experiment would directly test the load-bearing assumption that the residual can be separated from the noise using the pretrained backbone.","section":"Section III-A, Eq. (9)"}],"minor_comments":[{"comment":"The sentence preceding Eq. (8) says that z_hat_t^0 represents the estimated noise at step t, but z_hat_t^0 is the estimated clean latent, not the estimated noise. This typo should be corrected.","section":"Section III-A, Eq. (8)"},{"comment":"The description of the self-enhancement strategy says 'When the probability threshold p is less than P' without defining p. Presumably p is a random variable sampled uniformly in [0,1]; please state this explicitly.","section":"Section III-B"},{"comment":"The overall SSIM for the full pipeline is reported as 0.965 in Table I and 0.964 in Table IV. Please reconcile these numbers.","section":"Table IV vs Table I"},{"comment":"The paper states that some methods were rerun with author-provided code and others were taken from benchmark [51], but it does not say which methods fall into which category. Please specify this so that the comparison is transparent.","section":"Section IV-D"},{"comment":"For the SRD dataset, the paper uses publicly available shadow masks from [30] for evaluation. It should state explicitly whether the same masks were used for all compared methods and for the reported PSNR-S/SSIM-S values.","section":"Section IV-C"},{"comment":"The label 'Ours-w/o EMA' appears in Table IV while the text uses 'Ours w/o EMA', and the header row contains 'Ours full pipline'. These typos should be fixed.","section":"Section IV-F"}],"recommendation":"major_revision","confidential_remarks":"The central contribution is potentially useful, but the internal sign inconsistency in the sampling algorithm is a serious reproducibility problem, especially with no code release. I would ask the editor to require a corrected and complete sampling description and ideally the code before considering acceptance. The paper's comparison is otherwise reasonable, and the ablation study covers the main design choices, but the lack of error bars weakens the SOTA claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper proposes a clever but not radical adaptation of latent diffusion to shadow removal: instead of generating the shadow-free image from scratch, it generates and refines the residual between shadowed and shadow-free latents, keeping the pretrained noise schedule intact and adding a ControlNet branch plus a fine-tuned decoder. The reported results on ISTD+ and SRD are strong and consistent across metrics, and the ablations back up the main design choices. That part is genuinely useful.\n\nThe soft spots are in the presentation, and one of them is serious. The notation is sloppy throughout: α_t and ᾱ_t are mixed, and ᾱ_t is called a 'sum' when it is a cumulative product. More importantly, the inference section is internally inconsistent. Eq. 10 defines r̂_t = z_s − ẑ_t^0, and Eq. 11 correctly uses (β̄_{t−1} − 1) r̂_t, which at t=1 reduces to z'_0 = ẑ_1^0. But the text then says to 'directly add' r̂_1 to z_s, which gives 2z_s − ẑ_1^0. Those are different. A reader who follows the text literally will not reproduce the numbers. Since no code is released, this is more than a cosmetic typo: it blocks independent verification. The equations alone are consistent, so I think it is a typo rather than a fundamental flaw, but the authors need to fix it and ideally release code.\n\nThe reader's worry about the pretrained noise estimator failing on residual-mixed latents is not the main issue. The empirical results suggest it transfers fine, and the paper's design keeps the backbone weights frozen, which is a reasonable safety measure. A theoretical guarantee would be nice but is not standard for this kind of paper. The absence of error bars is a minor concern given the margins, though it would be better to see them.\n\nOverall, this deserves a serious referee. The contribution is real, the experiments are careful, and the flaws are fixable. I'd send it to review, but I'd ask the authors to clarify the final step, clean up the notation, and release code. If they do, this could be a solid addition to the shadow-removal literature. If not, the numbers stay unverifiable.","headline":"A solid residual-generation adaptation of latent diffusion for shadow removal with strong reported numbers, but the inference text has a sign error and no code is released, so the numbers aren't yet reproducible.","tokens_in":16429,"tokens_out":4185,"would_cite":false,"duration_ms":41751,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A diffusion model that removes shadows by generating only the shadow residual.","keywords":["shadow removal","latent diffusion model","residual generation","image restoration","diffusion sampling","detail preservation","generative prior","ControlNet"],"falsifier":"Run the trained model on shadow-free test images: if the residual schedule and decomposition are correct, the output should reconstruct the input almost exactly (a near-zero residual). Any invented texture or brightness change in those inputs would show that the model is generating content rather than only removing shadows.","tokens_in":15416,"feed_emoji":"🌓","tokens_out":4660,"duration_ms":45387,"temperature":0.7,"pith_summary":"This paper argues that the best way to apply a large pre-trained diffusion model to shadow removal is to have it generate and refine the shadow residual—the difference between the shadowed and shadow-free image—rather than regenerate the shadow-free image from scratch. That shift lets the model lean on the existing image's details, so the content outside and inside the shadow is preserved instead of being reinvented. To make this work, the paper adds a residual schedule to the diffusion forward process, a noise–residual decomposition to the sampling step, a cross-timestep training strategy that uses the model itself to correct its own trajectory, and a detail-preserving decoder that keeps high-frequency information intact. On ISTD+ and SRD, the method reports the best LPIPS and FID scores among the compared mask-free methods, and the highest PSNR (33.38 dB on ISTD+). The core claim is that a frozen generative backbone can be turned into a high-fidelity deterministic restorer if the diffusion process is steered to edit only what needs changing.","feed_headline":"Diffusion model removes shadows by generating only the residual","feed_subtitle":"A pre-trained latent diffusion model is repurposed to generate and refine shadow residuals, beating prior methods on fidelity metrics.","key_machinery":"The key machinery is the residual generation diffusion process: a shadow residual schedule $\\bar{\\beta}_t$ is layered on top of the unchanged DDIM noise schedule, so the forward latent becomes $z_t = \\sqrt{\\alpha_t}(z_0 + \\bar{\\beta}_t r) + \\sqrt{1-\\alpha_t}\\epsilon$ with $r = z_s - z_0$. During sampling, a Noise-Residual Decomposition (NRD) step extracts the estimated residual $\\hat{r}_t = z_s - \\hat{z}_0^t$ and subtracts the residual's contribution from the network output to recover the pure noise estimate (Eq. 12), then recombines them with the shadow latent to produce the next latent. The residual term is what lets the frozen pre-trained diffusion backbone estimate both noise and shadow correction from one output, while a ControlNet branch (initialized from the pre-trained model) learns the residual. Around this, the cross-timestep self-enhancement training uses an EMA weight copy to produce pseudo-inputs for the main network, and the detail-preserving decoder adds a controller with zero-initialized deformable convolutions that inject multi-scale shadow-image features into the decoding process and adds an image-domain residual.","core_discovery":"The central claim is that shadow removal can be reformulated as residual generation within a pre-trained latent diffusion model, yielding higher fidelity than both full regeneration and prior residual-based diffusion methods. The paper introduces a forward process in which the noisy latent interpolates between the shadowed and shadow-free latents via a residual schedule, and a sampling rule that decomposes the network output into a noise component and a residual component at every step. Because the backbone network is frozen and only a ControlNet-style branch is trained, the generative prior from the large model is retained while the new branch learns the shadow-specific correction. The authors further claim that a cross-timestep self-enhancement strategy—where an EMA copy of the network generates pseudo-inputs at earlier timesteps—prevents error accumulation, and that a decoder equipped with a controller, multi-scale skip connections, and zero-initialized deformable convolutions preserves details the original VQ-GAN decoder would distort. The evidence is the reported state-of-the-art perceptual metrics and the ablation results.","pith_inferences":["The residual-generation trick likely generalizes to other deterministic image-to-image tasks, such as dehazing, deraining, or low-light enhancement, wherever the input already contains most of the output content.","The noise-residual decomposition might be a general recipe for controlled editing with diffusion models: any editable attribute could be encoded as a residual schedule on top of a frozen backbone.","Since the backbone is frozen, swapping in different control branches could turn a single large model into a multi-task restorer, with only the small branch retrained per task.","A testable extension is to measure whether non-shadow regions pass through the pipeline unchanged; if the skip connections are the main fidelity driver, then removing the residual schedule should only slightly hurt non-shadow PSNR."],"forward_implications":["Mask-free shadow removal can reach higher perceptual fidelity than previous specialized methods, as measured by LPIPS and FID on ISTD+ and SRD.","A frozen pre-trained diffusion backbone can be reused for a deterministic restoration task by only training a control branch, avoiding full retraining.","The cross-timestep self-enhancement strategy reduces error accumulation, which should allow the model to use fewer sampling steps without degradation.","The detail-preserving decoder can be applied to any latent diffusion pipeline where the VQ-GAN codec destroys high-frequency content, not just shadow removal.","With a mask provided, the same framework improves further (PSNR 34.73, LPIPS 0.0228 on ISTD+), suggesting the architecture scales with task information."],"supporting_citations":[{"why":"Provides the pre-trained Paint-By-Example inpainting model used as the frozen backbone whose generative prior is harnessed.","marker":"[39]"},{"why":"Supplies the residual denoising diffusion concept and the linear residual schedule that the method adapts and extends.","marker":"[16]"},{"why":"Defines the ControlNet architecture that lets a small trained branch condition a frozen diffusion model.","marker":"[9]"},{"why":"Provides the deterministic DDIM sampling rule that the residual sampling process builds upon.","marker":"[7]"},{"why":"Motivates fine-tuning large diffusion models for restoration and represents the generative-restoration baseline the method improves on.","marker":"[12]"},{"why":"Introduces the VQ-GAN encoder-decoder whose information loss motivates the detail-preserving decoder design.","marker":"[41]"},{"why":"Shows that fine-tuning a decoder with a conditional encoder can restore missing details, an idea the proposed decoder extends.","marker":"[42]"},{"why":"Provides the deformable convolution with learnable offsets and modulation scalars used in the zero-initialized skip connections.","marker":"[43]"},{"why":"Supplies the ISTD+ benchmark dataset, including shadow masks and ground truth, used for training and evaluation.","marker":"[45]"},{"why":"Supplies the SRD benchmark dataset used to evaluate the method's generalization on another large shadow-removal corpus.","marker":"[46]"}],"fun_headline_variants":["Diffusion model erases shadows via residual synthesis alone","Residual-only generation improves shadow removal fidelity","Frozen diffusion prior plus new branch removes shadows precisely","Shadow-free images from residual diffusion, not full redo"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pre-trained diffusion network's learned denoising behavior remains valid when its input latent is a mix of Gaussian noise and a shadow residual, rather than pure noised images.","fun_headline_variants_meta":{"raw":{"variants":["Diffusion model erases shadows via residual synthesis alone","Residual-only generation improves shadow removal fidelity","Frozen diffusion prior plus new branch removes shadows precisely","Shadow-free images from residual diffusion, not full redo"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1379,"prompt_tokens":983,"completion_tokens":396,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":334}},"tokens_in":599,"tokens_out":396,"duration_ms":5244,"temperature":1.0,"reasoning_tokens":334,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:35:13.050843+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained model on shadow-free test images: if the residual schedule and decomposition are correct, the output should reconstruct the input almost exactly (a near-zero residual). Any invented texture or brightness change in those inputs would show that the model is generating content rather than only removing shadows.","supporting_citations":[{"cited_title":"Physics-based shadow image decomposition for shadow removal,","cited_arxiv_id":null,"evidence_quote":"Supplies the ISTD+ benchmark dataset, including shadow masks and ground truth, used for training and evaluation."},{"cited_title":"Deshadownet: A multi- context embedding deep network for shadow removal,","cited_arxiv_id":null,"evidence_quote":"Supplies the SRD benchmark dataset used to evaluate the method's generalization on another large shadow-removal corpus."}],"review_version":1}