{"id":"8712c053-aee6-49e6-87b9-cc4c4a2f7856","arxiv_id":"2412.02635","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A joint detection-removal-synthesis model, MetaShadow, reports state-of-the-art results for object-centered shadow editing on several benchmarks.","lead":"MetaShadow is a single AI system that finds, erases, and redraws shadows of objects in photos so edited images look realistic. It pairs an analyzer that understands an object's existing shadow with a generator that paints a matching shadow after the object is moved or inserted.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The shadow-knowledge transfer is the central claim's linchpin, but the Sec. 5.2 ablation does not rule out appearance leakage into Fms; the paper itself notes texture contamination in high-resolution features, so synthesis gains may not transfer to unseen objects.","rationale":"The strongest claim is broad, but its linchpin is the transfer mechanism: the paper needs Fms to carry shadow knowledge that generalizes beyond the training objects. I checked whether Sec. 5.2 actually isolates that mechanism. It does not: SSDM-Text and SSDM-CLIP differ from Ours in the conditioning source, the adaptor, and the amount of pixel-level information, and no ablation removes or corrupts the shadow-specific part of Fms. The paper's own observation that high-resolution features include texture within the shadow region reinforces this concern, because resizing those features to 32x32 aliases rather than removes the texture. Since the Synthesizer also receives the input image and object mask, it could in principle copy the visible reference shadow, making the cross-attention condition redundant. This is an empirical question, not a fatal flaw; the proposed check using Fms from the shadow-removed image would settle it. The paper otherwise presents a plausible architectural integration and strong numbers, so the verdict should remain CONDITIONAL pending that test rather than being rejected. Secondary issues—unfair detection protocol, unreleased datasets, and metric-dependent synthesis claims—are real but do not displace the transfer-mechanism concern as the most load-bearing point.","tokens_in":14074,"tokens_out":5650,"duration_ms":61151,"concrete_test":"Run the Sec. 5.2 ablation one step further: keep the Shadow Synthesizer frozen and feed it Fms computed by the Shadow Analyzer on (a) the original shadowed input and (b) the same input after the Analyzer's own shadow removal, with Io and Meo unchanged. Compare DESOBA Local RMSE/Bbox PSNR and Video DESOBA local RMSE over the full test sets with error bars. If (a) and (b) are statistically indistinguishable, Fms is carrying scene/object appearance rather than transferable shadow properties; if (a) is significantly better on shadow-sensitive metrics, the knowledge-transfer claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that shared shadow knowledge, carried by Shadow Analyzer multi-scale features Fms, is what makes the Synthesizer SOTA. The load-bearing premise is that Fms, after resizing features from sizes 16 to 128 into a uniform 32x32 conditioning (Sec. 4.2), encodes transferable shadow properties—position, softness, intensity, color, direction—rather than the texture or color of the original object or shadowed background. The paper itself flags this risk: Sec. 5.2 admits that larger-resolution features 'gradually include texture information within the shadow region, which is not desired,' yet the design still feeds features up to 128 resolution into the 32x32 conditioning. Downsizing does not remove texture; it aliases it into the conditioning. The only ablation (Table 5) compares Fms against T5 text and finetuned CLIP embeddings; it does not control for the amount of pixel-level image content injected, and it does not test whether the benefit survives when object appearance is changed while shadow properties are held fixed. If Fms leaks location-specific appearance, the reported DESOBA synthesis gains (Local RMSE 36.84 vs 64.90) and the Video DESOBA local RMSE gain would not generalize to unseen objects and scenes, and the 'shadow knowledge transfer' mechanism is not established. Table 4 independently shows the SOTA claim is metric-dependent: on Video DESOBA bbox PSNR, Ours scores 23.14 versus SGDiffusion's 36.54.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces MetaShadow, a two-stage framework that combines an object-centered GAN (Shadow Analyzer) for shadow detection and removal with a diffusion-based Shadow Synthesizer for reference-based shadow synthesis. The central idea is to transfer multi-scale intermediate features of the Shadow Analyzer into the diffusion model's cross-attention layers, providing shadow knowledge (position, softness, intensity, color, direction) to guide synthesis. The authors also introduce a synthetic training set (MOS), two real-world test sets (Moving DESOBA, Video DESOBA), and report large improvements over existing baselines on detection (SOBA mIoU 71.0 vs 55.8), removal (Bbox PSNR 32.97 vs 24.28), and synthesis (DESOBA Local RMSE 36.84 vs 56.44 for SGRNet). The paper claims to be the first framework to jointly handle all three object-centered shadow tasks with a shared-knowledge transfer mechanism.","tokens_in":14348,"tokens_out":6687,"duration_ms":66944,"significance":"If the claims are supported, this is a meaningful advance for object-centered image editing: a single framework that detects, removes, and synthesizes object shadows could simplify editing workflows and improve compositing realism. The proposed mechanism of injecting GAN-derived features into a diffusion model is a plausible and interesting approach to controllable shadow synthesis, and the new datasets are likely to be useful to the community. The paper is clearly written, includes visual comparisons and an ablation study, and the authors state that they will release the new test sets. However, several load-bearing issues in the evaluation protocol and in the evidence for the knowledge-transfer mechanism currently prevent full confidence in the SOTA claims.","major_comments":[{"comment":"The shadow detection comparison is not apples-to-apples. The table caption notes that SSISv2 must automatically detect shadow-object instance pairs, whereas MetaShadow receives the object mask as input. The mIoU gain from 55.8 to 71.0 can therefore be substantially attributed to the oracle object mask rather than to superior shadow detection. To support the claimed SOTA detection performance, the authors should either evaluate MetaShadow with predicted object masks from an off-the-shelf object detector, or also report object detection accuracy so that both methods solve the same task under the same input protocol.","section":"Sec. 5.1, Table 2"},{"comment":"The reference-based synthesis setup is not precisely specified. The text describes the Synthesizer as taking an object mask and an image containing the moved object, while the reference shadow appears to enter only through the Shadow Analyzer's features Fms computed from the original input image. For DESOBA the reference is described as another object in the image, and for Video DESOBA the first frame is used as the reference, but the paper does not state which image is fed to Stage I in these cases or how the network distinguishes the reference shadow from the target object's original shadow. Please provide a precise, implementable description of the reference input and the information flow, and clarify how a separate reference object or shadow is represented.","section":"Sec. 4.2 and Fig. 3"},{"comment":"The ablation for the shadow knowledge transfer does not rule out appearance leakage. Fms includes features from resolutions 16 to 128, resized to 32x32, and the paper itself notes in Sec. 5.2 that larger-resolution features contain texture information within the shadow region. The ablation replaces Fms with T5 text or CLIP image embeddings, but it does not control for the amount of pixel-level image content injected or test whether the benefit survives when object appearance is changed while shadow properties are held fixed. Without such a control experiment, the claim that Fms encodes transferable shadow properties rather than object texture is not established, and the synthesis gains may not generalize to unseen objects.","section":"Sec. 5.2, Table 5"},{"comment":"The SOTA synthesis claim is metric-dependent. On Video DESOBA, MetaShadow's Bbox PSNR (23.14) is far below SGDiffusion (36.54), while on DESOBA the Bbox SSIM gain over SGRNet is small (93.56 vs 91.08). The abstract highlights the reduction in local RMSE from 51.73 to 36.54, but this selects the most favorable dataset-level metric. The paper should report all metrics with error bars or significance tests, and discuss explicitly why local RMSE is the appropriate primary metric, rather than presenting the result as uniformly superior.","section":"Sec. 5.1, Table 4"},{"comment":"ObjectDrop [51] is listed in Table 1 as supporting detection, removal, and synthesis in an object-centered, reference-based manner, yet it is not included in any quantitative comparison. Since ObjectDrop is a very recent joint framework that appears to be highly relevant, omitting it weakens the claim that no existing method handles the same combination of tasks and that MetaShadow achieves SOTA. The authors should compare against ObjectDrop on the available benchmarks or justify its exclusion.","section":"Sec. 5.1 and Table 1"}],"minor_comments":[{"comment":"The Moving DESOBA and Video DESOBA test sets are introduced without basic statistics (number of images, number of objects, resolution, annotation type). Please add a summary table in the main text or supplementary material.","section":"Sec. 3"},{"comment":"The adaptor T(·) is described as containing 'a 2D convolution layer followed by a 1D convolution layer'; this is likely intended as a 1x1 convolution, but the text should be clarified or corrected.","section":"Sec. 4.2"},{"comment":"The notation Mes is used in the loss but not defined in the main text. Please state what Mes represents, when it is provided as input, and how it relates to the optional shadow mask in the Synthesizer.","section":"Eq. (1)"},{"comment":"The caption should explicitly state that MetaShadow takes an object mask as input while ShadowDiffusion takes a shadow mask, since the protocol difference is essential for interpreting the comparison.","section":"Sec. 5.1, Table 3 caption"},{"comment":"The four-step diffusion inference setting is mentioned only in the final paragraph of Sec. 5.2. Please state the inference step count in Sec. 4.2 or the implementation details, and report the step counts used for all baselines for transparency.","section":"Sec. 5.2"},{"comment":"References [47] and [48] appear to describe the same paper (Instance Shadow Detection, CVPR 2020) with different metadata; please consolidate to avoid a duplicate citation.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The central idea is promising and the experimental scope is broad, but the evaluation protocol for detection is unequal, the reference input to the synthesis network is underspecified, and the key ablation does not isolate shadow knowledge from appearance leakage. The absence of a quantitative comparison against ObjectDrop is a notable gap given the paper's SOTA claim. If the authors can address these points with additional experiments or by substantially reframing the claims, the paper could become a strong contribution. In the meantime, the manuscript is not ready for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MetaShadow is the first system I know of that does object-centered shadow detection, removal, and synthesis in one framework, and it backs the claim with a broad evaluation across four benchmarks. The detection and removal numbers are genuinely strong: SOBA mIoU jumps from 55.8 to 71.0, and on DESOBA removal they beat ShadowDiffusion even when the baseline gets ground-truth masks. The synthesis gains on DESOBA local RMSE (from ~64.9 to 36.8) are large, though not uniform across metrics. The GAN-feature-to-diffusion conditioning is the real novelty, and the ablation showing it beats T5 text and finetuned CLIP embeddings gives it at least preliminary support.\n\nThe soft spots are addressable but real. First, the detection comparison is not apples-to-apples: MetaShadow receives ground-truth object masks while SSISv2 has to detect object-shadow pairs automatically. That explains some of the gap, and the paper should say so more prominently or evaluate under the same input protocol. Second, there are no error bars or significance tests anywhere, and on Video DESOBA bbox PSNR they trail SGDiffusion (23.14 vs 36.54) despite winning on local RMSE. The \"significant improvements\" claim is oversold. Third, code, MOS data, and the two new test sets are not released, which makes the numbers hard to check.\n\nThe stress-test concern about appearance leakage is worth taking seriously. The paper itself admits that high-resolution analyzer features carry texture information, yet still resizes features up to 128 resolution into the 32x32 conditioning. The ablation doesn't control for the amount of pixel content injected, so the \"shadow knowledge transfer\" story is not fully nailed down. That said, this is a weakness in the analysis, not a fatal flaw: the joint training could still be learning useful shadow properties, and the improvement over CLIP suggests the features carry something more than generic appearance. A controlled ablation that varies object appearance while holding shadow properties fixed would settle it.\n\nOverall, this is a competent, useful piece of work that combines existing components in a new way and evaluates honestly, modulo protocol and release gaps. It deserves a serious referee. I'd recommend conditional acceptance, with requests for like-for-like detection evaluation, error bars, artifact release, and a cleaner knowledge-transfer ablation.","headline":"A solid three-in-one shadow editing system with real gains on detection and removal, but the synthesis mechanism needs a cleaner ablation and the detection comparison isn't like-for-like.","tokens_in":14933,"tokens_out":1800,"would_cite":true,"duration_ms":18433,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MetaShadow claims that one object-mask-conditioned model can detect, remove, and re-synthesize shadows, and that sharing shadow-rich features between a GAN-based analyzer and a diffusion-based synthesizer is what makes all three tasks…","keywords":["object-centered shadow detection","shadow removal","shadow synthesis","shadow knowledge transfer","GAN and diffusion models","image editing","reference-based synthesis","MOS dataset"],"falsifier":"Replace the Shadow Analyzer's transferred features with features taken from the object's shadow-free region or from a different object while keeping everything else fixed; if synthesis Local RMSE does not degrade sharply, the paper's claim that these features carry transferable shadow knowledge is wrong.","tokens_in":13802,"feed_emoji":"🌓","tokens_out":8586,"duration_ms":74747,"temperature":0.7,"pith_summary":"The paper sets out to show that shadow detection, removal, and synthesis do not have to be three separate models: one object-centered framework, MetaShadow, can handle all three, and the three tasks get better because they share knowledge. The framework pairs a GAN-based Shadow Analyzer, which takes an image and an object mask and produces both the object's shadow mask and a shadow-free image, with a reference-based diffusion Shadow Synthesizer that draws a new shadow. The shared knowledge is carried by multi-scale intermediate features from the Analyzer, injected into the Synthesizer through cross-attention after resizing to $32\\times32$. The reported results are concrete: detection mIoU on SOBA rises from 55.8 to 71.0, removal bbox PSNR on DESOBA beats ShadowDiffusion by 8.7 dB, and synthesis Local RMSE on Video DESOBA falls from 51.73 to 36.54. A sympathetic reader would take the paper's central assertion to be that object-centered editing becomes far more realistic when shadows are treated as an editable property of the object, not as an afterthought.","feed_headline":"One model detects, removes, and redraws shadows","feed_subtitle":"Object masks let the same model erase an object's shadow, then re-cast it with matching softness and direction.","key_machinery":"The load-bearing object is the shadow-knowledge-transfer channel between the two stages. From the Shadow Analyzer's spatial decoder, multi-scale features $F^i_{ms}$ at resolutions 16 through 128 are resized to a common $32\\times32$ grid, concatenated into a tensor of shape $[N,1348,32,32]$, and passed through an adaptor $T(\\cdot)$ made of a 2D convolution followed by a 1D convolution; an MLP then lifts the embedding dimension from 1344 to 2048, yielding the shadow embedding $E_s$ of shape $[N,1024,2048]$ that is fed into the diffusion U-Net's cross-attention layers as key and value. The paper's argument is that these features carry exactly the shadow properties the synthesizer needs—position, softness, intensity, color, direction—so the U-Net learns to paint a shadow that matches the reference instead of inventing one from text or generic image embeddings.","core_discovery":"The core discovery, on the paper's own terms, is that a single framework can jointly perform object-centered shadow detection, removal, and synthesis, and that the joint training actually improves each task rather than compromising it. The Shadow Analyzer, built on an inpainting GAN, detects the shadow cast by a masked object and removes it; the Shadow Synthesizer, a reference-based diffusion model, then generates a new shadow for the object at a target location. The mechanism that makes the combination work is shadow knowledge transfer: the Analyzer's shadow-rich multi-scale features, after being resized to a uniform $32\\times32$ resolution, are injected into the Synthesizer's cross-attention layers as a shadow embedding. The paper argues that this beats both pure GAN synthesis, which struggles with plausible shadow shapes, and text- or CLIP-conditioned diffusion synthesis, which lacks fine-grained control over shadow intensity, softness, color, and direction. It also claims a practical bonus: the conditioned synthesizer needs only four denoising steps, against 30 for the DDPM-based baseline and 50 for SGDiffusion.","pith_inferences":["If the transfer mechanism generalizes, then any object-conditional GAN that produces shadow-aware features could similarly condition a diffusion model, turning shadow knowledge transfer into a general recipe rather than a task-specific trick.","The four-step inference advantage suggests the feature conditioning may sharply reduce the diffusion sampling budget, which could matter for interactive editing tools even though the paper does not systematically study sampling efficiency.","A testable extension is to probe the transferred features under out-of-distribution lighting, such as indoor versus outdoor scenes or colored lights, to see whether softness and direction generalize beyond the static-background tripod videos in Video DESOBA.","The construction of a synthetic MOS dataset plus two real evaluation sets implies a reusable data recipe: render moving objects with shadows in Blender, then evaluate on real videos; the same recipe could apply to other intrinsic image decomposition tasks such as reflection or illumination editing."],"forward_implications":["A single object-mask-conditioned model can produce the object's shadow mask, a shadow-free version of the image, and a re-synthesized shadow, so editing pipelines no longer need to stitch together separate detectors and removers.","Object-centered shadow detection on SOBA improves from 55.8 to 71.0 mIoU, meaning per-object shadow masks become reliable enough for downstream editing.","Shadow removal on DESOBA reaches 32.97 dB bbox PSNR, an 8.7 dB gain over ShadowDiffusion even when that baseline is given ground-truth masks, implying cleaner shadow-free regions.","Shadow synthesis reduces Local RMSE to 36.84 on DESOBA and 36.54 on Video DESOBA, and does so in four diffusion steps, so reference-based shadow synthesis becomes both more accurate and much faster.","Because the synthesizer is conditioned on a reference shadow, users can control shadow color, softness, and direction by choosing the reference, which supports creative edits like relocating objects with matched lighting."],"supporting_citations":[{"why":"Supplies the DESOBA benchmark and the SGRNet baseline for object-centered shadow synthesis, and is one of the datasets used to train the Shadow Analyzer.","marker":"[13]"},{"why":"Provides SGDiffusion, the diffusion-based synthesis baseline that MetaShadow compares against on DESOBA and Video DESOBA.","marker":"[28]"},{"why":"Provides ShadowDiffusion, the diffusion-based removal baseline that MetaShadow outperforms on DESOBA.","marker":"[11]"},{"why":"Provides SSISv2, the instance shadow detector used as the main detection baseline and as the shadow-mask provider for the removal comparison.","marker":"[50]"},{"why":"Provides the SOBA dataset of object-shadow pairs used to evaluate object-centered shadow detection with mIoU metrics.","marker":"[47]"},{"why":"Provides the CM-GAN inpainting architecture that the Shadow Analyzer is built on and initialized from.","marker":"[59]"},{"why":"Provides the latent diffusion inpainting architecture that the Shadow Synthesizer adapts for shadow synthesis.","marker":"[37]"},{"why":"Provides the DDPM denoising backbone used by the Shadow Synthesizer and by the SSDM-Text baseline.","marker":"[12]"},{"why":"Supplies ISTD+, one of the partially annotated datasets used to train the Shadow Analyzer on general shadows.","marker":"[22]"},{"why":"Supplies SRD, a partially annotated shadow-removal dataset used in multi-source training and in the generalization comparison.","marker":"[34]"}],"fun_headline_variants":["One model detects, removes, and synthesizes shadows","Joint training boosts detection, removal, and synthesis of shadows","MetaShadow unifies object-centered shadow detection, removal, and synthesis","From erasing to re-casting shadows with a single framework"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework depends on the assumption that the $32\\times32$ multi-scale features injected into the synthesizer encode the shadow's transferable properties—position, softness, intensity, color, and direction—rather than the texture or color of the specific object or background; if that assumption fails, the claimed synthesis gains would not carry over to unseen objects and scenes.","fun_headline_variants_meta":{"raw":{"variants":["One model detects, removes, and synthesizes shadows","Joint training boosts detection, removal, and synthesis of shadows","MetaShadow unifies object-centered shadow detection, removal, and synthesis","From erasing to re-casting shadows with a single framework"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000706,"raw_usage":{"total_tokens":3171,"prompt_tokens":926,"completion_tokens":2245,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":2188}},"tokens_in":542,"tokens_out":2245,"duration_ms":16615,"temperature":1.0,"reasoning_tokens":2188,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:14:05.458137+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the Shadow Analyzer's transferred features with features taken from the object's shadow-free region or from a different object while keeping everything else fixed; if synthesis Local RMSE does not degrade sharply, the paper's claim that these features carry transferable shadow knowledge is wrong.","supporting_citations":[{"cited_title":"Shadow generation for composite image in real-world scenes","cited_arxiv_id":null,"evidence_quote":"Supplies the DESOBA benchmark and the SGRNet baseline for object-centered shadow synthesis, and is one of the datasets used to train the Shadow Analyzer."},{"cited_title":"Shadow generation for composite image 9 using diffusion model","cited_arxiv_id":null,"evidence_quote":"Provides SGDiffusion, the diffusion-based synthesis baseline that MetaShadow compares against on DESOBA and Video DESOBA."},{"cited_title":"ShadowDiffusion: When degradation prior meets diffusion model for shadow removal","cited_arxiv_id":null,"evidence_quote":"Provides ShadowDiffusion, the diffusion-based removal baseline that MetaShadow outperforms on DESOBA."},{"cited_title":"Instance shadow detection with a single-stage detector","cited_arxiv_id":null,"evidence_quote":"Provides SSISv2, the instance shadow detector used as the main detection baseline and as the shadow-mask provider for the removal comparison."},{"cited_title":"Instance shadow detection","cited_arxiv_id":null,"evidence_quote":"Provides the SOBA dataset of object-shadow pairs used to evaluate object-centered shadow detection with mIoU metrics."},{"cited_title":"CM-GAN: Image inpainting with cascaded modulation GAN and object-aware training","cited_arxiv_id":null,"evidence_quote":"Provides the CM-GAN inpainting architecture that the Shadow Analyzer is built on and initialized from."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Provides the latent diffusion inpainting architecture that the Shadow Synthesizer adapts for shadow synthesis."},{"cited_title":"Denoising diffu- sion probabilistic models","cited_arxiv_id":null,"evidence_quote":"Provides the DDPM denoising backbone used by the Shadow Synthesizer and by the SSDM-Text baseline."},{"cited_title":"Shadow removal via shadow image decomposition","cited_arxiv_id":null,"evidence_quote":"Supplies ISTD+, one of the partially annotated datasets used to train the Shadow Analyzer on general shadows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies SRD, a partially annotated shadow-removal dataset used in multi-source training and in the generalization comparison."}],"review_version":1}