{"id":"20780edf-4e68-47a4-ba8f-1faaffc36560","arxiv_id":"2412.12912","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A masking and Jacobian projection technique discovers unsupervised semantic directions in diffusion model latent space, enabling region-local editing without fine-tuning.","lead":"This paper proposes a training-free way to edit local image regions in pretrained diffusion models by finding semantic directions in the model's latent space with a masked Jacobian projection. It reports that region-specific edits such as smiles or closed eyes can be made while leaving surrounding pixels mostly unchanged, and that it outperforms several supervised baselines on face attributes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 14 is only a rank-one projection: if the unmasked Jacobian has rank > 1, it does not minimize unmasked changes, so the central claim is unsupported.","rationale":"The reader correctly identifies that Eq. 13 is not formally connected to Eq. 14. My stress-test sharpens this into a specific, checkable mathematical flaw: Eq. 14 removes only a single direction from J_m, so unless J_u is rank-one or the authors explicitly define J_m and J_u as dominant singular vectors, the projection does not minimize the full unmasked Jacobian response. This is more concrete than the reader's general 'no proof' concern and directly targets the mechanism behind the central claim. The paper's empirical evaluation (one attribute vs. supervised baselines, no error bars, missing appendix) supports a cautious verdict, but the mathematical gap is the load-bearing issue. If the proposed synthetic or real-model test shows that Eq. 14 leaves substantial unmasked sensitivity, the method's stated mechanism fails and the claim of superior outside-region preservation would need to be re-supported empirically. If the test shows near-orthogonality, the concern is resolved and the conditional can be lifted. Thus the appropriate verdict is CONDITIONAL: accept only with a corrected projection, an explicit rank-one justification, or a demonstration that the residual unmasked sensitivity is negligible. This is not an ad hominem or a consensus-based objection; it is an internal inconsistency between Eq. 13 and Eq. 14 as written.","tokens_in":11297,"tokens_out":5856,"duration_ms":59295,"concrete_test":"Build a synthetic linear check of Eq. 14: take h ∈ R^2, masked output m(h) = h_1, unmasked output u(h) = [h_1, h_2]. Then J_m = [1,0] and J_u = I_2. Applying Eq. 14 with either Frobenius inner product or vectorized J_u leaves a direction v whose unmasked response J_u v is nonzero, directly falsifying the claim that the projection zeroes out unmasked changes. Then, on a real DDPM at a fixed timestep, compute the top singular vector v_m of the masked Jacobian and the top r singular vectors v_u^k of the unmasked Jacobian; verify whether the direction from Eq. 14 has near-zero dot product with every v_u^k. If it does not, the projection is incomplete and the claim that non-masked regions are preserved is unsupported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central step is the claimed equivalence between the optimization objective (Eq. 13) and the orthogonal projection (Eq. 14). As written, Eq. 14 treats J_m and J_u as single vectors: J = J_m - (J_m·J_u)/(J_u·J_u) J_u. But in the Method section J_t is defined as a Jacobian matrix (Eq. 6, J_t = ∂f/∂h_t) with f vector-valued over pixels. If J_u is a matrix of rank r > 1, the subspace orthogonal to all unmasked sensitivities is not obtained by subtracting one rank-one term; the correct projection is J_m - J_u (J_u^T J_u)^+ J_u^T, or equivalently removing components along a full basis of singular vectors of J_u. The single-vector formula only cancels the component along one direction (e.g., the top singular vector if J_u is approximated by power iteration), leaving all other unmasked directions unconstrained. Therefore Eq. 13 is not in general solved by Eq. 14. This is load-bearing because the paper's headline contribution — precise local editing with non-masked regions preserved — rests entirely on this projection. The empirical results could still hold if J_u is well approximated by a single dominant direction, but the paper provides no evidence for that rank-one approximation, and it is inconsistent with the stated matrix formulation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Region-Based Editing (RBE), a training-free method for local semantic editing in the h-space of pretrained U-Net diffusion models. The user provides a mask, the method computes Jacobians of the masked and unmasked regions of the noise-prediction network, applies an orthogonal projection to suppress unmasked-region sensitivity, and injects the resulting semantic direction during DDIM reverse sampling. The authors report qualitative results on CelebA-HQ, LSUN-church, and LSUN-bedroom and compare quantitatively with Asyrp, Boundary Diffusion, and Semdiff on the smile attribute, claiming state-of-the-art performance and even superiority over supervised baselines.","tokens_in":11463,"tokens_out":6618,"duration_ms":66907,"significance":"The paper addresses a relevant and timely problem: training-free, mask-only semantic editing with preservation of the unmasked image content. If the projection step were rigorously justified, the method would be a useful and practical contribution, and the qualitative examples do suggest that outside-mask preservation is better than in the compared baselines. The paper also gives a clear statement of limitations in the Discussion. However, the central mathematical claim connecting the optimization objective to the projection is not established, and the quantitative evaluation is too narrow to support the headline claim of surpassing supervised methods. The potential is real, but the manuscript needs substantial revision before the contribution can be assessed reliably.","major_comments":[{"comment":"Equation (13) is not a well-defined optimization objective. It is written as 'arg max ... min ...' without specifying a norm, a scalarization, or the admissible set for h_t. Even if it is interpreted as 'maximize masked-region change while minimizing unmasked-region change,' the paper gives no derivation showing that the orthogonal projection in Eq. (14) optimizes this criterion. Because the entire method rests on this equivalence, the authors must either provide a rigorous proof under explicit assumptions or reformulate the objective and the projection as a heuristic and validate that heuristic experimentally.","section":"Methodology, Eq. (13)-(14)"},{"comment":"As written, Eq. (14) uses single-vector dot products, but J_t in Eq. (6) and J_masked in Eq. (12) are matrices, and J_m and J_u have different numbers of rows because they correspond to masked and unmasked pixel sets. If the intended operation is a Gram-Schmidt projection of one Jacobian vector against another, the formula cancels only one direction of unmasked sensitivity. For a Jacobian J_u of rank greater than one, the orthogonal complement of the unmasked subspace is not obtained by a single rank-one subtraction; the correct form is J_m - J_u (J_u^T J_u)^+ J_u^T J_m (or an equivalent basis projection). The authors should either use the full projection or explicitly state and empirically justify a rank-one approximation.","section":"Methodology, Eq. (14)"},{"comment":"The method can discover many right singular vectors for a masked region, but the paper never specifies how the vector corresponding to a desired attribute (e.g., 'smile' versus 'slanted mouth') is selected. Figure 3 shows that the same mouth mask yields different attributes, which suggests manual selection after the fact. Without a selection criterion, the method is not fully unsupervised, and the reported per-attribute results are not reproducible from the algorithmic description in the paper.","section":"Experiments, Figure 3 and 'Unsupervised Editing through Jacobian'"},{"comment":"The headline claim of surpassing supervised methods is based on a single attribute ('smile') and a single dataset, with no standard deviations, no number of runs, and no significance test. In addition, the MSE values in Table 2 are not decomposed into in-mask and out-of-mask components, even though outside-mask preservation is the central contribution of the paper; that decomposition is only shown in Figure 7 for one comparison against Semdiff. The comparison needs more attributes, more datasets, and statistical support to justify the strength of the claim.","section":"Experiments, Table 2"}],"minor_comments":[{"comment":"There is a typo in 'Diffusion autuencoders'; it should be 'Diffusion autoencoders'.","section":"Background"},{"comment":"The paragraph preceding Figure 7 says 'The qualitative results in Figure 7 also prove this point,' but Figure 7 is a quantitative bar-chart comparison of pixel changes; the wording should say 'quantitative results.'","section":"Experiments, Figure 7"},{"comment":"The caption states that the mask is used 'during training and do not need to add it during testing,' but the method is training-free; this should be rephrased as 'during editing' or the distinction should be clarified.","section":"Experiments, Figure 3 caption"},{"comment":"The acronym 'Asyrp' is used without being defined or cited; please provide the full method name and a reference.","section":"Experimental Setup"}],"recommendation":"major_revision","confidential_remarks":"The central derivation needs to be repaired before the paper can be accepted. The paper repeatedly refers to an appendix, but the appendix is not included in this version; please ensure the full version with implementation details and additional experiments is available for review. If the authors can either prove the projection claim or reframe the method as a clearly validated heuristic, the contribution could become publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe paper proposes RBE, a training-free method for local editing in diffusion models by using the Jacobian of the noise-prediction network with respect to h-space. The new ingredient is Eq. (14): an orthogonal projection of the masked Jacobian J_m against the unmasked Jacobian J_u, intended to suppress changes outside the edit region. That is a plausible idea, and the qualitative results indeed show better outside-region preservation than Semdiff. However, the central derivation is not there. In Eq. (14), J_m and J_u are treated as vectors, but they were defined as matrices (Eq. 6). The dot product is undefined as written. More importantly, the projection is a single rank-one subtraction, which only removes the component of J_m along one direction of J_u. If J_u has rank > 1—which it generally will, since it maps a multi-dimensional h-space to many pixels—this does not minimize unmasked changes. The correct projection would be J_m - J_u (J_u^T J_u)^+ J_u^T, i.e., removing the full row space. The authors give no argument that J_u is effectively rank-one, so the headline mechanism is unsupported.\n\nWhat the paper does well: the motivation is clear, the method is simple, and the experiments cover multiple datasets and architectures. The comparison against Semdiff on local vs. non-local MSE is a useful direct test. The authors honestly acknowledge limits for very small masks and global attributes.\n\nSoft spots beyond the theory: the connection between the optimization objective (Eq. 13) and the projection is asserted, not proven; the selection of the right singular vector for a desired attribute is not specified; and the quantitative comparison against supervised baselines uses only the smile attribute, with no error bars, and the paper's \"state-of-the-art\" claim overreaches. The appendix, where implementation details are promised, is missing from the arXiv version.\n\nThis is a likeable paper with a good idea, but the key equation does not do the advertised work. A referee should ask for a proper derivation (or a clear explanation of why a rank-one approximation suffices) and a more complete comparison. I would send it to review, but with the expectation of major revision.\n\nBest,\n[Your name]","headline":"The core projection (Eq. 14) is a rank-one Gram-Schmidt step while the Jacobians are matrices, so the derivation does not support the paper's central claim of suppressing all outside-region changes.","tokens_in":12110,"tokens_out":4269,"would_cite":false,"duration_ms":37895,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a coarse mask alone can steer semantic edits in a pretrained diffusion model by projecting the masked region's Jacobian orthogonal to the unmasked region's Jacobian, achieving local editing without labels or training.","keywords":["region-based image editing","diffusion models","unsupervised semantic discovery","Jacobian projection","h-space","mask-guided editing","face attribute editing","training-free editing"],"falsifier":"For a fixed image and timestep, compute the masked and unmasked Jacobians, then compare the paper's projected direction with the direction that actually maximizes $\\|J_m h\\|/\\|J_u h\\|$, for instance the leading generalized eigenvector of $J_m^T J_m$ relative to $J_u^T J_u$. If the projected direction is not among the optimizers, the core equivalence fails; the test can be run on one CelebA-HQ image or on a small synthetic linear map.","tokens_in":10970,"feed_emoji":"🖌️","tokens_out":8656,"duration_ms":69152,"temperature":0.7,"pith_summary":"This paper aims to make local semantic editing of pretrained diffusion models work without any labels, extra training, or fine segmentation masks. Its idea is to find a direction in the model's internal h-space that changes pixels inside a user-specified region while leaving the complement almost untouched. The direction comes from the Jacobian of the masked noise-prediction output, projected orthogonal to the Jacobian of the unmasked output. The authors report that on smile editing this unsupervised procedure beats supervised baselines on image quality, identity preservation, and perceptual distance, while keeping changes outside the mask smaller than the closest unsupervised competitor. If correct, it turns a rough bounding box into a sufficient interface for precise attribute editing in pretrained U-Net diffusion models.","feed_headline":"Mask-only projection keeps diffusion edits inside the chosen region","feed_subtitle":"No labels, no fine-tuning: one Gram-Schmidt-style projection separates local attributes from global structure.","key_machinery":"The object carrying the argument is the Jacobian of the masked noise-prediction network with respect to the h-space latent, where h-space is the semantic latent space formed by the U-Net bottleneck skip connections. The load-bearing step is a single Gram-Schmidt-style orthogonal projection: the masked Jacobian is projected onto the unmasked Jacobian and the projection is subtracted, so the retained direction is the part of the masked sensitivity that is orthogonal to the non-masked sensitivity. Singular vectors of this projected Jacobian are approximated by power iteration, and the leading vector is applied as the edit direction during DDIM sampling.","core_discovery":"The paper's central claim is that a local semantic edit direction can be computed from two Jacobians of the denoising network. With $f(h_t)=\\epsilon^\\theta_t(x_t,h_t)$ the noise prediction, the masked Jacobian $J_m=\\partial(\\epsilon^\\theta_t\\odot M)/\\partial h_t$ encodes the sensitivity of the region of interest, and $J_u$ encodes the sensitivity of the complement. The paper proposes $J = J_m - \\frac{J_m\\cdot J_u}{J_u\\cdot J_u}J_u$, a single orthogonal projection of the masked direction onto the subspace orthogonal to the unmasked direction, and claims this solves the objective of maximizing masked-region change while minimizing change elsewhere. The resulting vector is used as an edit direction in h-space across selected denoising time steps. Because changes in h-space propagate to x-space through the noise prediction network, suppressing the unmasked Jacobian component is claimed to suppress unwanted global changes.","pith_inferences":["The paper leaves implicit that the projection is only one Gram-Schmidt step against a single unmasked Jacobian; iterating it over several regional Jacobians or using a generalized eigenvector of $J_m^T J_m$ relative to $J_u^T J_u$ could satisfy the stated max/min objective more faithfully.","A testable extension is that because only a coarse mask is required, the interface should transfer to region-constrained editing outside faces, such as object-level or medical-image editing, wherever a U-Net diffusion model is available.","The paper also leaves implicit that h-space locality implies a family of timestep-dependent edit directions; checking whether a single direction stays optimal across all timesteps would clarify the robustness of the mask-only interface."],"forward_implications":["A user can edit a local attribute with only a rough bounding box; no attribute labels, text prompt, or fine-tuning are needed.","The same mask can yield multiple distinct semantic edits, such as smile, left-slant, and right-slant for a mouth region.","Outside-mask preservation should transfer across datasets and U-Net diffusion architectures, since the method relies only on h-space Jacobians.","On smile editing, the unsupervised result can beat supervised edits on FID, identity similarity, and LPIPS, so supervision is not a prerequisite for high-quality local edits."],"supporting_citations":[{"why":"Defines h-space as a semantic latent space in diffusion models and introduces the Asyrp editing procedure that this work builds on.","marker":"Kwon, Jeong, and Uh 2022"},{"why":"Semdiff contributes the masked local Jacobian idea and serves as the closest unsupervised baseline.","marker":"Haas et al. 2024"},{"why":"Boundary Diffusion is the supervised baseline that the paper compares against, motivating the need for label-free editing.","marker":"Zhu et al. 2024"},{"why":"DDIM inversion and sampling supply the denoising trajectory on which the edit direction is applied.","marker":"Song, Meng, and Ermon 2020"},{"why":"DDPM is the base generative model family whose U-Net architecture this method edits.","marker":"Ho, Jain, and Abbeel 2020"}],"fun_headline_variants":["Jacobian projection confines diffusion edits to one region","No-training local edits: project out the rest of the image","One Gram-Schmidt step keeps diffusion edits in the mask","Diffusion region editing without labels or extra training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the assumption that one orthogonal projection of the masked Jacobian onto the unmasked Jacobian is equivalent to maximizing masked-region change while minimizing unmasked-region change, and the paper does not prove this equivalence.","fun_headline_variants_meta":{"raw":{"variants":["Jacobian projection confines diffusion edits to one region","No-training local edits: project out the rest of the image","One Gram-Schmidt step keeps diffusion edits in the mask","Diffusion region editing without labels or extra training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000666,"raw_usage":{"total_tokens":3004,"prompt_tokens":875,"completion_tokens":2129,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":2065}},"tokens_in":491,"tokens_out":2129,"duration_ms":14651,"temperature":1.0,"reasoning_tokens":2065,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:36:00.701494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a fixed image and timestep, compute the masked and unmasked Jacobians, then compare the paper's projected direction with the direction that actually maximizes $\\|J_m h\\|/\\|J_u h\\|$, for instance the leading generalized eigenvector of $J_m^T J_m$ relative to $J_u^T J_u$. If the projected direction is not among the optimizers, the core equivalence fails; the test can be run on one CelebA-HQ image or on a small synthetic linear map.","supporting_citations":[{"cited_title":"S.; and Michaeli, T","cited_arxiv_id":null,"evidence_quote":"Semdiff contributes the masked local Jacobian idea and serves as the closest unsupervised baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Boundary Diffusion is the supervised baseline that the paper compares against, motivating the need for label-free editing."}],"review_version":1}