Pith. sign in

REVIEW 4 major objections 6 minor 73 references

HarmonPaint: Harmonized Training-Free Diffusion Inpainting

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By masking object-background attention and substituting mean key/value features, HarmonPaint produces structurally faithful and stylistically harmonized inpaints with no training.

desk verdict A genuinely training-free inpainting recipe (self-attention quadrant masking plus key/value statistics injection) with strong qualitative results, but the evaluation is too self-referential (StyleID-made benchmarks) and the masking mechanism is under-specified without row renormalization. read the letter →

arxiv 2507.16732 v1 pith:LTU4GLYT submitted 2025-07-22 cs.CV

Ying Li , Xinzhe Li , Yong Du , Yangyang Xu , Junyu Dong , Shengfeng He This is my paper
classification cs.CV
keywords text-guidedimageinpaintingtraining-freeself-attentionmaskingstyletransferdiffusionmodelsharmonizationattentionsteerloss
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HarmonPaint is a training-free modification of the Stable Diffusion inpainting model, and the paper's central claim is that it can fill masked regions with content that is both structurally faithful and stylistically consistent with the rest of the image. Existing text-guided inpainting methods either retrain or fine-tune specialized heads, yet still produce unnatural boundaries or style mismatches when the image has a strong artistic style. HarmonPaint instead intervenes directly in the attention blocks: it masks object-background interactions in self-attention maps so the masked region develops its own principal components, and it copies mean key and value features from the unmasked region into the masked region so the new content adopts the surrounding style. The paper reports that this training-free recipe beats fine-tuned inpainting models and a large generative baseline on stylized benchmarks and in a user study, and that it degrades only when the mask covers more than 90 percent of the image. If true, the result shows that attention-level control alone, without new parameters, is enough for harmonized text-guided inpainting across styles.

What carries the argument

The central object is the self-attention map of the diffusion U-Net. HarmonPaint partitions this map into object-object, background-background, and object-background blocks using the inpainting mask, zeroes the object-background block with a soft version of the mask to avoid hard boundaries, and feeds the reweighted map back into the encoder so masked and unmasked regions occupy distinct principal components. The style-transfer mechanism is the Mask-Adjusted Key-Value Strategy: the mean of K and V computed over unmasked patches replaces the K and V of masked patches, with the original K concatenated alongside the replaced key as [K, lambda eK] to keep content and style in balance. The Attention Steer Loss, built from cross-attention maps at resolutions 16 and 32, pushes prompt-token attention to concentrate inside the mask, and the division strategy runs structural masking in the early denoising stage and style transfer in the late stage.

What would settle it

Take an image whose unmasked region contains two distinctly styled halves, inpaint an object in the middle with HarmonPaint, and inspect whether the generated object adopts a coherent blend of both styles or collapses to an average that matches neither; the second outcome would falsify the claim that mean key/value features carry the surrounding style.

Watch

Extended reading notes

Core claim

On its own terms, the paper discovers that the failure of diffusion inpainting to harmonize with its background can be traced to two properties of self-attention, and both can be corrected without training. First, self-attention maps in the U-Net encoder treat masked and unmasked regions as one principal component, so background features leak into the inpainted object; masking the object-background entries of the attention map separates the regions and preserves structure. Second, the key and value features of decoder self-attention carry style, so replacing the masked region's K and V with the mean of the unmasked region's K and V transfers the image's style into the generated content; concatenating the original and replaced keys with a strength parameter lambda prevents the style transfer from destroying object shape. An auxiliary cross-attention loss concentrates each prompt token's attention inside the mask, and a two-stage denoising schedule applies structural control early and style control late. The paper's conclusion is that these mechanisms together produce inpainting that is text-aligned, structurally coherent, and stylistically unified across oil-painting, sketch, and natural-image inputs, and that the approach outperforms both trained inpainting heads and larger generative models on the metrics and user preferences reported.

Load-bearing premise

The load-bearing premise is that averaging the key and value features from the unpainted region captures the image's style well enough that copying those averages into the masked region transfers style without eroding object content; this is borrowed from a single prior result on style injection rather than established inside the paper.

Editorial extensions

If this is right

  • No retraining or fine-tuning is needed: the same frozen diffusion inpainting weights handle stylized images, natural images, segmentation masks, bounding-box masks, and loose user scribbles by changing attention maps alone.
  • The method's style transfer is controlled by a single strength parameter lambda; setting it to zero reverts the model to ordinary inpainting, and lowering it to 0.8 adapts the same pipeline to natural images.
  • Because the masked region becomes its own principal component in self-attention, the generated object follows the mask shape even when the mask carries no semantic outline, as shown with arbitrary butterfly-shaped masks.
  • The reported gains over fine-tuned inpainting heads and a large generative baseline, if they hold, imply that task-specific training data for style is not a prerequisite for harmonized inpainting.
  • The paper's own stated bound is that the method relies on the unmasked region as its style source; when more than 90 percent of the image is missing, the style signal becomes insufficient for harmonization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same mean key/value substitution should apply to other localized edits, such as object replacement, relighting, or background swap, wherever the unmasked area defines the target style; the paper only demonstrates inpainting.
  • Beyond the paper: the global mean assumes one dominant style in the unmasked region, so a natural extension is region-adaptive means, for example clustering unmasked K/V features, for images containing several distinct style zones.
  • Beyond the paper: because the benchmark images are themselves produced by an external style-transfer model, the reported harmony is measured against that model's notion of style; real artwork with independent style labels would be a stiffer test.
  • Beyond the paper: the concatenation trick means attention is computed against both the original and the style-averaged key in one softmax, so whether lambda should vary over denoising steps rather than staying fixed is an open question that could refine the style-content trade-off.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. HarmonPaint proposes a training-free extension of Stable Diffusion Inpainting that intervenes in self-attention to improve structural fidelity and stylistic harmony. The method has three main ingredients: a Self-Attention Masking Strategy (SAMS) that suppresses object-background attention in the U-Net encoder, a Mask-Adjusted Key-Value Strategy (MAKVS) that propagates mean-pooled key/value features from unmasked to masked regions, and an Attention Steer Loss that concentrates cross-attention on the masked region. The denoising schedule is also split into a structure-focused stage and a style-focused stage. Experiments are reported on two stylized datasets derived from MSCOCO and OpenImages, with quantitative metrics, ablations, a user study, and comparisons against BrushNet, PowerPaint, ControlNet Inpainting, Blended Latent Diffusion, and FLUX.

Significance. If the claims are substantiated, the paper would show that a fully training-free attention intervention can match or beat dedicated fine-tuned inpainting heads on stylized benchmarks, which is a practically valuable result. The paper has several strengths: the two mechanisms are cleanly separated, the qualitative comparisons are extensive, the authors include a user study, a comparison with FLUX, mask-sensitivity experiments, and an honest limitation statement for large masked regions. However, the current evidence is not yet conclusive because the evaluation benchmark is generated by the same StyleID mechanism that motivates MAKVS, and because the core equations contain dimensional ambiguities that make the method not fully reproducible as written.

major comments (4)
  1. [Sec. IV-B, Eqs. (11)-(12)] The central MAKVS operation is dimensionally inconsistent as written. With Q, K, V of shape [HW, d], the term [K, λeK]^T has shape [2d, HW], so Q × [K, λeK]^T is not a valid matrix product unless Q is also expanded to [HW, 2d], which is not stated. Likewise, Eq. (12) writes the output as eA_self × [V; eV], which changes the output feature dimension to 2d without specifying any subsequent projection or reshaping. Since MAKVS is a load-bearing component, the authors should give exact tensor shapes (including heads and batch dimensions) or provide pseudocode. Without this, the method is not reproducible and the reported 'only key's mean' baseline in Fig. 3 cannot be checked.
  2. [Sec. IV-A, Eqs. (4)-(7)] Because masking is applied after softmax without row renormalization, SAMS does more than 'selectively mask obj-bg interactions': it also scales down the output features for masked-region tokens by the remaining row sum. The same issue applies to the soft-mask variant in Eq. (7), whose row sums are also uncontrolled. The PCA visualizations and Table III are therefore consistent with an alternative explanation in which the structural-fidelity gain comes from attenuating masked-region feature magnitudes rather than from separating object/background principal components. Please compare SAMS against a row-normalized version of Eqs. (4)-(6), and report either attention row sums or feature norms across layers and timesteps. If the row-normalized version retains the gains, the semantic-separation interpretation is supported; if not, the paper should be reworded.
  3. [Sec. V-A3 and Tables I-II] The evaluation is circular in a way that favors the method. The Stylized-COCO and Stylized-OpenImages benchmarks are produced by applying StyleID to natural images, and MAKVS is directly motivated by StyleID's observation that K and V carry style. The method is therefore evaluated in-distribution for its core style-transfer assumption. Please add at least one benchmark that does not rely on StyleID, for example real stylized images (e.g., artwork with object masks), or synthetic stylization performed with a different model family (e.g., a GAN-based style transfer method), and report the same metrics per method. This is necessary to support the claim that HarmonPaint generalizes across diverse styles rather than only reproducing the style statistics of the model that generated the benchmark.
  4. [Sec. V-B2, Tables I-II, and Appendix A] The quantitative claims are not accompanied by any measure of variance or statistical testing. The datasets contain only 50 images per source, and several reported differences are small (e.g., CS 28.86 vs. 28.12 for PowerPaint in Table I; CMMD 0.103 vs. 0.109 for BrushNet). Please report standard deviations or confidence intervals across multiple seeds and run a paired significance test (e.g., Wilcoxon signed-rank) for the main CS/IR/AS/CMMD comparisons. The user study in Appendix A likewise reports single percentages from 40 participants on 20 images without confidence intervals; a paired preference test would strengthen the claim of user-level superiority.
minor comments (6)
  1. [Sec. V-C3, Fig. 9 caption] The caption says 'when λ exceeds 0.8, the quality of the inpainted content begins to noticeably decline,' but the figure varies τ, not λ. This is likely a typo and should be corrected.
  2. [Sec. IV-A, Eq. (7)] The soft mask is written as cM_f = (1-τ)M_f + τ/HW, but HW is a scalar count while M_f is a vector; the intended expression is presumably a constant vector with entries τ/HW applied to all positions. Please clarify the notation.
  3. [Sec. IV-B, Eq. (10) and Fig. 3] The text says the baseline uses 'only the key's mean,' but Eq. (10) defines eK as preserving unmasked keys and replacing only masked keys. It should be explicitly stated whether the baseline in Fig. 3 replaces all keys with the mean or only masked keys, since this changes the interpretation of the comparison.
  4. [Sec. V-A1 and Sec. IV-C] The layer ranges are not fully specified: SAMS is applied to layers 2-6 and MAKVS to the final 8 layers, but it is unclear whether 'layers' refer to encoder/decoder blocks, how heads are treated, and whether the ranges overlap. Please give a complete per-block configuration for the U-Net.
  5. [Fig. 6 caption] The caption reads 'Input w/o SAMS HarmonPaint w/o w/o MAKVS,' which appears to contain a duplicated 'w/o' and is confusing. The intended labels are likely 'Input,' 'w/o SAMS,' 'HarmonPaint,' and 'w/o MAKVS.'
  6. [Appendix C] FLUX is only compared in the appendix and not listed among the main competitors in Section V-A2. Given that the method is training-free, FLUX is a relevant baseline and should at least be cited and positioned in the main experimental section.

Circularity Check

1 steps flagged · score 2.0 of 10

Stylized benchmarks are generated by StyleID, the same model whose K/V-style premise underlies MAKVS, giving the stylistic-harmony comparison a self-confirming component; the rest of the pipeline is not circular.

  1. other [Section V-A3 (Test Benchmarks) and Section IV-B (MAKVS), Eqs. (10)-(12)]
    "K and V of self-attention in the U-Net decoder is capable of extracting style information [61], and we propose a Mask-Adjusted Key-Value Strategy (MAKVS) to integrate these features into the masked region for stylistic harmony... we created a Stylized-COCO dataset by selecting 50 reference images in various styles from WikiArt [67] and applying StyleID [61] to transform MSCOCO images into different artistic styles."

    MAKVS replaces the masked region's K and V with the mean of the unmasked region's K and V, which is the same self-attention K/V mechanism that StyleID manipulates to create the stylized test images. On the Stylized-COCO and Stylized-OpenImages benchmarks, the 'correct' style of the masked region is, by construction, the StyleID-stylized background; copying its K/V mean into the masked region therefore makes the stylistic-harmony component of the evaluation partially self-confirming rather than an independent test of harmonization. This does not invalidate the structural-fidelity claims (SAMS, Ls) or the general-inpainting results, so the overall circularity is limited.

full rationale

The paper's structural-fidelity chain is not circular: SAMS is a concrete post-softmax masking operation whose effect on principal components is a stated empirical observation, and its benefit to final image quality is measured against external competitors rather than derived from the masking formula itself. The Ls loss is a direct optimization objective, not a fitted parameter renamed as a prediction. The K/V-style premise of MAKVS is imported from an external prior work (StyleID), so it is a borrowed assumption rather than a self-citation; however, the evaluation set is also generated by StyleID, which gives the stylistic-harmony comparison a self-confirming flavor for the MAKVS component specifically. No load-bearing self-citations were found: references that may share authors with this paper appear only in related-work surveys and do not justify the core mechanism. The unnormalized row sums in Eqs. (4)-(7) are a mechanistic ambiguity that could explain part of SAMS's effect, but that is a correctness or reproducibility concern, not circularity. Overall, the central structural-fidelity derivation is self-contained, so the paper receives a low score with one flagged benchmark-construction issue.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or architectural entities; it only reweights existing attention maps and key-value statistics. The free parameters are tuned by hand through visual inspection, which is acceptable for a method paper but reduces reproducibility.

free parameters (4)
  • tau (smoothing factor) = 0.1
    Controls how much the binary mask is softened in SAMS. Chosen by visual inspection and small sensitivity analysis in Section V-C3.
  • lambda (style strength) = 1.4 (stylized), 0.8 (general)
    Balances content and style in the concatenated key injection. Tuned by visual inspection as shown in Fig. 8.
  • eta (stage split) = 0.6
    Splits the denoising trajectory into a structure phase and a style phase. Set manually with no reported ablation.
  • Layer ranges for SAMS and MAKVS = SAMS on layers 2-6, MAKVS on final 8 layers
    Chosen without ablation; no analysis of sensitivity to layer choice.
assumptions (3)
  • domain assumption The U-Net encoder's self-attention principal components capture spatial layout, and separating masked from unmasked regions in this space improves structural fidelity.
    Supported by self-attention PCA visualizations in Fig. 2 and Fig. 12, but the actual causal link from PCA separation to final quality is asserted, not proven.
  • domain assumption The K and V projections in self-attention of the U-Net decoder carry style information that can be averaged over unmasked regions and injected into masked regions.
    Cites StyleID [61] for the general claim, but StyleID is a style transfer method between two images; applying it as local style transfer within one image is an extrapolation.
  • domain assumption The cross-attention loss Ls, defined in Eq. (9), increases text alignment without causing adversarial or degenerate attention maps.
    No analysis of potential side effects of maximizing attention inside the mask, such as attention collapse or ignoring background context that could help object placement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HarmonPaint: Harmonized Training-Free Diffusion Inpainting." pith.science (2026). https://pith.science/paper/LTU4GLYT

@misc{pith2026250716732,
  author       = {Pith},
  title        = {Pith review of: HarmonPaint: Harmonized Training-Free Diffusion Inpainting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LTU4GLYT}},
  note         = {Machine review of arXiv:2507.16732}
}
read the original abstract

Existing inpainting methods often require extensive retraining or fine-tuning to integrate new content seamlessly, yet they struggle to maintain coherence in both structure and style between inpainted regions and the surrounding background. Motivated by these limitations, we introduce HarmonPaint, a training-free inpainting framework that seamlessly integrates with the attention mechanisms of diffusion models to achieve high-quality, harmonized image inpainting without any form of training. By leveraging masking strategies within self-attention, HarmonPaint ensures structural fidelity without model retraining or fine-tuning. Additionally, we exploit intrinsic diffusion model properties to transfer style information from unmasked to masked regions, achieving a harmonious integration of styles. Extensive experiments demonstrate the effectiveness of HarmonPaint across diverse scenes and styles, validating its versatility and performance.

Figures

Figures reproduced from arXiv: 2507.16732 by the authors.

Figure 1
Figure 1. We propose HarmonPaint, a training-free inpainting framework that achieves harmonized, text-aligned inpainting results. In comparison to existing methods such as (b) ControlNet Inpainting (CNI) [1], (c) BrushNet [2], (d) PowerPaint [3], and (e) Blended Latent Diffusion (BLD) [4], (f) our approach accurately captures image style and produces structural fidelity results. The bottom row showcases various harmonized inp… view at source ↗
Figure 2
Figure 2. Overview of the HarmonPaint. HarmonPaint introduces two key mechanisms to enhance inpainting quality: (1) Self￾Attention Masking Strategy, which reweights self-attention maps to ensure structural fidelity, and (2) Mask-Adjusted Key-Value Strategy, transferring style information from unmasked to masked regions to maintain stylistic harmony. a reference image for semantic guidance, limiting its use in text-guided inpa… view at source ↗
Figure 3
Figure 3. Comparison results between HarmonPaint and the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison with state-of-the-art methods on the Stylized-COCO Dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Qualitative comparison with state-of-the-art methods on the Stylized-OpenImages Dataset. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Qualitative ablation study on proposed components. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Masking bg-bg interactions shows little effect on back [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 7
Figure 7. Figure 7: Experiment results of masking the background [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Visualization of the effects of λ. As λ increases, the stylistic harmony of the image improves. However, when λ exceeds 1.4, the quality of the inpainted content begins to noticeably decline. it does not benefit object generation. Conversely, masking obj-obj interactio…
Figure 9
Figure 9. Figure 9: Visualization of the effects of τ . As τ increases, the structural fidelity of the image improves. However, when λ exceeds 0.8, the quality of the inpainted content begins to noticeably decline. "sheep" “flower” ”bird“ ”dumplings“ Input Result [PITH_FULL_IMAGE:figures…
Figure 11
Figure 11. Figure 11: When the image is more than 90% missing, especially [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]
Figure 12
Figure 12. Figure 12: Additional PCA visualizations of self-attention maps show that as timesteps decrease, our method effectively generates [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: Additional cross-attention map visualizations at resolutions 16 and 32. Red boxes indicate instances where the text [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 15
Figure 15. Figure 15: Our method demonstrates strong adaptability to var [PITH_FULL_IMAGE:figures/full_fig_p012_15.png]
Figure 14
Figure 14. Figure 14: Qualitative comparison with FLUX. patches of the image, failing to concentrate on the target area within the mask. This leads to inpainting results that are poorly aligned with the text prompt. By incorporating Ls, our method effectively refines the noise map, ensurin…
Figure 17
Figure 17. Figure 17: Visualization of partial inpainting. APPENDIX G ADDITIONAL VISUAL RESULTS We provide additional qualitative comparisons with other competitors in [PITH_FULL_IMAGE:figures/full_fig_p013_17.png]
Figure 18
Figure 18. Figure 18: Additional qualitative comparisons on the Stylized-COCO dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_18.png]
Figure 19
Figure 19. Figure 19: Additional qualitative comparisons on the Stylized-OpenImages dataset. [PITH_FULL_IMAGE:figures/full_fig_p015_19.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

73 extracted references · 51 canonical work pages

  1. [1]

    Adding conditional control to text-to-image diffusion models,

    L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in ICCV, 2023, pp. 3836–3847

    2023

  2. [2]

    Brushnet: A plug-and-play image inpainting model with decomposed dual-branch diffusion,

    X. Ju, X. Liu, X. Wang, Y . Bian, Y . Shan, and Q. Xu, “Brushnet: A plug-and-play image inpainting model with decomposed dual-branch diffusion,” in ECCV. Springer, 2024, pp. 150–168

    work page 2024

  3. [3]

    A task is worth one word: Learning with task prompts for high-quality versatile image inpainting,

    J. Zhuang, Y . Zeng, W. Liu, C. Yuan, and K. Chen, “A task is worth one word: Learning with task prompts for high-quality versatile image inpainting,” in ECCV. Springer, 2024, pp. 195–211

    work page 2024

  4. [4]

    Blended latent diffusion,

    O. Avrahami, O. Fried, and D. Lischinski, “Blended latent diffusion,” ACM TOG, vol. 42, no. 4, pp. 1–11, 2023

    work page 2023

  5. [5]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” NeurIPS, vol. 33, pp. 6840–6851, 2020

    2020

  6. [6]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in CVPR, 2022, pp. 10 684–10 695

    2022

  7. [7]

    Photorealistic text-to-image diffusion models with deep language understanding,

    C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans et al., “Photorealistic text-to-image diffusion models with deep language understanding,” NeurIPS, vol. 35, pp. 36 479–36 494, 2022

    work page 2022

  8. [8]

    Diffusion models beat gans on image synthesis,

    P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” NeurIPS, vol. 34, pp. 8780–8794, 2021

    2021

Show all 73 references
  1. [9]

    Score-based generative modeling through stochastic differ- ential equations,

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differ- ential equations,” in ICLR, 2021

    2021

  2. [10]

    Patch- match: A randomized correspondence algorithm for structural image editing,

    C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman, “Patch- match: A randomized correspondence algorithm for structural image editing,” ACM TOG, vol. 28, no. 3, p. 24, 2009

    2009

  3. [11]

    Generating diverse structure for image inpainting with hierarchical vq-vae,

    J. Peng, D. Liu, S. Xu, and H. Li, “Generating diverse structure for image inpainting with hierarchical vq-vae,” in CVPR, 2021, pp. 10 775– 10 784

    2021

  4. [12]

    Pluralistic image completion,

    C. Zheng, T.-J. Cham, and J. Cai, “Pluralistic image completion,” in CVPR, 2019, pp. 1438–1447

    2019

  5. [13]

    Image inpainting with cascaded modulation gan and object-aware training,

    H. Zheng, Z. Lin, J. Lu, S. Cohen, E. Shechtman, C. Barnes, J. Zhang, N. Xu, S. Amirghodsi, and J. Luo, “Image inpainting with cascaded modulation gan and object-aware training,” in ECCV. Springer, 2022, pp. 277–296

    2022

  6. [14]

    Ciri: curricular inactivation for residue-aware one-shot video inpainting,

    W. Zheng, C. Xu, X. Xu, W. Liu, and S. He, “Ciri: curricular inactivation for residue-aware one-shot video inpainting,” inICCV, 2023, pp. 13 012– 13 022

    2023

  7. [15]

    Generative diffusion prior for unified image restoration and enhancement,

    B. Fei, Z. Lyu, L. Pan, J. Zhang, W. Yang, T. Luo, B. Zhang, and B. Dai, “Generative diffusion prior for unified image restoration and enhancement,” in CVPR, 2023, pp. 9935–9946

    2023

  8. [16]

    Repaint: Inpainting using denoising diffusion probabilistic models,

    A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool, “Repaint: Inpainting using denoising diffusion probabilistic models,” in CVPR, 2022, pp. 11 461–11 471

    2022

  9. [17]

    Blended diffusion for text- driven editing of natural images,

    O. Avrahami, D. Lischinski, and O. Fried, “Blended diffusion for text- driven editing of natural images,” in CVPR, 2022, pp. 18 208–18 218

    2022

  10. [18]

    Smartbrush: Text and shape guided object inpainting with diffusion model,

    S. Xie, Z. Zhang, Z. Lin, T. Hinz, and K. Zhang, “Smartbrush: Text and shape guided object inpainting with diffusion model,” in CVPR, 2023, pp. 22 428–22 437

    2023

  11. [19]

    Imagen editor and editbench: Advancing and evaluating text-guided image inpainting,

    S. Wang, C. Saharia, C. Montgomery, J. Pont-Tuset, S. Noy, S. Pelle- grini, Y . Onoe, S. Laszlo, D. J. Fleet, R. Soricut et al., “Imagen editor and editbench: Advancing and evaluating text-guided image inpainting,” in CVPR, 2023, pp. 18 359–18 369

    2023

  12. [20]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” NeurIPS, 2017

    2017

  13. [21]

    Generative adversarial networks,

    I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio, “Generative adversarial networks,” Communications of the ACM , vol. 63, no. 11, pp. 139–144, 2020

    2020

  14. [22]

    Large scale gan training for high fidelity natural image synthesis,

    A. Brock, J. Donahue, and K. Simonyan, “Large scale gan training for high fidelity natural image synthesis,” in ICLR, 2019

    2019

  15. [23]

    Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks,

    H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas, “Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks,” in ICCV, 2017, pp. 5907–5915

    2017

  16. [24]

    A style-based generator architecture for generative adversarial networks,

    T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in CVPR, 2019, pp. 4401–4410

    2019

  17. [25]

    Attngan: Fine-grained text to image generation with attentional generative adversarial networks,

    T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He, “Attngan: Fine-grained text to image generation with attentional generative adversarial networks,” in CVPR, 2018, pp. 1316–1324

    2018

  18. [26]

    Cross-modal contrastive learning for text-to-image generation,

    H. Zhang, J. Y . Koh, J. Baldridge, H. Lee, and Y . Yang, “Cross-modal contrastive learning for text-to-image generation,” in CVPR, 2021, pp. 833–842

    2021

  19. [27]

    Analyzing and improving the image quality of stylegan,

    T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” in CVPR, 2020, pp. 8110–8119

    2020

  20. [28]

    Progressive growing of gans for improved quality, stability, and variation,

    T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” in ICLR, 2018

    2018

  21. [29]

    Alias-free generative adversarial networks,

    T. Karras, M. Aittala, S. Laine, E. H ¨ark¨onen, J. Hellsten, J. Lehtinen, and T. Aila, “Alias-free generative adversarial networks,” NeurIPS, vol. 34, pp. 852–863, 2021

    2021

  22. [30]

    Veegan: Reducing mode collapse in gans using implicit variational learning,

    A. Srivastava, L. Valkov, C. Russell, M. U. Gutmann, and C. Sutton, “Veegan: Reducing mode collapse in gans using implicit variational learning,” NeurIPS, vol. 30, 2017

    2017

  23. [31]

    Denoising diffusion implicit models,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” in ICLR, 2021

    2021

  24. [32]

    Classifier-free diffusion guidance,

    J. Ho and T. Salimans, “Classifier-free diffusion guidance,” arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  25. [33]

    Hierarchical text-conditional image generation with clip latents,

    A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” arXiv preprint arXiv:2204.06125, vol. 1, no. 2, p. 3, 2022

    2022 arXiv

  26. [34]

    Laion- 5b: An open large-scale dataset for training next generation image-text models,

    C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsmanet al., “Laion- 5b: An open large-scale dataset for training next generation image-text models,” NeurIPS, vol. 35, pp. 25 278–25 294, 2022

    2022

  27. [35]

    Improving text-guided object inpainting with semantic pre-inpainting,

    Y . Chen, J. Chen, Y . Pan, Y . Li, T. Yao, Z. Chen, and T. Mei, “Improving text-guided object inpainting with semantic pre-inpainting,” in ECCV. Springer, 2025, pp. 110–126

    2025

  28. [36]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in ICML. PMLR, 2021, pp. 8748–8763

    2021

  29. [37]

    Udifftext: A unified framework for high-quality text synthesis in arbitrary images via character-aware diffusion models,

    Y . Zhao and Z. Lian, “Udifftext: A unified framework for high-quality text synthesis in arbitrary images via character-aware diffusion models,” in European Conference on Computer Vision. Springer, 2024, pp. 217– 233

    2024

  30. [38]

    Deep image harmonization with globally guided feature transformation and relation distillation,

    L. Niu, L. Tan, X. Tao, J. Cao, F. Guo, T. Long, and L. Zhang, “Deep image harmonization with globally guided feature transformation and relation distillation,” in ICCV, 2023, pp. 7723–7732

    2023

  31. [39]

    Tf-icon: Diffusion-based training- free cross-domain image composition,

    S. Lu, Y . Liu, and A. W.-K. Kong, “Tf-icon: Diffusion-based training- free cross-domain image composition,” in ICCV, 2023, pp. 2294–2305

    2023

  32. [40]

    Paste, inpaint and harmonize via denoising: Subject-driven image editing with pre-trained diffusion model,

    X. Zhang, J. Guo, P. Yoo, Y . Matsuo, and Y . Iwasawa, “Paste, inpaint and harmonize via denoising: Subject-driven image editing with pre-trained diffusion model,” arXiv preprint arXiv:2306.07596 , 2023

    2023 arXiv

  33. [41]

    Relightful harmonization: Lighting-aware portrait back- ground replacement,

    M. Ren, W. Xiong, J. S. Yoon, Z. Shu, J. Zhang, H. Jung, G. Gerig, and H. Zhang, “Relightful harmonization: Lighting-aware portrait back- ground replacement,” in CVPR, 2024, pp. 6452–6462. 10

    2024

  34. [42]

    Neural preset for color style transfer,

    Z. Ke, Y . Liu, L. Zhu, N. Zhao, and R. W. Lau, “Neural preset for color style transfer,” in CVPR, 2023, pp. 14 173–14 182

    2023

  35. [43]

    Region-aware adaptive instance normalization for image harmonization,

    J. Ling, H. Xue, L. Song, R. Xie, and X. Gu, “Region-aware adaptive instance normalization for image harmonization,” in CVPR, 2021, pp. 9361–9370

    2021

  36. [44]

    Dccf: Deep comprehensible color filter learning framework for high-resolution image harmonization,

    B. Xue, S. Ran, Q. Chen, R. Jia, B. Zhao, and X. Tang, “Dccf: Deep comprehensible color filter learning framework for high-resolution image harmonization,” in ECCV. Springer, 2022, pp. 300–316

    2022

  37. [45]

    Paint by example: Exemplar-based image editing with diffusion models,

    B. Yang, S. Gu, B. Zhang, T. Zhang, X. Chen, X. Sun, D. Chen, and F. Wen, “Paint by example: Exemplar-based image editing with diffusion models,” in CVPR, 2023, pp. 18 381–18 391

    2023

  38. [46]

    Taming transformers for high- resolution image synthesis,

    P. Esser, R. Rombach, and B. Ommer, “Taming transformers for high- resolution image synthesis,” in CVPR, 2021, pp. 12 873–12 883

    2021

  39. [47]

    U-net: Convolutional networks for biomedical image segmentation,

    O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in MICCAI. Springer, 2015, pp. 234–241

    2015

  40. [48]

    Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing,

    M. Cao, X. Wang, Z. Qi, Y . Shan, X. Qie, and Y . Zheng, “Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing,” in ICCV, 2023, pp. 22 560–22 570

    2023

  41. [49]

    Plug-and-play diffusion features for text-driven image-to-image translation,

    N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel, “Plug-and-play diffusion features for text-driven image-to-image translation,” in CVPR, 2023, pp. 1921–1930

    2023

  42. [50]

    Analysis of a complex of statistical variables into principal components

    H. Hotelling, “Analysis of a complex of statistical variables into principal components.” Journal of educational psychology , vol. 24, no. 6, p. 417, 1933

    1933

  43. [51]

    Faster diffusion: Rethinking the role of unet encoder in diffusion models,

    S. Li, T. Hu, F. Shahbaz Khan, L. Li, S. Yang, Y . Wang, M.-M. Cheng, and J. Yang, “Faster diffusion: Rethinking the role of unet encoder in diffusion models,” arXiv e-prints, pp. arXiv–2312, 2023

    2023

  44. [52]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations , 2020

    2020

  45. [53]

    Dense text-to-image generation with attention modulation,

    Y . Kim, J. Lee, J.-H. Kim, J.-W. Ha, and J.-Y . Zhu, “Dense text-to-image generation with attention modulation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 7701–7711

    2023

  46. [54]

    Be yourself: Bounded attention for multi-subject text-to-image generation,

    O. Dahary, O. Patashnik, K. Aberman, and D. Cohen-Or, “Be yourself: Bounded attention for multi-subject text-to-image generation,” in Euro- pean Conference on Computer Vision . Springer, 2024, pp. 432–448

    2024

  47. [55]

    Attentive eraser: Unleashing diffusion model’s object removal potential via self-attention redirection guidance,

    W. Sun, B. Cui, J. Tang, and X.-M. Dong, “Attentive eraser: Unleashing diffusion model’s object removal potential via self-attention redirection guidance,” arXiv preprint arXiv:2412.12974 , 2024

    2024

  48. [56]

    Rethinking the inception architecture for computer vision,

    C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in CVPR, 2016, pp. 2818–2826

    2016

  49. [57]

    Predicated diffusion: Predicate logic- based attention guidance for text-to-image diffusion models,

    K. Sueyoshi and T. Matsubara, “Predicated diffusion: Predicate logic- based attention guidance for text-to-image diffusion models,” in Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 8651–8660

    2024

  50. [58]

    Prompt-to-prompt image editing with cross-attention control,

    A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y . Pritch, and D. Cohen-or, “Prompt-to-prompt image editing with cross-attention control,” in ICLR, 2023

    2023

  51. [59]

    Style aligned image generation via shared attention,

    A. Hertz, A. V oynov, S. Fruchter, and D. Cohen-Or, “Style aligned image generation via shared attention,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 4775–4785

    2024

  52. [60]

    Fontstudio: shape-adaptive diffusion model for coherent and consistent font effect generation,

    X. Mu, L. Chen, B. Chen, S. Gu, J. Bao, D. Chen, J. Li, and Y . Yuan, “Fontstudio: shape-adaptive diffusion model for coherent and consistent font effect generation,” in European Conference on Computer Vision . Springer, 2024, pp. 305–322

    2024

  53. [61]

    Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer,

    J. Chung, S. Hyun, and J.-P. Heo, “Style injection in diffusion: A training-free approach for adapting large-scale diffusion models for style transfer,” in CVPR, 2024, pp. 8795–8805

    2024

  54. [62]

    ediff-i: Text-to-image diffu- sion models with an ensemble of expert denoisers,

    Y . Balaji, S. Nah, X. Huang, A. Vahdat, J. Song, Q. Zhang, K. Kreis, M. Aittala, T. Aila, S. Laine et al. , “ediff-i: Text-to-image diffu- sion models with an ensemble of expert denoisers,” arXiv preprint arXiv:2211.01324, 2022

    2022 arXiv

  55. [63]

    Prospect: Prompt spectrum for attribute-aware personalization of diffusion models,

    Y . Zhang, W. Dong, F. Tang, N. Huang, H. Huang, C. Ma, T.-Y . Lee, O. Deussen, and C. Xu, “Prospect: Prompt spectrum for attribute-aware personalization of diffusion models,” ACM TOG , vol. 42, no. 6, pp. 1–14, 2023

    2023

  56. [64]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in ECCV. Springer, 2014, pp. 740–755

    2014

  57. [65]

    The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale,

    A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont- Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov et al. , “The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale,” IJCV, vol. 128, no. 7, p...

    1956

  58. [66]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” inCVPR, 2009, pp. 248–255

    2009

  59. [67]

    Improved artgan for conditional synthesis of natural image and artwork,

    W. R. Tan, C. S. Chan, H. E. Aguirre, and K. Tanaka, “Improved artgan for conditional synthesis of natural image and artwork,” IEEE TIP, vol. 28, no. 1, pp. 394–409, 2018

    2018

  60. [68]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium,

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” NeurIPS, vol. 30, 2017

    2017

  61. [69]

    Demysti- fying mmd gans,

    M. Bi ´nkowski, D. J. Sutherland, M. Arbel, and A. Gretton, “Demysti- fying mmd gans,” in ICLR, 2018

    2018

  62. [70]

    Rethinking fid: Towards a better evaluation metric for image generation,

    S. Jayasumana, S. Ramalingam, A. Veit, D. Glasner, A. Chakrabarti, and S. Kumar, “Rethinking fid: Towards a better evaluation metric for image generation,” in CVPR, 2024, pp. 9307–9315

    2024

  63. [71]

    Clipscore: A reference-free evaluation metric for image captioning,

    J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y . Choi, “Clipscore: A reference-free evaluation metric for image captioning,” in EMNLP, 2021, pp. 7514–7528

    2021

  64. [72]

    Imagereward: Learning and evaluating human preferences for text-to- image generation,

    J. Xu, X. Liu, Y . Wu, Y . Tong, Q. Li, M. Ding, J. Tang, and Y . Dong, “Imagereward: Learning and evaluating human preferences for text-to- image generation,” NeurIPS, vol. 36, 2024

    2024

  65. [73]

    Flux,” https://github.com/black-forest-labs/flux, 2024. 11 HarmonPaint w/o SAMS TimeStep=1000 TimeStep=1

    B. F. Labs, “Flux,” https://github.com/black-forest-labs/flux, 2024. 11 HarmonPaint w/o SAMS TimeStep=1000 TimeStep=1 "cat" Input Result Self-Attention Map in the U-Net Encoder HarmonPaint w/o SAMS "rabbit" Fig. 12: Additional PCA visualizations of self-attention maps show tha...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.