Pith. sign in

REVIEW 3 major objections 5 minor 45 references

MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MTADiffusion claims that combining automatically annotated mask-text pairs, a joint edge-prediction task, and a VGG Gram-matrix style loss yields state-of-the-art object inpainting on BrushBench and EditBench.

desk verdict The dataset and annotation pipeline are the real contribution; the style-consistency loss is under-validated and the SOTA claim is thinner than the abstract suggests, but the paper deserves a serious referee. read the letter →

arxiv 2506.23482 v1 pith:MHPAKPTQ submitted 2025-06-30 cs.CV

Jun Huang , Ting Liu , Yihang Wu , Xiaochao Qu , Luoqi Liu , Xiaolin Hu This is my paper
classification cs.CV
keywords objectinpaintingtext-guideddiffusionmask-textalignmentedgepredictionstyleconsistencyGrammatrixVGGfeaturelossdatasetannotationpipeline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that object inpainting can be made simultaneously more semantically accurate, structurally stable, and stylistically consistent by combining three ingredients: a large automatically annotated dataset of masks with detailed text descriptions, a multi-task training objective that also predicts edge maps, and a Gram-matrix style loss computed on VGG features of intermediate diffusion latents. These ingredients are assembled in MTADiffusion, a dual-branch diffusion model built on Stable Diffusion and the BrushNet architecture. Evaluated on BrushBench and EditBench, the method reports state-of-the-art results on image quality, masked-region preservation, and text-alignment metrics, plus a user study favoring its outputs.

What carries the argument

The architecture is a UNet branch (Stable Diffusion) plus a brush branch whose multi-resolution self-attention blocks take the noisy latent, masked image latent, and downsampled mask, feeding into each UNet layer through zero convolutions. The brush branch's final layer predicts a Sobel-edge map, adding the structural loss $L_{\text{structure}}$. For style, predicted and ground-truth latents at timestep $t{-}1$, obtained via the denoise and noise-adding functions, are passed through a VGG network whose first layer is expanded from 3 to 4 input channels, and $L_{\text{style}}$ compares Gram matrices of VGG features, weighted by $\delta = 100$.

What would settle it

Train the same model with $L_{\text{style}}$ removed or replaced by a style loss computed on decoded clean images, and check whether the user-study style-consistency votes and the reported gains collapse; if they do not, the latent Gram-loss mechanism is not the carrier of the claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, MTADiffusion establishes that the missing ingredient in text-guided object inpainting is tight alignment between the mask and a detailed description of the object inside it, rather than a caption of the whole image. The authors build MTAPipeline to produce such pairs at scale: Grounded-SAM extracts masks and labels, LLaVA writes content-and-style descriptions, and the resulting 5-million-image, 25-million-pair MTADataset is used to train a dual-branch diffusion model alongside an edge-prediction head and a VGG-based latent style loss. The combined objective $L = \gamma L_{\text{noise}} + \delta L_{\text{style}} + \eta L_{\text{structure}}$ yields gains on both object-mask and random-mask benchmarks.

Load-bearing premise

The style-consistency loss is trusted to work on 4-channel noisy diffusion latents through a VGG network pretrained on clean 3-channel RGB images, and that trust carries a weight of 100 in the final loss.

Editorial extensions

If this is right

  • Automatic mask-text annotation can substitute for human-labeled inpainting data, making large-scale training practical.
  • Auxiliary edge prediction strengthens structure without requiring edge maps at inference time.
  • Latent-space style loss with VGG Gram matrices improves style consistency without extra decoding steps.
  • The training strategies are model-agnostic and can be lifted into other inpainting or editing models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same MTAPipeline could be applied to other region-level tasks such as image harmonization or object replacement, where local text-style grounding matters.
  • Because caption quality is named as the bottleneck, upgrading LLaVA to a reasoning-stronger vision-language model may improve color and attribute fidelity more than further scaling the data.
  • A risk the paper does not analyze is that the 4-channel VGG input adaptation may distort early-layer features; a calibration study comparing channel-expansion strategies would test this directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes MTADiffusion, a text-guided object inpainting model with three main contributions: (i) an automatic annotation pipeline (MTAPipeline) and a large dataset (MTADataset, 5M images and 25M mask-text pairs), (ii) a multi-task training strategy that jointly predicts noise and Sobel edge maps, and (iii) a style-consistency loss computed with a VGG network and Gram matrices on denoised latent maps. The model is evaluated on BrushBench and EditBench against four baselines (SDI, PowerPaint, CNI, BrushNet), and the authors claim state-of-the-art performance in semantic alignment, structural stability, and style consistency.

Significance. If fully validated, the paper would provide a valuable large-scale mask-text annotation resource for inpainting and a set of training objectives that could generalize beyond the specific architecture. The dataset scale (5M images, 25M mask-text pairs) is a substantial contribution, and the idea of jointly training edge prediction with inpainting is plausible. However, the central SOTA claim is currently supported only by small or inconsistent metric differences, and the style-consistency loss is not adequately validated. The ablation study for the losses is deferred to a supplementary that is not included in the submitted text, which makes the individual contributions of the proposed components impossible to assess.

major comments (3)
  1. [Section 3.3.2/3.4/4.1, Eq. (5)-(8)] The style-consistency loss is a core contribution, but its validity is not established. The first VGG convolutional layer is expanded from 3 to 4 channels (Section 4.1) and applied to noisy latent maps X_{t-1} and X~_{t-1} at intermediate denoising steps, yet no evidence is given that ImageNet-pretrained VGG features remain meaningful for VAE latents at those noise levels. Moreover, the manuscript is internally inconsistent about whether the VGG is frozen: Section 3.3.2 calls it a 'pre-trained VGG network,' while Section 3.4 says the 'VGG latent style extractor' is trained and Section 4.1 gives it a learning rate of 1e-7. If the extractor is trainable, minimizing the Gram loss in Eq. (7) can be trivially achieved by feature collapse, not by improving style consistency. With delta=100 in Eq. (8), this unvalidated objective dominates the total loss and could actively distort denoising. No main-text ablation isolates the effect of L_style; the pointer in Section 5 to supplementary material for loss ablations is unfulfilled in the submitted text. This issue is load-bearing because the style-consistency loss is one of the three named contributions in the Abstract and is credited for the style gains in the user study.
  2. [Tables 1 and 2] The state-of-the-art claim is not supported by the quantitative tables. On BrushBench, the gains over BrushNet are extremely small: PSNR 31.87 vs 31.82, LPIPS 18.94 vs 18.95, and MSE 0.80 vs 0.82. On EditBench, the proposed method is worse than BrushNet on PSNR (33.31 vs 33.66) and MSE (0.65 vs 0.63), and essentially tied on LPIPS. The claimed superiority rests entirely on IR, CLIP, and VQA scores. The paper reports no error bars, no multiple-seed results, and no significance tests. Given the small and sometimes contradictory margins, the abstract's 'state-of-the-art performance' claim is overstated and should either be supported with statistical evidence or appropriately qualified.
  3. [Section 3.2 and Section 5] The multi-task edge prediction is a central mechanism for the claimed structural stability improvement, but no ablation demonstrates its contribution. The only ablation pointer in Section 5 ('See also supplementary material for the ablation studies of different losses') is not backed by content in the submitted manuscript. Since the user study attributes structural stability to the full system, the individual effects of L_structure and of the proposed dataset cannot be disentangled. The authors should provide main-text ablations that remove or reweight each loss component, and ideally also isolate the dataset contribution, to substantiate the structural-stability claim.
minor comments (5)
  1. [Eq. (7)] The summation index i is reused for both the batch and the style-feature count; the second sum should use a different index (e.g., j) for the N style features.
  2. [Section 4.1] The phrase 'the outputs of convi 1, where the i = (1, . . . ,5) layers' is unclear; it should read 'the outputs of the conv1_i layers for i = 1, ..., 5'.
  3. [Tables 1 and 2] Table 1 reports a VQA Score but Table 2 does not; please either add the same metric to both tables or explain the omission.
  4. [Section 4.6] The user study reports only aggregate vote percentages; providing per-image vote distributions or confidence intervals would strengthen the claim that the differences are meaningful.
  5. [Section 3.1] The statement that each image contains 'approximately 5 masks' and the dataset contains '25 million mask-text pairs' is consistent only if exactly 5 masks per image is the average; please clarify that this is an average, not an exact count.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: external benchmarks, training objectives, and disclosed hyperparameters; no derivation reduces to its inputs.

full rationale

The paper is an empirical systems contribution whose central claims are evaluated on external benchmarks (BrushBench and EditBench), not on quantities fitted from or defined by the method itself. The MTADataset is constructed by an automatic pipeline using Grounded-SAM and LLaVA, and its value is tested by training BrushNet on BrushData versus MTADataset and measuring the difference on BrushBench (Table 3); this is a controlled external comparison, not a circular prediction. The multi-task edge loss (Eq. 2) is a training objective computed from a Sobel ground truth, and the style loss (Eq. 7) is a Gram-matrix objective between a predicted denoised latent and a ground-truth noised latent; both are optimization losses, not claims that reproduce their own inputs at evaluation time. Hyperparameters gamma=1, delta=100, eta=0.1 are disclosed and hand-set rather than fitted to the test benchmarks. There are no load-bearing self-citations: references to BrushNet and other methods are independent prior work used for comparison or architectural inspiration, and no uniqueness or performance claim rests on the authors' own prior publications. The acknowledged limitation that LLaVA captions can be inaccurate, and the note that ablation studies of different losses are in the supplementary material, are completeness or robustness concerns rather than circularity: the absence of a main-text ablation does not make the style loss's contribution equivalent to its input by construction. Overall, no step in the paper's claimed derivation chain reduces to its own inputs, so the appropriate circularity score is 0.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

The central contribution is empirical; the free parameters are training hyperparameters and filtering thresholds. The key assumptions are that automatic annotation and VGG-on-latent style features behave as intended, both of which are either acknowledged limitations or unvalidated. No new physical entities are introduced.

free parameters (9)
  • Loss weight gamma = 1
    Weights the noise loss in the final objective; set by hand (Section 3.4, Section 4.1).
  • Loss weight delta = 100
    Weights the style-consistency loss; set by hand (Section 3.4, Section 4.1).
  • Loss weight eta = 0.1
    Weights the structure/edge loss; set by hand (Section 3.4, Section 4.1).
  • Learning rate for attention branch = 1e-5
    Set in Section 4.1.
  • Learning rate for VGG style extractor = 1e-7
    Set in Section 4.1.
  • Aesthetic score threshold for dataset filtering = > 5.8
    LAION images selected with aesthetic score greater than 5.8 (Section 3.1).
  • Grounded-SAM confidence threshold = 0.6
    Masks with recognition confidence greater than 0.6 selected (Section 3.1).
  • VGG style layers = conv1_1 to conv5_1
    Outputs of conv1_1 through conv5_1 considered as style features (Section 4.1).
  • Training iterations = 200,000
    Total training iterations on 8 V100 GPUs (Section 4.1).
assumptions (5)
  • domain assumption Pretrained Stable Diffusion UNet and VAE are suitable frozen backbones for inpainting
    Section 3.4 freezes the VAE and UNet, assuming their features support the inpainting task.
  • domain assumption Grounded-SAM and LLaVA produce accurate enough masks and descriptions
    Section 3.1 relies on these models for the dataset; the conclusion explicitly notes LLaVA descriptions can be inaccurate, e.g., object color.
  • domain assumption Sobel edge map of the full input image is a valid structural target
    Section 3.2 Eq. 1-2 assumes that predicting the original object's edges in the masked region improves structural stability.
  • ad hoc to paper VGG features on a 4-channel latent represent perceptual style
    Section 3.3.2 and Section 4.1 expand the VGG input from 3 to 4 channels without evidence that pretrained weights transfer to the latent space.
  • domain assumption ImageReward, Aesthetic Score, CLIP, and VQA are valid proxies for style, structure, and semantic alignment
    Section 4.3 uses these metrics to support the state-of-the-art claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting." pith.science (2026). https://pith.science/paper/MHPAKPTQ

@misc{pith2026250623482,
  author       = {Pith},
  title        = {Pith review of: MTADiffusion: Mask Text Alignment Diffusion Model for Object Inpainting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MHPAKPTQ}},
  note         = {Machine review of arXiv:2506.23482}
}
read the original abstract

Advancements in generative models have enabled image inpainting models to generate content within specific regions of an image based on provided prompts and masks. However, existing inpainting methods often suffer from problems such as semantic misalignment, structural distortion, and style inconsistency. In this work, we present MTADiffusion, a Mask-Text Alignment diffusion model designed for object inpainting. To enhance the semantic capabilities of the inpainting model, we introduce MTAPipeline, an automatic solution for annotating masks with detailed descriptions. Based on the MTAPipeline, we construct a new MTADataset comprising 5 million images and 25 million mask-text pairs. Furthermore, we propose a multi-task training strategy that integrates both inpainting and edge prediction tasks to improve structural stability. To promote style consistency, we present a novel inpainting style-consistency loss using a pre-trained VGG network and the Gram matrix. Comprehensive evaluations on BrushBench and EditBench demonstrate that MTADiffusion achieves state-of-the-art performance compared to other methods.

Figures

Figures reproduced from arXiv: 2506.23482 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The framework of MTADiffusion. Our approach begins with constructing the MTAPipeline, which generates training samples for the MTADataset in the form of mask and text description pairs. The architecture of our network adopts a dual-branch strategy by applying the outputs of the upper branch to each layer of the UNet branch. To enhance the structural stability of the model, we jointly train the inpainting tasks and e… view at source ↗
Figure 3
Figure 3. MTAPipeline for annotating MTADataset. We first employ Grounded-SAM [24] to extract labels, bounding boxes, and masks. Subsequently, we utilize LLaVA [15] to annotate mask region with detailed content and style information. style loss to balance the trade-off between style migration and content preservation. Inspired by these works, we use the VGG network to extract style features and then compute style loss with th… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Results on BrushBench compared to other methods. With our method, the inpainting region achieves effective improvements in semantic alignment, structural stability, and style consistency [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Results on EditBench compared to other methods. Our model trained on MTADataset demonstrates stronger semantic capabilities under random masking conditions. Bench are shown in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 15 canonical work pages

  1. [1]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Dosovitskiy Alexey. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv: 2010.11929, 2020. 2, 3

    arXiv 2010

  2. [2]

    Blended latent diffusion

    Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM transactions on graphics (TOG) , 42 (4):1–11, 2023. 3

    work page 2023

  3. [3]

    Improving text-guided object inpainting with semantic pre-inpainting

    Yifu Chen, Jingwen Chen, Yingwei Pan, Yehao Li, Ting Yao, Zhineng Chen, and Tao Mei. Improving text-guided object inpainting with semantic pre-inpainting. In European Con- ference on Computer Vision, pages 110–126. Springer, 2025. 2, 3

    work page 2025

  4. [4]

    Diffedit: Diffusion-based seman- tic image editing with mask guidance

    Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based seman- tic image editing with mask guidance. arXiv preprint arXiv:2210.11427, 2022. 2

    arXiv 2022

  5. [5]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learn- ing, 2024. 2

    2024

  6. [6]

    Prompt-to-prompt im- age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 2

    arXiv 2022

  7. [7]

    Denoising dif- fusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. 2

    2020

  8. [8]

    Arbitrary style transfer in real-time with adaptive instance normalization

    Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceed- ings of the IEEE international conference on computer vi- sion, pages 1501–1510, 2017. 3

    2017

Show all 45 references
  1. [9]

    Brushnet: A plug-and-play image inpaint- ing model with decomposed dual-branch diffusion

    Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. Brushnet: A plug-and-play image inpaint- ing model with decomposed dual-branch diffusion. arXiv preprint arXiv:2403.06976, 2024. 2, 3, 4, 5, 6, 7

    2024 arXiv

  2. [10]

    Auto-encoding variational bayes

    Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 2, 5

    2013 arXiv

  3. [11]

    Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ´ar, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023. 2

    2023 arXiv

  4. [12]

    Openimages: A public dataset for large-scale multi-label and multi-class image classification

    Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai...

  5. [13]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 7

    2023

  6. [14]

    Evaluating text-to-visual generation with image-to-text gen- eration

    Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text gen- eration. In European Conference on Computer Vision, pages 366–384. Springer, 2024. 6

    2024

  7. [15]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 4, 7

    2024

  8. [16]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 2

    2023 arXiv

  9. [17]

    Painterly image harmonization using diffusion model

    Lingxiao Lu, Jiangtong Li, Junyan Cao, Li Niu, and Liqing Zhang. Painterly image harmonization using diffusion model. In Proceedings of the 31st ACM International Con- ference on Multimedia, pages 233–241, 2023. 2, 3

    2023

  10. [18]

    Repaint: Inpainting using denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022. 3

    2022

  11. [19]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  12. [20]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 2

    2023 arXiv

  13. [21]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  14. [22]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 2

    2020

  15. [23]

    Hierarchical text-conditional image gener- ation with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gener- ation with clip latents. arXiv preprint arXiv:2204.06125, 1 (2):3, 2022. 2

    2022 arXiv

  16. [24]

    Grounded sam: Assembling open-world models for diverse visual tasks,

    Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kun- chang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks,

  17. [25]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 2, 3, 6

    2022

  18. [26]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2250...

    2023

  19. [27]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  20. [28]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural In- f...

    2022

  21. [29]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 2, 3, 5

    2014 arXiv

  22. [30]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International confer- ence on machine learning, pages 2256–2265. PMLR, 2015. 2

    2015

  23. [31]

    Measuring style similarity in diffusion models

    Gowthami Somepalli, Anubhav Gupta, Kamal Gupta, Shra- may Palta, Micah Goldblum, Jonas Geiping, Abhinav Shri- vastava, and Tom Goldstein. Measuring style similarity in diffusion models. arXiv preprint arXiv:2404.01292 , 2024. 2, 3

    2024 arXiv

  24. [32]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 2

    2010 arXiv

  25. [33]

    Generative modeling by esti- mating gradients of the data distribution

    Yang Song and Stefano Ermon. Generative modeling by esti- mating gradients of the data distribution. Advances in neural information processing systems, 32, 2019. 2

    2019

  26. [34]

    Imagen editor and editbench: Advancing and evaluating text-guided im- age inpainting

    Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont- Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided im- age inpainting. In Proceedings of the IEEE/CVF c...

    2023

  27. [35]

    Stylediffusion: Controllable disentangled style transfer via diffusion models

    Zhizhong Wang, Lei Zhao, and Wei Xing. Stylediffusion: Controllable disentangled style transfer via diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7677–7689, 2023. 3

    2023

  28. [36]

    Godiva: Gen- erating open-domain videos from natural descriptions

    Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Gen- erating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806, 2021. 6

    2021 arXiv

  29. [37]

    Not only generative art: Stable diffusion for content-style disentangle- ment in art analysis

    Yankun Wu, Yuta Nakashima, and Noa Garcia. Not only generative art: Stable diffusion for content-style disentangle- ment in art analysis. In Proceedings of the 2023 ACM Inter- national conference on multimedia retrieval, pages 199–208,

    2023

  30. [38]

    Smartbrush: Text and shape guided object inpainting with diffusion model

    Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22428–22437, 2023. 1, 2, 3

    2023

  31. [39]

    Imagere- ward: Learning and evaluating human preferences for text- to-image generation

    Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagere- ward: Learning and evaluating human preferences for text- to-image generation. Advances in Neural Information Pro- cessing Systems, 36, 2024. 6

    2024

  32. [40]

    Uni-paint: A unified framework for multimodal image inpainting with pretrained diffusion model

    Shiyuan Yang, Xiaodong Chen, and Jing Liao. Uni-paint: A unified framework for multimodal image inpainting with pretrained diffusion model. In Proceedings of the 31st ACM International Conference on Multimedia, pages 3190–3199,

  33. [41]

    Zero-shot contrastive loss for text-guided diffusion image style transfer

    Serin Yang, Hyunmin Hwang, and Jong Chul Ye. Zero-shot contrastive loss for text-guided diffusion image style transfer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22873–22882, 2023. 2, 3

    2023

  34. [42]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2, 3, 6

    2023

  35. [43]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shecht- man, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 586–595, 2018. 6

    2018

  36. [44]

    Inversion-based style transfer with diffusion models

    Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu. Inversion-based style transfer with diffusion models. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10146–10156, 2023. 3

    2023

  37. [45]

    A task is worth one word: Learning with task prompts for high-quality versatile image inpainting

    Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting. arXiv preprint arXiv:2312.03594, 2023. 1, 2, 3, 6

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.