Pith. sign in

REVIEW 4 major objections 5 minor 55 references

Do We Need to Design Specific Diffusion Models for Different Tasks? Try ONE-PIC

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ONE-PIC turns every image-generation task into a masked inpainting problem and matches task-specific models with only 0.618% learnable parameters.

desk verdict Simple, plausible, and clearly written, but the headline claims are not yet supported by quantitative evidence; worth a serious referee after major revisions. read the letter →

arxiv 2412.05619 v1 pith:BRVFBFFI submitted 2024-12-07 cs.CV

Ming Tao , Bing-Kun Bao , Yaowei Wang , Changsheng Xu This is my paper · ORCID
classification cs.CV
keywords ONE-PICdiffusionmodelsparameter-efficientfine-tuningLoRAmaskedinpaintingvisualcontextvirtualtry-onimageediting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ONE-PIC claims that a single, parameter-light fine-tuning recipe can replace task-specific diffusion-model designs across very different image-generation jobs. Instead of adding branch networks, task-specific losses, or per-task training strategies, it arranges the reference or condition images and the target image into one composite visual context, masks the target region, and fine-tunes only the self-attention layers of a pretrained SDXL with LoRA to predict the noise in that masked region. On visual conditional control, DreamBooth-style subject generation, instruction-based image editing, and virtual try-on, the paper reports performance comparable to task-specific methods while using about 0.618% learnable parameters and a small fraction of the training time and data, including roughly 2% of the resources of StableVTON for virtual try-on. A sympathetic reader should care because the claim is that the bottleneck in adapting diffusion models is not new architecture but how the task is packaged as an inpainting problem.

What carries the argument

In-Visual-Context Tuning is the central object: it arranges source and target images into a single image so that downstream fine-tuning resembles the model's pretraining objective, letting the network read both fine-grained spatial detail and high-level semantics from the grid. The Masking Strategy is the second load-bearing piece: a masked noising process (Eq. 1-2) adds noise only to target-region latents and evaluates the DDPM loss only on that masked region (Eq. 3), turning every task into the prediction of masked parts. The third piece is LoRA applied to the U-Net self-attention layers (attn1.q/k/v, rank 32, alpha 4), which keeps learnable parameters at 0.618%. The paper also documents layout design choices, such as 2x2 grids for precise positional control, center placement for tasks with several conditions, and 1x2 or 2x1 grids for faster inference, as evidence that visual context plays a role analogous to prompts in language models.

What would settle it

Run the released ONE-PIC code on VITON-HD with the standard train/test split and compute LPIPS and FID against StableVTON; if the numbers are not within a small margin of the task-specific baseline, the 'comparable performance with 2% resources' claim is refuted. A second check: fine-tune ONE-PIC on pose-to-image control with the 2x2 context and compare to ControlNet's reported results; if it cannot match on standard metrics, the claim that one inpainting-style recipe replaces task-specific branches fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that large pretrained text-to-image diffusion models already contain the machinery needed for many downstream generation tasks; what was missing is a way to present those tasks in the model's native format. ONE-PIC presents each task as masked inpainting: the source and condition images are stitched together with the target image into one image, noise is added only to the latent of the target region, and the model is trained with the standard denoising loss restricted to the masked part. Because only the LoRA weights on the self-attention query, key, and value projections are updated, the pretrained knowledge stays almost intact, and the paper argues this is why adaptation is fast and needs little data. The same procedure, with different visual-context layouts per task, is shown to handle spatial control, subject-driven generation, instruction-guided editing, and virtual try-on.

Load-bearing premise

The claim rests on the assumption that a pretrained SDXL U-Net, with only its self-attention LoRA weights updated, can read a stitched grid of source and target images well enough that a single masked-inpainting objective is sufficient for tasks as different as pose control and virtual try-on; if that cross-task visual-context reading fails, the whole recipe collapses.

Editorial extensions

If this is right

  • A single base SDXL plus a small LoRA delta can be reused across tasks, since the network architecture is identical for all downstream jobs and only the input layout and mask change.
  • Adaptation costs fall from days and thousands of steps to hours and hundreds of steps, making fine-tuning accessible to users without large compute budgets.
  • New downstream tasks can be added by deciding how to stitch the images and where to place the mask, without designing task-specific losses or side networks.
  • Because only self-attention LoRA weights are stored per task, model sharing and deployment become simpler than keeping separate full models.
  • The unified masked-inpainting formulation extends to additional tasks such as colorization, fashion extraction, image segmentation, and identity-preserved portrait generation, as the paper demonstrates qualitatively.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim holds, the research question for a new image-generation task shifts from 'what network do we add?' to 'how do we lay out the visual context and mask?', a design choice that can be explored systematically rather than by inventing new modules.
  • The analogy to prompt learning suggests a stronger possibility the paper leaves implicit: some tasks may need no fine-tuning at all if their layout and mask are chosen well, extending the zero-shot inpainting results of RePaint to broader task families.
  • The method's dependence on the pretrained model's spatial priors, such as its center bias and the preference for certain grid shapes, implies that tasks with unusual geometry, like non-rectangular targets or multiple disjoint regions, are the natural stress test; this is an inference, not a claim in the paper.
  • Combining ONE-PIC's context layout with a task-specific loss or post-processing could close any remaining gap to specialized models, since the paper's evidence is 'comparable' rather than strictly better; that is an editorial suggestion.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ONE-PIC, a unified fine-tuning framework for large pretrained diffusion models. The method keeps the SDXL architecture unchanged, constructs a task-specific visual context by stitching reference and target images into a single grid, applies a masking strategy that adds noise only to the target region, and fine-tunes only LoRA adapters on the self-attention layers. The training objective is the standard masked diffusion noise-prediction loss of Eq. (3). The authors claim that this single recipe achieves performance comparable to task-specific methods across visual conditional control, DreamBooth, image editing, virtual try-on, and several additional tasks, while using only 0.618% learnable parameters and, for virtual try-on, about 2% of the resources required by StableVTON. The experimental sections present qualitative results (Figures 6, 8, 9, 10, 11), a convergence comparison with ControlNet (Figure 7), and per-task training times and data subsets.

Significance. If the central claims were quantitatively established, the contribution would be significant: a single, parameter-efficient fine-tuning recipe that leverages pretrained diffusion knowledge across multiple image-generation tasks would simplify adaptation, reduce training cost, and lower the barrier for downstream users. The method is simple and clearly described, the loss in Eq. (3) is a standard and sound masked diffusion objective, the architecture introduces no invented modules, and the code is publicly released. However, the significance is currently conditional: the claims of 'comparable performance' and '2% resources' rest on curated qualitative figures and unspecified resource accounting, with no quantitative evaluation against baselines. The paper also relies on task-specific visual-context layouts, which partially qualifies the 'one framework for all tasks' narrative. The potential is real, but the evidence presented is not yet sufficient for a journal-level claim.

major comments (4)
  1. [Sec. 4.1–4.4, Figures 6, 8, 9, 10] The central claim of performance comparable to task-specific fine-tuning methods is not backed by any quantitative metric. Neither FID, LPIPS, SSIM, CLIP score, DINO similarity, nor user studies are reported for any of the four main tasks. For DreamBooth, prior work evaluates on DreamBench with CLIP and DINO metrics, but Sec. 4.2 reports only qualitative examples; for virtual try-on, Sec. 4.4 reports no LPIPS/SSIM/FID against VITON-HD or StableVTON baselines; for image editing, Sec. 4.3 reports no comparison on MagicBrush. Without matched quantitative comparisons, the strongest claim in the abstract and introduction is unfalsifiable as presented and needs to be added before the paper can be evaluated.
  2. [Abstract and Sec. 1] The claim that ONE-PIC uses '2% of the resources required by StableVTON' is not defined or computed. The manuscript does not specify whether the resource is GPU-hours, memory, number of parameters, training steps, or a composite measure, nor does it give the arithmetic behind the ratio. Likewise, the 0.618% learnable-parameter figure in Sec. 4 is stated without specifying the denominator or comparing with the parameter counts of ControlNet, StableVTON, or other baselines. These resource-efficiency claims are load-bearing for the paper's contribution and must be made precise and reproducible.
  3. [Sec. 4.1, Figure 7] The convergence comparison with ControlNet is not a controlled experiment. The authors state that they 'utilized the convergence process images from the ControlNet paper for this comparison,' but no common evaluation metric, dataset, or hardware setting is used, and the x-axes of the two panels are not defined. The claim that ONE-PIC 'had already grasped the fine-tuning task' after 400 steps is based on qualitative inspection of generated images, not on any measured generation-quality metric. A proper comparison should report, at multiple training steps, the same quantitative metric for ONE-PIC and for a baseline trained on the same data and compute budget.
  4. [Sec. 3.2, Figure 5, Sec. 4.6] The method requires a hand-selected visual-context layout for each task (2×2 for conditional control, 1×3 for virtual try-on, 3-position layouts for DreamBooth and editing), and Sec. 4.6 provides heuristics rather than a fixed rule. This is a task-specific design choice that partially qualifies the claim of a single unified fine-tuning framework. The paper should either provide a principled selection criterion or include an ablation quantifying how much the layout choice affects performance; without this, the generality claim is weaker than stated and the per-task layouts act as additional free parameters.
minor comments (5)
  1. [Eq. (1)–(2)] The notation '˜xt' is introduced without definition, and the relationship between the noised latent in Eq. (1) and the composed latent in Eq. (2) should be stated explicitly. The text description symbol 'e' should be defined consistently with standard notation (e.g., 'c' or 'text').
  2. [Sec. 4.1] The dataset name appears as 'LIAON-Art', which is likely a typo for 'LAION-Art'. Please correct the spelling and provide the exact dataset version or URL.
  3. [Sec. 4.3 and Fig. 9] The description of the 'mask-given image editing scenario' is unclear. The text says the top-left images are replaced with images containing the mask information, but it does not explain how the mask is encoded (e.g., as a binary overlay, a colored region, or a separate channel) or where the target mask is applied. Please clarify.
  4. [Throughout] The paper alternates between 'DreamBooth' and 'Dreambooth', and between 'StableVTON' and 'StableViton'. Please unify the capitalization and check all references for consistency.
  5. [Sec. 4.4] The statement that the 1×3 layout 'performed better' than the 2×2 layout for virtual try-on is not accompanied by any quantitative comparison or ablation figure. Even a small table or a supplementary figure would help the reader judge the claimed difference.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: ONE-PIC's masked-noise objective is a standard diffusion loss; the self-citation to StoryImager is motivating only, and unsupported efficiency claims are an evidence gap, not circularity.

full rationale

The paper's derivation chain is self-contained with respect to circularity concerns. The objective in Eq. (3) is the standard DDPM/LDM noise-prediction loss restricted to a binary mask; it is not derived from the downstream-task outputs, and no task metric is fitted and then relabeled as a prediction. ONE-PIC's mechanism, concatenating source and target latents into one grid and noising only the target region, is explicitly framed as a special case of masked inpainting, with external support cited (RePaint [21]) rather than a self-citation. The only self-citation is StoryImager [44], used as motivation for visual-context stitching; it is not load-bearing for the four-task results or for Eq. (3). The per-task layouts in Sec. 4.6 are empirical design choices selected from comparisons, not parameters fitted to the evaluation data, so they do not create a circular prediction. The headline claims of 'comparable performance' and '2% of StableVTON's resources' are asserted without quantitative metrics or a defined resource measure; that is a correctness or evidence deficiency, not a definitional circularity, so it does not raise the circularity score. Overall, no step in the derivation reduces by construction to its own input.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on empirical premises about pretrained diffusion models and visual context, plus hand-selected per-task layout choices. No new physical entities are introduced.

free parameters (2)
  • Visual context layout (grid arrangement) per task = 2x2 for conditional control; 1x2/2x1 for editing; 1x3 for try-on; three reference images top-left for DreamBooth
    Chosen empirically per task in Sec. 4.6; the method is not fully general because each task requires a hand-selected layout.
  • LoRA rank and alpha = r=32, alpha=4
    Hyperparameters for the LoRA fine-tuning, selected as standard values; they control capacity but are not fitted to the evaluation data.
assumptions (3)
  • domain assumption The pretrained SDXL model carries enough generative prior and inpainting-like ability to be transferred to all tested tasks via visual context and masked prediction.
    Invoked in Sec. 3.1 and 3.3; the method freezes most of the U-Net and relies on this prior.
  • ad hoc to paper Arranging source and target images into one grid is sufficient for self-attention layers to use the visual context as a task condition without extra modules.
    Core premise of In-Visual-Context Tuning in Sec. 3.2; asserted and shown qualitatively, not proven or ablated.
  • domain assumption Computing the diffusion loss only on the masked target region (Eq. 3) is sufficient to learn each downstream task mapping.
    Standard inpainting-style assumption used in Sec. 3.3; no ablation compares against full-latent loss.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do We Need to Design Specific Diffusion Models for Different Tasks? Try ONE-PIC." pith.science (2026). https://pith.science/paper/BRVFBFFI

@misc{pith2026241205619,
  author       = {Pith},
  title        = {Pith review of: Do We Need to Design Specific Diffusion Models for Different Tasks? Try ONE-PIC},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BRVFBFFI}},
  note         = {Machine review of arXiv:2412.05619}
}
read the original abstract

Large pretrained diffusion models have demonstrated impressive generation capabilities and have been adapted to various downstream tasks. However, unlike Large Language Models (LLMs) that can learn multiple tasks in a single model based on instructed data, diffusion models always require additional branches, task-specific training strategies, and losses for effective adaptation to different downstream tasks. This task-specific fine-tuning approach brings two drawbacks. 1) The task-specific additional networks create gaps between pretraining and fine-tuning which hinders the transfer of pretrained knowledge. 2) It necessitates careful additional network design, raising the barrier to learning and implementation, and making it less user-friendly. Thus, a question arises: Can we achieve a simple, efficient, and general approach to fine-tune diffusion models? To this end, we propose ONE-PIC. It enhances the inherited generative ability in the pretrained diffusion models without introducing additional modules. Specifically, we propose In-Visual-Context Tuning, which constructs task-specific training data by arranging source images and target images into a single image. This approach makes downstream fine-tuning closer to the pertaining, allowing our model to adapt more quickly to various downstream tasks. Moreover, we propose a Masking Strategy to unify different generative tasks. This strategy transforms various downstream fine-tuning tasks into predictions of the masked portions. The extensive experimental results demonstrate that our method is simple and efficient which streamlines the adaptation process and achieves excellent performance with lower costs. Code is available at https://github.com/tobran/ONE-PIC.

Figures

Figures reproduced from arXiv: 2412.05619 by the authors.

Figure 1
Figure 1. Our ONE-PIC provides a simple and versatile framework for fine-tuning across various downstream image generation tasks. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) The current adaptation process for downstream fine-tuning models in diffusion models. (b) Our ONE-PIC significantly [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. (a) Pretrained large language models can be applied to [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: The architecture of ONE-PIC for different downstream tasks. We show the visual context of visual conditional control, virtual [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: In visual conditional control generation tasks, different [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: The synthesized results of ONE-PIC under different visual conditional controls. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: (a) The convergence process of ControlNet[ [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: The synthesized images (in red box) of ONE-PIC under DreamBooth [ [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: The synthesized results (in red box) of ONE-PIC under different editing requirements. The top-left corner contains the masked [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: The synthesized results of ONE-PIC under virtual try-on tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]
Figure 11
Figure 11. Figure 11: The visual context and generated results of Image Colorization(a), Fashion Extraction(b), Image Segmentation(c), and Identity [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 31 canonical work pages

  1. [1]

    Tim Brooks, Aleksander Holynski, and Alexei A. Efros. In- structpix2pix: Learning to follow image editing instructions. CoRR, abs/2211.09800, 2022. 3

    arXiv 2022

  2. [2]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 2

    2023

  3. [3]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. In NeurIPS, 2020. 2, 4

    work page 2020

  4. [4]

    Viton-hd: High-resolution virtual try-on via misalignment-aware normalization

    Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo. Viton-hd: High-resolution virtual try-on via misalignment-aware normalization. In CVPR, pages 14131– 14140, 2021. 3, 8

    work page 2021

  5. [5]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 2021. 3

    2021

  6. [6]

    Make-a-scene: Scene- based text-to-image generation with human priors

    Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene- based text-to-image generation with human priors. InECCV,

    work page

  7. [7]

    An image is worth one word: Personalizing text-to- image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patash- nik, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. An image is worth one word: Personalizing text-to- image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022. 3

    arXiv 2022

  8. [8]

    Making pre- trained language models better few-shot learners

    Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre- trained language models better few-shot learners. arXiv preprint arXiv:2012.15723, 2020. 2, 4

    arXiv 2012

Show all 55 references
  1. [9]

    Talecrafter: Interactive story visualization with multiple characters

    Yuan Gong, Youxin Pang, Xiaodong Cun, Menghan Xia, Haoxin Chen, Longyue Wang, Yong Zhang, Xintao Wang, Ying Shan, and Yujiu Yang. Talecrafter: Interactive story visualization with multiple characters. arXiv preprint arXiv:2305.18247, 2023. 3

    2023 arXiv

  2. [10]

    Vec- tor quantized diffusion model for text-to-image synthesis

    Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vec- tor quantized diffusion model for text-to-image synthesis. In CVPR, 2022. 3 9

    2022

  3. [11]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. NeurIPS, 2020. 3, 5

    2020

  4. [12]

    Cascaded diffusion models for high fidelity image generation

    Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. JMLR, 23:47–1,

  5. [13]

    Stableviton: Learning semantic correspon- dence with latent diffusion model for virtual try-on

    Jeongho Kim, Guojung Gu, Minho Park, Sunghyun Park, and Jaegul Choo. Stableviton: Learning semantic correspon- dence with latent diffusion model for virtual try-on. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8176–8185, 2024. 2, 3, 8

    2024

  6. [14]

    High-resolution virtual try-on with misalignment and occlusion-handled conditions

    Sangyun Lee, Gyojung Gu, Sunghyun Park, Seunghwan Choi, and Jaegul Choo. High-resolution virtual try-on with misalignment and occlusion-handled conditions. In ECCV, pages 204–219. Springer, 2022. 3

    2022

  7. [15]

    Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing

    Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre- trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Pro- cessing Systems, 36, 2024. 3

    2024

  8. [16]

    Controlnet++: Improving conditional controls with efficient consistency feedback, 2024

    Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaon- ing Wang, Xuefeng Xiao, and Chen Chen. Controlnet++: Improving conditional controls with efficient consistency feedback, 2024. 3

    2024

  9. [17]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521, 2023. 3

    2023

  10. [18]

    Intelligent grimm–open-ended visual storytelling via latent diffusion models

    Chang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang, and Weidi Xie. Intelligent grimm–open-ended visual storytelling via latent diffusion models. arXiv preprint arXiv:2306.00973, 2023. 3

    2023 arXiv

  11. [19]

    Open-edit: Open- domain image manipulation with open-vocabulary instruc- tions

    Xihui Liu, Zhe Lin, Jianming Zhang, Handong Zhao, Quan Tran, Xiaogang Wang, and Hongsheng Li. Open-edit: Open- domain image manipulation with open-vocabulary instruc- tions. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- in...

    2020

  12. [20]

    P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short ...

    2022

  13. [21]

    Repaint: Inpainting using denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022. 2, 5

    2022

  14. [22]

    Peft: State-of-the-art parameter-efficient fine-tuning meth- ods

    Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning meth- ods. https://github.com/huggingface/peft ,

  15. [23]

    SDEdit: Guided image synthesis and editing with stochastic differential equa- tions

    Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jia- jun Wu, Jun-Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equa- tions. In International Conference on Learning Representa- tions, 2022. 3

    2022

  16. [24]

    Ladi-vton: Latent diffusion textual-inversion enhanced virtual try-on

    Davide Morelli, Alberto Baldrati, Giuseppe Cartella, Mar- cella Cornia, Marco Bertini, and Rita Cucchiara. Ladi-vton: Latent diffusion textual-inversion enhanced virtual try-on. arXiv preprint arXiv:2305.13501, 2023. 3

    2023 arXiv

  17. [25]

    T2i- adapter: Learning adapters to dig out more controllable abil- ity for text-to-image diffusion models, 2023

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable abil- ity for text-to-image diffusion models, 2023. 3

    2023

  18. [26]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In ICML, 2021. 3

    2021

  19. [27]

    GLIDE: towards photorealis- tic image generation and editing with text-guided diffusion models

    Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealis- tic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, I...

    2022

  20. [28]

    Glide: Towards photorealis- tic image generation and editing with text-guided diffusion models

    Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealis- tic image generation and editing with text-guided diffusion models. In ICML, 2022. 3

    2022

  21. [29]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  22. [30]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 3, 5

    2023 arXiv

  23. [31]

    Unicontrol: A unified diffusion model for controllable visual generation in the wild

    Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffusion model for controllable visual generation in the wild. arXiv preprint arXiv:2305.11147, 2023. 3

    2023 arXiv

  24. [32]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 2, 4

    2019

  25. [33]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 2, 4

    2021

  26. [34]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Pe- ter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21(140):1–67, 2020. 2, 4

    2020

  27. [35]

    Make-a-story: Visual memory conditioned consistent story generation

    Tanzila Rahman, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Shweta Mahajan, and Leonid Sigal. Make-a-story: Visual memory conditioned consistent story generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2493–2502, 2023. 3 10

    2023

  28. [36]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021. 2, 3

    2021

  29. [37]

    Hierarchical text-conditional image gen- eration with clip latents

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image gen- eration with clip latents. arXiv preprint arXiv:2204.06125,

    arXiv

  30. [38]

    High-resolution image syn- thesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj¨orn Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022. 2, 3

    2022

  31. [39]

    Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation

    Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500...

    2023

  32. [40]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv...

    2022 arXiv

  33. [41]

    Deep unsupervised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015. 3

    2015

  34. [42]

    Df-gan: A simple and effective baseline for text-to-image synthesis

    Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Xu Changsheng. Df-gan: A simple and effective baseline for text-to-image synthesis. In CVPR, 2022. 3

    2022

  35. [43]

    Galip: Generative adversarial clips for text-to-image synthe- sis

    Ming Tao, Bing-Kun Bao, Hao Tang, and Changsheng Xu. Galip: Generative adversarial clips for text-to-image synthe- sis. arXiv preprint arXiv:2301.12959, 2023. 3

    2023 arXiv

  36. [44]

    Storyimager: A unified and efficient frame- work for coherent story visualization and completion

    Ming Tao, Bing-Kun Bao, Hao Tang, Yaowei Wang, and Changsheng Xu. Storyimager: A unified and efficient frame- work for coherent story visualization and completion. In European Conference on Computer Vision, pages 479–495. Springer, 2025. 2

    2025

  37. [45]

    In- context learning unlocked for diffusion models

    Zhendong Wang, Yifan Jiang, Yadong Lu, Pengcheng He, Weizhu Chen, Zhangyang Wang, Mingyuan Zhou, et al. In- context learning unlocked for diffusion models. Advances in Neural Information Processing Systems , 36:8542–8562,

  38. [46]

    Omnigen: Unified image generation

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xin- grun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. arXiv preprint arXiv:2409.11340, 2024. 3, 6

    2024 arXiv

  39. [47]

    Gp- vton: Towards general purpose virtual try-on via collabo- rative local-flow global-parsing learning

    Zhenyu Xie, Zaiyu Huang, Xin Dong, Fuwei Zhao, Haoye Dong, Xijin Zhang, Feida Zhu, and Xiaodan Liang. Gp- vton: Towards general purpose virtual try-on via collabo- rative local-flow global-parsing learning. In CVPR, pages 23550–23559, 2023. 3

    2023

  40. [48]

    Attngan: Fine- grained text to image generation with attentional generative adversarial networks

    Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine- grained text to image generation with attentional generative adversarial networks. In CVPR, 2018. 3

    2018

  41. [49]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  42. [50]

    Scaling autoregres- sive models for content-rich text-to-image generation

    Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gun- jan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yin- fei Yang, Burcu Karagol Ayan, et al. Scaling autoregres- sive models for content-rich text-to-image generation. arXiv preprint arXiv:2206.10789, 2022. 3

    2022 arXiv

  43. [51]

    Stack- gan: Text to photo-realistic image synthesis with stacked generative adversarial networks

    Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiao- gang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stack- gan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017. 3

    2017

  44. [52]

    Magicbrush: A manually annotated dataset for instruction- guided image editing

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction- guided image editing. Advances in Neural Information Pro- cessing Systems, 36, 2024. 3, 7

    2024

  45. [53]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3836–3847, 2023. 3, 4, 6

    2023

  46. [54]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 2

    2023

  47. [55]

    Dm- gan: Dynamic memory generative adversarial networks for text-to-image synthesis

    Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm- gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In CVPR, 2019. 3 11

    2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.