Pith. sign in

REVIEW 5 major objections 3 minor 76 references

Generating Compositional Scenes via Text-to-image RGBA Instance Generation

T0 review · 5 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proposes a two-stage diffusion pipeline that first generates each object as a transparent RGBA image and then blends these pre-generated instances into a composite scene, claiming finer attribute and layout control and…

desk verdict The RGBA instance generator is the real contribution and holds up; the scene-composition superiority claim is not yet measured and should be quantified or toned down. read the letter →

arxiv 2411.10913 v1 pith:5RDEP3XP submitted 2024-11-16 cs.CV cs.LG

Alessandro Fontanella , Petru-Daniel Tudosiu , Yongxin Yang , Shifeng Zhang , Sarah Parisot This is my paper · ORCID
classification cs.CVcs.LG
keywords text-to-imagegenerationRGBAimagestransparencymulti-layercompositionnoiseblendinglatentdiffusionmodelslayoutcontrolsceneediting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that image generation can be split into two controllable stages: a diffusion model fine-tuned to emit isolated objects as RGBA images (red, green, blue, plus a transparency channel), and a multi-layer noise-blending process that assembles those pre-generated instances into a full scene. The motivation is that layout-conditioned generators produce all objects in one denoising pass, so changing an attribute, moving an object, or reordering overlaps forces re-generation with little preservation. If the claim holds, users would generate an object once and then place, resize, replace, or remove it within a scene while keeping the rest of the image consistent. The measured evidence for the instance stage is KID 0.0150, IoU 0.892, and CLIP score 18.49, which the paper reports as better than matting-based and layer-based baselines.

What carries the argument

The load-bearing mechanism is a disentangled RGBA latent space: the VAE predicts two separate Gaussian distributions, one for RGB and one for $\alpha$, each with its own KL loss, so the diffusion model can sample and denoise the two channels with mutual conditioning. At inference the $\alpha$ latent is updated first, and the RGB update is conditioned on it; the paper reports that this conditional scheme corrects small mask details. The second mechanism is multi-layer noise blending: $K+1$ images are initialised from the same noise, each instance's DDIM-inverted latents are pasted into successively higher layers using the downsampled $\alpha$ mask for the first $n$ timesteps ($y^k_t = y^{k-1}_t (1-m_k) + x^k_t m_k$), with optional background blending and cross-layer consistency, so instance appearance and layout are fixed early while later steps harmonise the scene.

What would settle it

Evaluate the RGBA generator's alpha output against human-annotated mattes from a held-out matting benchmark and compare that IoU with the reported 0.892; a large drop, or parity with matting-based baselines, would show the transparency quality is an artifact of training and evaluating on correlated automatic masks.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a latent diffusion model can be made to generate RGBA instances natively, rather than generating an RGB image and then extracting a mask with matting. The authors fine-tune a PixArt-α latent diffusion model with a VAE whose latent space predicts separate RGB and alpha distributions, and they train the diffusion model to predict RGB and alpha noise jointly under mutual conditioning, so at inference the alpha latent is denoised first and then guides the RGB update. They report that this explicit transparency modelling yields the best KID, IoU, and CLIP scores among the compared instance-generation baselines. For composition, each pre-generated instance is encoded and inverted back to its initial noise with DDIM, and its noisy latent is injected into a shared denoising trajectory layer by layer, weighted by the downsampled alpha mask, for the first n timesteps, so later steps harmonise the assembled layers. The result, they argue, is a pipeline with intrinsic layout and attribute editing: replacing an instance, moving a bounding box, or changing an attribute only requires regenerating that layer and re-running composition.

Load-bearing premise

The entire transparency quality claim rests on the automatically extracted alpha masks in the MuLAn training set being accurate enough to teach the model what real object boundaries look like, because if those masks are systematically wrong, the learned transparency and the IoU score (measured against another automatic matting estimate) inherit the same error.

Editorial extensions

If this is right

  • The RGBA generator lets a user specify each object's attributes in its own prompt and get an isolated transparent instance, removing the matting step and the attribute-bleeding it introduces.
  • The multi-layer noise blending turns layout changes into layer edits: moving or resizing an instance only requires re-inverting that instance and re-running the composition, rather than regenerating the whole scene.
  • Overlapping objects can be composed with explicit ordering, because instances are injected one layer at a time instead of being averaged together.
  • Scene manipulation such as attribute replacement, instance substitution, and layout adjustment is demonstrated without any explicit image-preservation or editing machinery, relying only on the layer structure and a fixed seed.
  • The paper reports the best KID, IoU, and CLIP score among the compared instance-generation methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension is to expose the layer structure as an editing interface: select an instance, re-prompt it, and re-blend only that layer; the paper demonstrates the pieces but does not build the interface.
  • The mutual-conditioning recipe for RGB and alpha should transfer to other correlated output channels, such as depth, surface normals, or segmentation maps, wherever a latent diffusion model must produce two dependent fields.
  • The composition step only assumes inverted latents and masks, so the same layer-wise blending could be tested on video frames or multi-view images if instances can be generated consistently; the paper does not explore temporal or cross-view consistency.
  • The reported IoU should be re-checked against human-annotated mattes rather than masks produced by an automatic matting estimator; that would separate genuine boundary quality from agreement between two automatic estimators.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. This paper proposes a two-stage pipeline for controllable text-to-image generation. First, a latent diffusion model (PixArt-α) is fine-tuned to generate isolated RGBA instances, using a VAE with a disentangled latent space and a mutual-conditioning procedure between RGB and alpha latents. Second, pre-generated instances are assembled into scenes by multi-layer noise blending, where instance latents are injected layer by layer with parameters controlling blending, background blending, and cross-layer consistency. The RGBA stage is evaluated with KID, IoU against ICON masks, and CLIP score, while the scene-composition stage is evaluated qualitatively against PixArt-α, GLIGEN, MultiDiffusion, and Instance Diffusion.

Significance. If the RGBA-generation result holds, this is a useful contribution: explicit transparency-aware latent modeling yields high-quality instance generation with attribute control, and the scene-manipulation results are visually promising. The paper reports a large quantitative gap in KID (0.0150 vs. the next best 0.0447) and IoU (0.892 vs. 0.811), and the ablations support the proposed training and sampling choices. The training details and limitations are documented in unusual depth. The scene-composition idea of sequential layer injection is a plausible improvement over simultaneous layer assembly. However, the central composition claim is not yet supported quantitatively, and some method details need clarification before the work can be fully assessed.

major comments (5)
  1. [4.2, Figs. 5–6 and 9; Section 5; Appendix D] The central claim of a higher degree of control over layout and object attributes than competing methods is currently supported only by qualitative comparisons. The paper says baselines were generated 'with multiple seeds and select the best result,' but it does not state how many seeds were tried, what criterion was used for selection, or whether the same procedure was applied to the proposed method; under this protocol, a favorable comparison can be produced by cherry-picking baseline failures. Sections 5 and D also admit that independent instance generation makes coherent scene assembly challenging, which makes quantitative support especially important. Please add a quantitative composition evaluation, e.g., layout accuracy against the input bounding boxes, per-instance attribute accuracy via CLIP or a detector, and a user study, with error bars over seeds.
  2. [Table 1; Checklist item 7] The quantitative claims in Table 1 are point estimates from a single run. The checklist explicitly states that multiple runs were not performed due to compute limitations, but the paper contains stochastic experiments, so answer 'NA' for statistical significance is not appropriate. KID is a Monte Carlo estimator and diffusion sampling is seed-dependent; without error bars or confidence intervals, the reported gaps (e.g., KID 0.0150 vs. 0.0447, IoU 0.892 vs. 0.811) cannot be assessed for significance. Please report mean and standard deviation over at least 3–5 seeds, or bootstrap confidence intervals, for the main metrics and the ablation rows.
  3. [Appendix B, Algorithm 1] The pseudocode for cross-layer consistency is inconsistent with the method description. The consistency branch is written as 'Elif t ≥ n + ns' attached to the preceding 'If t ≥ n'; since n + ns > n and the loop index t decreases, that branch is unreachable. The text in §3.3 describes consistency as applying for 'ns subsequent timesteps' after the n blending steps. Please correct the pseudocode and specify exactly how n, b, ns relate to the discrete denoising schedule; this is needed to reproduce the composition procedure and to interpret Fig. 12.
  4. [3.2] The disentangled RGBA latent space is central to the method but is not fully specified. The text says the VAE 'predicts two separate distributions N(x : µRGB, ΣRGB) and N(x : µα, Σα)' while preserving a 4-channel latent space, but it is unclear whether these are two heads over the same latent, two disjoint latent partitions, or two separate codes that are later concatenated, and it is unclear how the RGB and alpha latents used for mutual conditioning in the diffusion model are derived. Please clarify the latent-space construction with tensor shapes and a short pseudocode description of the VAE forward pass.
  5. [4.1, Table 1; Appendix A.1] The IoU evaluation uses masks from ICON as 'ground truth,' while the majority of the training data comes from MuLAn, which the appendix describes as 'automatically generated' and 'noisy.' If the automatic extraction pipeline that produced MuLAn shares estimation biases with ICON, the IoU of 0.892 could overstate alpha quality for the proposed model and distort the comparison with matting baselines. This is a correctness-risk concern rather than a claim of circularity; a concrete test is to report IoU on a held-out portion of the human-annotated matting datasets listed in Appendix A.1 (e.g., PPM-100, AM-2k) or on manually corrected masks.
minor comments (3)
  1. [Checklist item 5] The checklist answers 'NA' for open access to code and data, although the paper contains experiments. If code and model checkpoints are not released, the answer should be 'No' rather than 'NA,' and the paper should state what artifacts will be available for reproduction.
  2. [Eq. (1)] The loss in Eq. (1) omits the squared norm; it should read ∥ϵ − ϵθ(yt, t, C)∥² or an explicit mean-square-error notation.
  3. [Table 1 caption] The caption writes 'IoU (Jac-card)'; this appears to be a typo for 'Jaccard.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central RGBA-generation and composition claims are trained and evaluated against external baselines and public data, not reduced to the paper's own inputs.

full rationale

The paper is empirical rather than derivational, and I find no step in which a claimed result is equivalent by construction to its inputs. The RGBA generator is produced by fine-tuning a pre-trained PixArt model on public instance data (MuLAn plus matting datasets), and its quality is measured against external baselines (SD, PixArt, Matte Anything, Text2Layer, LayerDiffusion) with KID, IoU against ICON masks, and CLIP score; none of these metrics is used as a training loss or as a fitted parameter of the model. The mutual-conditioning training and sampling scheme is a recursive denoising procedure, not a self-referential definition: the network is trained to predict noise from paired latents at possibly different timesteps, and inference alternates updates, so the output is not identical to the input by construction. The scene-composition algorithm (Eqs. 2-3) intentionally injects inverted instance latents into the denoising trajectory, so instance presence in the output is by design; but the paper does not claim to predict composition quality from an independent first-principles law. That section is evaluated only qualitatively and with best-seed selection for baselines, which is a genuine validation weakness for the 'higher degree of control' claim, but it is not circularity. The one notable self-citation, MuLAn, is a publicly released dataset produced by overlapping authors and is the main training source; however it is used as data, not as an assumed theorem or fitted output, and the evaluation is external to it. The paper even discloses the dataset's noisy automatic masks and its own limitations, so there is no hidden reduction of the central claim to a self-citation chain or to the evaluation protocol. Score 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 1 invented entities

The paper's main scientific currency is a trained model plus a procedural recipe, so its ledger is dominated by training data and hyperparameters rather than mathematical assumptions. The most consequential uncharged items are the quality of the automatically generated MuLAn alpha masks and the hand-picked composition hyperparameters (n, b, ns), which are tuned visually and demonstrated to have a large effect on output.

free parameters (5)
  • n (noise blending timesteps) = 30
    Selected by visual inspection over {0,10,20,30,40,50}; the paper shows a qualitative analysis in Fig. 11 and concludes n=30 yields the smoothest blending. This is a hand-tuned, visually motivated parameter rather than a quantity derived from data.
  • b (background blending timesteps) = 20 (composition experiments), 0 (editing experiments)
    Chosen by visual inspection; the paper states b has a subtler impact and too-high b risks introducing instance information into the background.
  • ns (cross-layer consistency timesteps) = 10
    Selected visually from Fig. 12; higher values improve instance consistency but the paper found reduced composition performance with similar strategies.
  • guidance scale and guidance rescaling = GS=2.5, GR=0.25
    Selected by a grid search on validation data using KID (Fig. 10); this is a free parameter tuned to the validation set.
  • KL regularization weight wKL = 1
    A training choice for the RGBA VAE; not fitted to data but is an important ad hoc decision affecting disentanglement and image quality.
assumptions (3)
  • domain assumption MuLAn training instances, generated by an automatic decomposition pipeline, provide a valid ground truth for RGBA instance generation.
    The paper relies on 87,989 MuLAn instances for Stage 1 fine-tuning and Appendix A.1 itself calls this data 'noisy'. If the automatic decomposition is systematically biased, the generator's alpha masks inherit that bias.
  • domain assumption The ICON pretrained segmenter provides a trustworthy 'ground truth' for alpha masks.
    IoU is computed between generated alpha masks and ICON segmentation masks external to the system. The paper acknowledges this convention follows [60], but a segmenter with imperfect boundaries could compress the difference between masks.
  • domain assumption A stable diffusion prior in PixArt-alpha remains usable after full fine-tuning and can be recovered via a tuned guidance scale.
    The method fine-tunes the VAE and the full LDM; the paper notes the model operates in a new latent space. The assumption is that DDIM inversion on the new latent space still provides a meaningful trajectory for blending.
invented entities (1)
  • Disentangled latent structure with separate RGB and alpha posterior distributions in a shared VAE latent space
    purpose: Enables a diffusion model to generate RGB content and transparency as separate but mutually conditioned latents.
    The disentanglement claim is demonstrated through subjective visual comparisons (Fig. 7) and end-to-end metrics (Table 1), but the paper provides no direct probe of whether alpha and RGB information are actually separated in the latent. The improvement could come from the higher KL weight alone.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generating Compositional Scenes via Text-to-image RGBA Instance Generation." pith.science (2026). https://pith.science/paper/5RDEP3XP

@misc{pith2026241110913,
  author       = {Pith},
  title        = {Pith review of: Generating Compositional Scenes via Text-to-image RGBA Instance Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5RDEP3XP}},
  note         = {Machine review of arXiv:2411.10913}
}
read the original abstract

Text-to-image diffusion generative models can generate high quality images at the cost of tedious prompt engineering. Controllability can be improved by introducing layout conditioning, however existing methods lack layout editing ability and fine-grained control over object attributes. The concept of multi-layer generation holds great potential to address these limitations, however generating image instances concurrently to scene composition limits control over fine-grained object attributes, relative positioning in 3D space and scene manipulation abilities. In this work, we propose a novel multi-stage generation paradigm that is designed for fine-grained control, flexibility and interactivity. To ensure control over instance attributes, we devise a novel training paradigm to adapt a diffusion model to generate isolated scene components as RGBA images with transparency information. To build complex images, we employ these pre-generated instances and introduce a multi-layer composite generation process that smoothly assembles components in realistic scenes. Our experiments show that our RGBA diffusion model is capable of generating diverse and high quality instances with precise control over object attributes. Through multi-layer composition, we demonstrate that our approach allows to build and manipulate images from highly complex prompts with fine-grained control over object appearance and location, granting a higher degree of control than competing methods.

Figures

Figures reproduced from arXiv: 2411.10913 by the authors.

Figure 1
Figure 1. Overview of key components of our proposed methodology. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Our model can generalise to different styles and to follow detailed instructions. Top row: ‘a [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Instances generated with the captions: ‘a majestic brown bear with dark brown fur, its [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Our proposed training and sampling approaches [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visual examples of scene composition results. RGBA instances are highlighted in [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Visual examples of scene manipulations compared to Instance Diffusion. Our layer-based [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Images generated with our LDM fine-tuned in the latent space of VAEs that were trained [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Sample RGBA instances. We are able to generate a wide variety of subjects. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Additional scene composition results. RGBA instances and their attributes are bolded in [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: KID obtained with different combination of the guidance scale and guidance rescaling [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Influence of scene composition parameters over generated scene content. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: Influence of layer consistency parameter [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Impact of changing the random seed on scene composition consistency. Image caption: a [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Visual examples of LayerDiffusion scene composition results. RGBA instances are [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 32 canonical work pages

  1. [1]

    Phi-3 technical re- port: A highly capable language model locally on your phone

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  2. [2]

    Multidiffusion: Fusing diffusion paths for controlled image generation

    Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiffusion: Fusing diffusion paths for controlled image generation. In Proceedings of the 40th International Conference on Machine Learning, 2023

    work page 2023

  3. [3]

    Demystifying mmd gans

    Mikołaj Bi´nkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.01401, 2018

    arXiv 2018

  4. [4]

    Instructpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.09800, 2022

    arXiv 2022

  5. [5]

    Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing

    Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22560– 22570, 2023

    work page 2023

  6. [6]

    Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023

    arXiv

  7. [7]

    Anydoor: Zero-shot object-level image customization

    Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. arXiv preprint arXiv:2307.09481, 2023

    arXiv 2023

  8. [8]

    Zero-shot spatial layout conditioning for text-to-image diffusion models

    Guillaume Couairon, Marlène Careil, Matthieu Cord, Stéphane Lathuilière, and Jakob Verbeek. Zero-shot spatial layout conditioning for text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2174–2183, 2023

    work page 2023

Show all 76 references
  1. [9]

    Diffusion self-guidance for controllable image generation

    Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. Advances in Neural Information Processing Systems, 36:16222–16239, 2023

    2023

  2. [10]

    User- guided deep human image matting using arbitrary trimaps

    Xiaonan Fang, Song-Hai Zhang, Tao Chen, Xian Wu, Ariel Shamir, and Shi-Min Hu. User- guided deep human image matting using arbitrary trimaps. IEEE Transactions on Image Processing, 31:2040–2052, 2022

    2022

  3. [11]

    Diffusion models for counterfactual generation and anomaly detection in brain images

    Alessandro Fontanella, Grant Mair, Joanna Wardlaw, Emanuele Trucco, and Amos Storkey. Diffusion models for counterfactual generation and anomaly detection in brain images. arXiv preprint arXiv:2308.02062, 2023

    2023 arXiv

  4. [12]

    Prompt-to-prompt image editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022

    2022 arXiv

  5. [13]

    Clipscore: A reference-free evaluation metric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718, 2021

    2021 arXiv

  6. [14]

    Classifier-free diffusion guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022

    2022 arXiv

  7. [15]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  8. [16]

    T2i-compbench: A compre- hensive benchmark for open-world compositional text-to-image generation

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A compre- hensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723–78747, 2023. 11

    2023

  9. [17]

    Diffstyler: Controllable dual diffusion for text-driven image stylization

    Nisha Huang, Yuxin Zhang, Fan Tang, Chongyang Ma, Haibin Huang, Weiming Dong, and Changsheng Xu. Diffstyler: Controllable dual diffusion for text-driven image stylization. IEEE Transactions on Neural Networks and Learning Systems, 2024

    2024

  10. [18]

    Layerdiff: Exploring text-guided multi-layered composable image synthesis via layer-collaborative diffusion model, 2024

    Runhui Huang, Kaixin Cai, Jianhua Han, Xiaodan Liang, Renjing Pei, Guansong Lu, Songcen Xu, Wei Zhang, and Hang Xu. Layerdiff: Exploring text-guided multi-layered composable image synthesis via layer-collaborative diffusion model, 2024

    2024

  11. [19]

    Modnet: Real-time trimap-free portrait matting via objective decomposition

    Zhanghan Ke, Jiayu Sun, Kaican Li, Qiong Yan, and Rynson WH Lau. Modnet: Real-time trimap-free portrait matting via objective decomposition. InProceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1140–1147, 2022

    2022

  12. [20]

    Diffusionclip: Text-guided diffusion models for robust image manipulation

    Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426–2435, 2022

    2022

  13. [21]

    Diffusion models already have a semantic latent space

    Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210.10960, 2022

    2022 arXiv

  14. [22]

    Bridging composite and real: towards end-to-end deep image matting

    Jizhizi Li, Jing Zhang, Stephen J Maybank, and Dacheng Tao. Bridging composite and real: towards end-to-end deep image matting. International Journal of Computer Vision, 130(2):246– 266, 2022

    2022

  15. [23]

    Deep automatic natural image matting

    Jizhizi Li, Jing Zhang, and Dacheng Tao. Deep automatic natural image matting. arXiv preprint arXiv:2107.07235, 2021

    2021 arXiv

  16. [24]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521, 2023

    2023

  17. [25]

    Common diffusion noise schedules and sample steps are flawed

    Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5404–5411, 2024

    2024

  18. [26]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023

    2023

  19. [27]

    Pseudo numerical methods for diffusion models on manifolds

    Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778, 2022

    2022 arXiv

  20. [28]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  21. [29]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775–5787, 2022

    2022

  22. [30]

    Repaint: Inpainting using denoising diffusion probabilistic models

    Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461–11471, 2022

    2022

  23. [31]

    Dragondiffusion: Enabling drag-style manipulation on diffusion models

    Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.02421, 2023

    2023 arXiv

  24. [32]

    Glide: Towards photorealistic image generation and editing with text-guided diffusion models

    Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741, 2021

    2021 arXiv

  25. [33]

    Diffusion with offset noise, 2023

    Nicholas Guttenberg. Diffusion with offset noise, 2023. https://www.crosslabs.org/ blog/diffusion-with-offset-noise ,. 12

    2023

  26. [34]

    Zero-shot image-to-image translation

    Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–11, 2023

    2023

  27. [35]

    Best prompts for text-to-image models and how to find them

    Nikita Pavlichenko and Dmitry Ustalov. Best prompts for text-to-image models and how to find them. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2067–2071, 2023

    2023

  28. [36]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195–4205, 2023

    2023

  29. [37]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023

    2023 arXiv

  30. [38]

    Layered rendering diffusion model for zero-shot guided image synthesis

    Zipeng Qi, Guoxi Huang, Zebin Huang, Qin Guo, Jinwen Chen, Junyu Han, Jian Wang, Gang Zhang, Lufei Liu, Errui Ding, et al. Layered rendering diffusion model for zero-shot guided image synthesis. arXiv preprint arXiv:2311.18435, 2023

    2023 arXiv

  31. [39]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  32. [40]

    Move anything with layered scene diffusion, 2024

    Jiawei Ren, Mengmeng Xu, Jui-Chieh Wu, Ziwei Liu, Tao Xiang, and Antoine Toisoul. Move anything with layered scene diffusion, 2024

    2024

  33. [41]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  34. [42]

    U-net: Convolutional networks for biomedical image segmentation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part ...

    2015

  35. [43]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  36. [44]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020

    2010 arXiv

  37. [45]

    Towards real-time text-driven image manipulation with unconditional diffusion models

    Nikita Starodubcev, Dmitry Baranchuk, Valentin Khrulkov, and Artem Babenko. Towards real-time text-driven image manipulation with unconditional diffusion models. arXiv preprint arXiv:2304.04344, 2023

    2023 arXiv

  38. [46]

    Human instance matting via mutual guidance and multi-instance refinement

    Yanan Sun, Chi-Keung Tang, and Yu-Wing Tai. Human instance matting via mutual guidance and multi-instance refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2647–2656, 2022

    2022

  39. [47]

    Mulan: A multi layer annotated dataset for controllable text-to-image generation

    Petru-Daniel Tudosiu, Yongxin Yang, Shifeng Zhang, Fei Chen, Steven McDonagh, Gerasimos Lampouras, Ignacio Iacobacci, and Sarah Parisot. Mulan: A multi layer annotated dataset for controllable text-to-image generation. arXiv preprint arXiv:2404.02790, 2024

    2024 arXiv

  40. [48]

    Plug-and-play diffusion features for text-driven image-to-image translation

    Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1921–1930, 2023

    1921

  41. [49]

    In- stancediffusion: Instance-level control for image generation, 2024

    Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. In- stancediffusion: Instance-level control for image generation, 2024. 13

    2024

  42. [50]

    Stylediffusion: Controllable disentangled style transfer via diffusion models

    Zhizhong Wang, Lei Zhao, and Wei Xing. Stylediffusion: Controllable disentangled style transfer via diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7677–7689, 2023

    2023

  43. [51]

    Boxdiff: Text-to-image synthesis with training-free box-constrained diffu- sion

    Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained diffu- sion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7452–7461, 2023

    2023

  44. [52]

    Dreaminpainter: Text-guided subject-driven image inpainting with diffusion models

    Shaoan Xie, Yang Zhao, Zhisheng Xiao, Kelvin CK Chan, Yandong Li, Yanwu Xu, Kun Zhang, and Tingbo Hou. Dreaminpainter: Text-guided subject-driven image inpainting with diffusion models. arXiv preprint arXiv:2312.03771, 2023

    2023 arXiv

  45. [53]

    Accelerating diffusion sampling with optimized time steps.arXiv preprint arXiv:2402.17376, 2024

    Shuchen Xue, Zhaoqiang Liu, Fei Chen, Shifeng Zhang, Tianyang Hu, Enze Xie, and Zhenguo Li. Accelerating diffusion sampling with optimized time steps.arXiv preprint arXiv:2402.17376, 2024

    2024 arXiv

  46. [54]

    Reco: Region-controlled text-to-image generation

    Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14246–14255, 2023

    2023

  47. [55]

    Matte anything: Interactive natural image matting with segment anything model

    Jingfeng Yao, Xinggang Wang, Lang Ye, and Wenyu Liu. Matte anything: Interactive natural image matting with segment anything model. Image and Vision Computing, page 105067, 2024

    2024

  48. [56]

    Mask guided matting via progressive refinement network

    Qihang Yu, Jianming Zhang, He Zhang, Yilin Wang, Zhe Lin, Ning Xu, Yutong Bai, and Alan Yuille. Mask guided matting via progressive refinement network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1154–1163, 2021

    2021

  49. [57]

    Transparent image layer diffusion using latent trans- parency

    Lvmin Zhang and Maneesh Agrawala. Transparent image layer diffusion using latent trans- parency. arXiv preprint arXiv:2402.17113, 2024

    2024 arXiv

  50. [58]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023

    2023

  51. [59]

    Paste, inpaint and harmonize via denoising: Subject-driven image editing with pre-trained diffusion model

    Xin Zhang, Jiaxian Guo, Paul Yoo, Yutaka Matsuo, and Yusuke Iwasawa. Paste, inpaint and harmonize via denoising: Subject-driven image editing with pre-trained diffusion model. arXiv preprint arXiv:2306.07596, 2023

    2023 arXiv

  52. [60]

    Text2layer: Layered image generation using latent diffusion model

    Xinyang Zhang, Wentian Zhao, Xin Lu, and Jeff Chien. Text2layer: Layered image generation using latent diffusion model. arXiv preprint arXiv:2307.09781, 2023

    2023 arXiv

  53. [61]

    Write a detailed caption of image

    Mingchen Zhuge, Deng-Ping Fan, Nian Liu, Dingwen Zhang, Dong Xu, and Ling Shao. Salient object detection via integrity learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3738–3752, 2022. 14 A RGBA training details and more experiments A.1 Datasets ...

    2022

  54. [62]

    • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: We present the experiments justifying our claims in the Experiments Section and in the Appendix Guidelines: • The answ...

  55. [63]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss methods limitations in the conclusion of the main paper, and in a dedicated section in the Appendix. Guidelines: • The answer NA means tha...

  56. [64]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory Assumptions and Proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 24 Answer: [NA] Justification: We do not present theoretical results. Guidelines: • The answer NA means that the paper do...

  57. [65]

    All datasets used in this work are publicly available

    Experimental Result Reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  58. [66]

    Open sourcing of our code will depend on internal approval

    Open access to data and code 25 Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [NA] Justification: The dataset used is avail...

  59. [67]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: They are discussed in the Exp...

  60. [68]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment Statistical Significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [NA] Justification: We were not able to perform multiple runs due to co...

  61. [69]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: In the Experiments and ...

  62. [70]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: We made sure to follow the Ethics Guidelines. Guidelines: • The answer NA means...

  63. [71]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: This is discussed in a dedicated broader and societal impact section in the Appendix. Guidelines: • The a...

  64. [72]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: Potential safe...

  65. [73]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite the sour...

  66. [74]

    Guidelines: • The answer NA means that the paper does not release new assets

    New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: No released new assets. Guidelines: • The answer NA means that the paper does not release new assets. • Researchers s...

  67. [75]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  68. [76]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.