Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

ROICtrl: Boosting Instance Control for Visual Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ROI-Unpool lets diffusion models crop, refine, and paste region features, making box-grounded multi-instance generation accurate and cheap.

desk verdict ROICtrl's ROI-Unpool is a clean, well-validated idea for instance control in diffusion models; the headline gains hold in aggregate, but the evaluation lacks error bars and the heavy-overlap failure is a real limit. read the letter →

arxiv 2411.17949 v1 pith:M7Z3SWKZ submitted 2024-11-27 cs.CV

Yuchao Gu , Yipin Zhou , Yunfan Ye , Yixin Nie , Licheng Yu , Pingchuan Ma , Kevin Qinghong Lin , Mike Zheng Shou This is my paper · ORCID
classification cs.CV
keywords regionalinstancecontrolROI-UnpoolROI-Aligndiffusionmodeladaptermulti-instancegenerationspatialalignmentboundingboxtext-to-image
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that diffusion models can control many instances precisely if region-of-interest features are handled explicitly, by cropping them, processing them on a small grid, and pasting them back. It introduces ROI-Unpool, the inverse of ROI-Align, which restores cropped features to their original coordinates without quantization error. On top of this operation it builds ROICtrl, an adapter that injects each instance caption through pretrained cross-attention inside the cropped region and blends the result with the global attention output. The paper claims that this design outperforms prior implicit- and mask-based instance-control adapters in spatial alignment and regional text alignment while using less memory and compute. If correct, it would make multi-instance generation from boxes and free-form captions practical inside existing diffusion models and their add-ons.

What carries the argument

The load-bearing object is ROI-Unpool, the inverse of ROI-Align. ROI-Align maps an arbitrary box in the spatial feature map to a fixed $r \times r$ grid by bilinear sampling from the four nearest feature points; ROI-Unpool maps that grid back to the same spatial location without coordinate quantization, computing partial values at borders and leaving non-ROI positions empty. This operation is what lets the adapter process each instance separately on a tiny feature grid, with cost independent of the full feature-map resolution, and then reinsert the result so the pretrained decoder still sees a complete spatial feature map.

What would settle it

Run ROICtrl on layouts with heavily overlapping boxes whose captions differ only by one attribute, such as color, and measure per-box attribute accuracy as overlap increases. The paper's own limitation figure shows instability in this regime; if accuracy drops sharply once boxes overlap by more than about half their area, then the claimed precision of the learned blending is bounded exactly where instance separation matters most.

Watch

Extended reading notes

Core claim

The central discovery is that explicit ROI manipulation is viable in generative diffusion models if the extraction operation comes with a matching pasting operation. ROI-Align crops a variable-sized region into a fixed grid of size $r \times r$ using bilinear interpolation; ROI-Unpool inverts this by scattering the $r \times r$ features back to the four nearest grid points at the original coordinates, filling non-ROI positions with zeros and computing partial values at borders. ROICtrl applies pretrained cross-attention to each cropped region to inject its instance caption, refines the result with a small ROI self-attention, un-pools it, and blends it with the global attention output through a softmax fusion whose weight is regularized so the instance caption dominates inside the box. The paper reports that this achieves higher mIoU and regional-text accuracy than the implicit-injection and masked-attention baselines on MIG-Bench, InstDiff-Bench, and the newly introduced ROICtrl-Bench, while cutting memory use by about half against the masked-attention variant and speeding inference by a factor of about ten relative to Instance Diffusion.

Load-bearing premise

The load-bearing premise is that cross-attention layers pretrained on full-resolution feature maps will still bind words to the right content when applied to small cropped region features, as long as a small amount of ROI self-attention is added.

Editorial extensions

If this is right

  • ROICtrl can be trained once on a base diffusion model and then dropped into community models fine-tuned from that base without retraining.
  • Because it reuses pretrained cross-attention for instance captions rather than adding new learnable injection modules, it stays compatible with spatial add-ons and embedding-based identity add-ons, extending those tools to multi-instance settings.
  • Explicit ROI extraction with unpooling avoids coordinate-quantization errors, which yields better average precision on small objects than masked-attention baselines.
  • The same adapter supports continuous generation, letting a user regenerate or modify one region while the rest of the image is preserved.
  • On the new ROICtrl-Bench, which includes free-form and out-of-distribution captions, the claimed gains hold for both template and free-form tracks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: applying ROI-Unpool inside transformer-based diffusion backbones is a direct next step the paper leaves open; the operation itself is agnostic to whether the spatial feature map comes from a U-Net or a transformer, so the key uncertainty is how to coordinate the ROI token grid with global attention.
  • Editorial inference: the failure mode shown for heavily overlapping boxes with similar captions suggests the learned blending weight is the bottleneck; a testable fix would be to make the global-to-instance weight condition on overlap area or on caption similarity, rather than a single per-location softmax.
  • Editorial inference: because the cost of ROI processing depends on the number and size of boxes rather than image resolution, the method should scale favorably to very high resolutions, where mask-based attention becomes prohibitive; a benchmark at 2K resolution with many small boxes would check this directly.
  • Editorial inference: the ROICtrl-Bench evaluation could be applied as-is to future models, including captioning-based systems, since it measures spatial mIoU and regional caption match through open-vocabulary detectors and a multimodal judge.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ROICtrl, an adapter for pretrained text-to-image diffusion models that adds regional instance control via bounding boxes paired with free-form captions. The core technical novelty is ROI-Unpool, a complementary operation to ROI-Align: ROI-Align crops variable-sized regions from a feature map, a lightweight instance-caption injection reuses the pretrained cross-attention on the cropped ROI features, ROI self-attention refines them, and ROI-Unpool pastes the refined features back at their original coordinates. The global and instance attention outputs are combined with learnable per-pixel fusion weights, with a regularization term that down-weights the global path inside ROI regions. The method is evaluated on MIG-Bench, InstDiff-Bench, and a newly constructed ROICtrl-Bench, where it reports improved spatial and regional text alignment over GLIGEN, MIGC, and Instance Diffusion, together with reduced training memory and faster inference. The paper also demonstrates compatibility with community models and with add-ons such as ControlNet, T2I-Adapter, IP-Adapter, and ED-LoRA.

Significance. If the reported results hold, the work makes a useful contribution: ROI-Unpool is a simple, intuitively appealing primitive that avoids full-resolution masked attention and enables explicit ROI processing at high resolutions, and the compatibility results with existing add-ons are practically valuable. The ablations (ROI self-attention, Lreg, multi-scale ROI, versus mask-based and embedding-based injection) are informative and support the main design choices. The paper also ships a new benchmark intended to cover free-form instance captions, which addresses a real gap in existing evaluation. However, the quantitative evidence is weakened by the absence of error bars or significance tests, by the reliance on an author-constructed VLM-based benchmark without human validation, and by an acknowledged failure mode for heavily overlapping boxes that is not probed by any of the reported benchmarks. These gaps leave the strength of the central accuracy claim not fully established in the regimes where instance control is hardest.

major comments (3)
  1. [Sec. 3.2 / Fig. 10 / Sec. 8.1] The ROI-Unpool operation is underspecified for overlapping or adjacent ROIs. When two ROI features are unpooled to the same spatial location, the manuscript does not state the aggregation rule (e.g., overwrite, average, or max), yet the acknowledged instability in Fig. 10 for heavily overlapping boxes with similar captions is precisely the regime where this ambiguity should matter. The aggregate results in Tables 2 and 4 are not stratified by overlap or by instance count, so the reader cannot tell whether the claimed improvements persist for dense, overlapping layouts. Because the headline claim is about 'explicit, efficient, and accurate ROI manipulation' for multi-instance control, this is a load-bearing gap; please specify the operation formally and report benchmark numbers split by overlap and by box size.
  2. [Tables 2-4 / Sec. 4.2] All quantitative claims are reported as point estimates with no error bars, confidence intervals, or significance tests, despite some differences being small (e.g., ROICtrl-Bench mIoU 0.652 vs Instance Diffusion 0.607, Acc 48.7 vs 45.6). In addition, ROICtrl-Bench uses MiniCPM-V 2.6 as an automated judge for regional text alignment without reporting human agreement or a validation of the judge against human ratings. Since ROICtrl-Bench is constructed by the authors and the comparison on existing benchmarks is also single-run, the evidence for 'superior performance' is not yet statistically grounded. Please provide multiple seeds, report variances, and include a human or second-judge validation on a sample of the VLM-assessed examples.
  3. [Sec. 3.3.1 / Table 4] The central design choice is to reuse pretrained cross-attention, trained at full UNet resolutions, on small ROI crops (r = 25, 19, 13, 7). The claim that this transfer works is only indirectly supported: the ablation in Table 4 shows that removing ROI self-attention drops ROICtrl-Bench mIoU from 0.652 to 0.540 and InstDiff-Bench AP from 41.0 to 32.7, a roughly 17% relative drop. This shows that the cross-attention transfer is not automatic and that the added ROI self-attention carries much of the burden. The manuscript should analyze when the pretrained cross-attention fails (small objects, OOD captions, heavily overlapping boxes) and demonstrate that the compensation is sufficient, rather than reporting only aggregate gains; the admitted overlap limitation suggests there are regimes where the compensation is incomplete.
minor comments (6)
  1. [Sec. 4.1] The text says 'recaptioning each instance with free-form text generated by CogVLM [2]' but reference [2] is the DALL-E 3 report; CogVLM is not listed in the references. Please correct the citation or add the missing reference.
  2. [Sec. 4.3] The paper acknowledges that ROICtrl is 'slightly worse' than Instance Diffusion on out-of-distribution subjects on ROICtrl-Bench and attributes this to training data size. This is an honest caveat, but it should be discussed more prominently in the comparison summary, since Table 2(c) still claims overall superiority; a breakdown by track would help readers see where the method actually wins.
  3. [Eq. (1)] The diffusion loss notation can be cleaned up: the expectation should be over the noise and timestep distributions, and the squared L2 norm is written with a subscript that may be confused with a vector norm; consider using \mathbb{E}_{z,\epsilon,t} and \|\cdot\|_2^2.
  4. [Table 3] The inference speed test conditions are stated in the caption, but the training memory column does not specify the resolution or batch size used for the memory measurement; please report the exact configuration so that the memory comparison is reproducible.
  5. [Sec. 3.2] The description of ROI-Unpool says 'positions that do not correspond to the ROI region are left empty'; this is ambiguous about boundary handling and about whether the output feature map is zero-initialized before unpooling. A precise algorithmic definition (or a reference to the released code) would remove ambiguity.
  6. [Sec. 4.1] The ROI size formula r = 6 log2 R - 11 is introduced without an ablation of the formula's coefficients; since this choice controls the computational cost and the amount of context per ROI, a brief sensitivity study or a rationale would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; ROICtrl's claims are empirical and corroborated by external benchmarks.

full rationale

ROICtrl's central pipeline is an empirical adapter trained with the standard diffusion denoising loss (Eq. 1) plus a regularization term; no equation defining ROI-Unpool, the blending weights, or the ROI-size schedule is derived from the target benchmark numbers, and no predicted quantity is constructed from the fitted values it is compared against. The claimed advantage is supported by external benchmarks (MIG-Bench and InstDiff-Bench), which are not constructed by the authors, in addition to the author-built ROICtrl-Bench; self-citations to ED-LoRA and VideoSwap appear only in related-work and application context and are not load-bearing. The hand-picked ROI-size formula r = 6 log2 R - 11 and the learnable blending weights are design choices, not fitted inputs renamed as predictions. The admitted limitation for heavily overlapping boxes (Sec. 8.1, Fig. 10) is an honest scope statement, not a circular derivation. Therefore no step reduces by construction to its own inputs, and the paper is self-contained against external benchmarks.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim rests on the trained adapter parameters, a few hand-chosen hyperparameters, and the assumption that pretrained attention and automated evaluators transfer to the ROI-based setting.

free parameters (4)
  • Adapter parameters (fusion weights, ROI self-attention, 1x1 conv) = not enumerated, trained on MS-COCO
    Optimized via Eq. 1; the trained values are the empirical core of the method.
  • alpha (regularization weight) = 0.01
    Hand-set in Section 3.3.3 to balance diffusion loss and global-attention regularization.
  • ROI size formula coefficients = r = 6*log2(R) - 11
    Ad hoc mapping from feature resolution to ROI grid size, introduced in Section 4.1.
  • Box embedding layer = learned
    Adds box-coordinate conditioning for occlusion, following GLIGEN.
assumptions (4)
  • domain assumption Pretrained cross-attention can be directly reused for ROI feature conditioning after a small self-attention step.
    The whole adapter design in Section 3.3.1 assumes this transfer works; if false, the method fails.
  • domain assumption ROI-Align and ROI-Unpool bilinear resampling does not introduce harmful quantization or boundary artifacts in diffusion feature maps.
    The claimed advantage over masked attention depends on this; the paper motivates from object detection but does not prove it for generative features.
  • domain assumption Automated evaluators (YOLO-World, Grounding-DINO, MiniCPM-V, CLIP) provide valid measures of spatial and attribute alignment.
    All headline comparisons rely on these automated judges; no human study is reported.
  • domain assumption CogVLM recaptioning of MS-COCO is accurate enough to serve as training supervision.
    Used in Section 4.1 for free-form instance captions; errors would propagate into training.
invented entities (1)
  • ROI-Unpool operation
    purpose: Pastes refined ROI features back to their original positions on the high-resolution feature map, enabling explicit ROI injection in generation.
    It is a new operation introduced by the paper; evidence is internal to the paper's benchmarks and ablations, with no external validation or independent implementation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ROICtrl: Boosting Instance Control for Visual Generation." pith.science (2026). https://pith.science/paper/M7Z3SWKZ

@misc{pith2026241117949,
  author       = {Pith},
  title        = {Pith review of: ROICtrl: Boosting Instance Control for Visual Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M7Z3SWKZ}},
  note         = {Machine review of arXiv:2411.17949}
}
read the original abstract

Natural language often struggles to accurately associate positional and attribute information with multiple instances, which limits current text-based visual generation models to simpler compositions featuring only a few dominant instances. To address this limitation, this work enhances diffusion models by introducing regional instance control, where each instance is governed by a bounding box paired with a free-form caption. Previous methods in this area typically rely on implicit position encoding or explicit attention masks to separate regions of interest (ROIs), resulting in either inaccurate coordinate injection or large computational overhead. Inspired by ROI-Align in object detection, we introduce a complementary operation called ROI-Unpool. Together, ROI-Align and ROI-Unpool enable explicit, efficient, and accurate ROI manipulation on high-resolution feature maps for visual generation. Building on ROI-Unpool, we propose ROICtrl, an adapter for pretrained diffusion models that enables precise regional instance control. ROICtrl is compatible with community-finetuned diffusion models, as well as with existing spatial-based add-ons (\eg, ControlNet, T2I-Adapter) and embedding-based add-ons (\eg, IP-Adapter, ED-LoRA), extending their applications to multi-instance generation. Experiments show that ROICtrl achieves superior performance in regional instance control while significantly reducing computational costs.

Figures

Figures reproduced from arXiv: 2411.17949 by the authors.

Figure 1
Figure 1. Grid test for instance control. (a) We structure the region [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Applications of ROICtrl. A trained ROICtrl adapter can extend existing diffusion models (a) and their community-finetuned [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Illustration of different ROI injection designs. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Illustration of ROI-Unpool. The dashed grid represents [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Detailed structure of ROICtrl. In parallel with the pretrained global caption injection, we introduce an additional instance caption [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Qualitative comparison on ROICtrl-Bench. Track 1 and 2 examine template-based instance caption, while track 3 and 4 evaluate [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Ablation study comparing ROICtrl and embedding-based injection (GLIGEN*). ROICtrl achieves faster convergence, improved [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 9
Figure 9. Figure 9: Comparison of regional and global coordinate condition [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Limitation of ROICtrl. ROICtrl prioritizes the use of [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Qualitative comparison of ROICtrl and previous methods on Instdiff-Bench [ [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Applications of ROICtrl on Video Instance Control. We encourage readers to [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FICGen: Frequency-Inspired Contextual Disentanglement for Layout-driven Degraded Image Generation

    cs.CV 2025-09 conditional novelty 6.0 of 10

    A frequency-guided layout-to-image generation framework, FICGen, improves fidelity, layout alignment, and detector trainability on degraded scenes across five benchmarks.

  2. AnimeShooter: A Multi-Shot Animation Dataset for Reference-Guided Video Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    AnimeShooter provides hierarchical story and shot annotations plus reference images for 148K one-minute animation stories, and AnimeShooterGen trained on it shows improved cross-shot consistency.

Reference graph

Works this paper leans on

51 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Improving image generation with better captions

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023. 1, 7

    work page 2023

  3. [3]

    Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models

    Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1–10, 2023. 3

    2023

  4. [4]

    Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffu- sion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7310– 7320, 2024. 12

    work page 2024

  5. [5]

    Training-free layout control with cross-attention guidance

    Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5343–5353, 2024. 3

    2024

  6. [6]

    Yolo-world: Real-time open-vocabulary object detection

    Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xing- gang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901–16911, 2024. 6, 7

    work page 2024

  7. [7]

    Emu: Enhanc- ing image generation models using photogenic needles in a haystack

    Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xi- aofang Wang, Abhimanyu Dubey, et al. Emu: Enhanc- ing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309.15807, 2023. 1

    arXiv 2023

  8. [8]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009. 2, 4

    work page 2009

Show all 51 references
  1. [9]

    Scaling recti- fied flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas M ¨uller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling recti- fied flow transformers for high-resolution image synthesis. In Forty-first International Conference on Mach...

    2024

  2. [10]

    Muffin or chihuahua? challenging multimodal large lan- guage models with multipanel vqa

    Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching-Chen Kuo, Yang Zhao, Xinze Guan, and Xin Wang. Muffin or chihuahua? challenging multimodal large lan- guage models with multipanel vqa. In Proceedings of the 62nd Annual Meeting of the Association for Computational L...

  3. [11]

    Training-free structured diffusion guidance for compositional text-to-image synthesis

    Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032, 2022. 1

    2022 arXiv

  4. [12]

    Ranni: Taming text-to-image diffu- sion for accurate instruction following

    Yutong Feng, Biao Gong, Di Chen, Yujun Shen, Yu Liu, and Jingren Zhou. Ranni: Taming text-to-image diffu- sion for accurate instruction following. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4744–4753, 2024. 2, 4

    2024

  5. [13]

    Emu video: Factoriz- ing text-to-video generation by explicit image conditioning

    Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Du- val, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factoriz- ing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709, 2023. 1

    2023 arXiv

  6. [14]

    Mix-of-show: Decentralized low- rank adaptation for multi-concept customization of diffusion models

    Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yun- peng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low- rank adaptation for multi-concept customization of diffusion models. Advances in Neural Information Processing ...

    2024

  7. [15]

    Videoswap: Customized video subject swapping with interactive semantic point cor- respondence

    Yuchao Gu, Yipin Zhou, Bichen Wu, Licheng Yu, Jia-Wei Liu, Rui Zhao, Jay Zhangjie Wu, David Junhao Zhang, Mike Zheng Shou, and Kevin Tang. Videoswap: Customized video subject swapping with interactive semantic point cor- respondence. In Proceedings of the IEEE/CVF Conference o...

    2024

  8. [16]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 2, 4

    2016

  9. [17]

    Mask r-cnn

    Kaiming He, Georgia Gkioxari, Piotr Doll ´ar, and Ross Gir- shick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017. 2, 3, 4, 5

    2017

  10. [18]

    Lo- calized text-to-image generation for free via cross attention control

    Yutong He, Ruslan Salakhutdinov, and J Zico Kolter. Lo- calized text-to-image generation for free via cross attention control. arXiv preprint arXiv:2306.14636, 2023. 3

    2023 arXiv

  11. [19]

    Prompt-to-prompt im- 9 age editing with cross attention control

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt im- 9 age editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022. 3

    2022 arXiv

  12. [20]

    Imagen video: High definition video generation with diffusion mod- els

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 1

    arXiv

  13. [21]

    YOLO by Ultralytics, 2023

    Glenn Jocher, Ayush Chaurasia, and Jing Qiu. YOLO by Ultralytics, 2023. 6

    2023

  14. [22]

    Dense text-to-image generation with attention modulation

    Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7701–7711, 2023. 3

    2023

  15. [23]

    Imagenet classification with deep convolutional neural net- works

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural net- works. Advances in neural information processing systems , 25, 2012. 2, 4

    2012

  16. [24]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511–22521, 2023. 2,...

    2023

  17. [25]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  18. [26]

    Feature pyra- mid networks for object detection

    Tsung-Yi Lin, Piotr Doll ´ar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyra- mid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2117–2125, 2017. 4

    2017

  19. [27]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 6

    2023 arXiv

  20. [28]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 4296–4304, 20...

    2024

  21. [29]

    Compositional text-to-image generation with dense blob representations

    Weili Nie, Sifei Liu, Morteza Mardani, Chao Liu, Benjamin Eckart, and Arash Vahdat. Compositional text-to-image generation with dense blob representations. arXiv preprint arXiv:2405.08246, 2024. 4, 8

    2024 arXiv

  22. [30]

    Pytorch: An im- perative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems ...

    2019

  23. [31]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 4195–4205,

  24. [32]

    Grounded text-to-image synthesis with attention refocusing

    Quynh Phung, Songwei Ge, and Jia-Bin Huang. Grounded text-to-image synthesis with attention refocusing. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7932–7942, 2024. 3

    2024

  25. [33]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  26. [34]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information process- ing systems, 28, 2015. 2, 4

    2015

  27. [35]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  28. [36]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information...

    2022

  29. [37]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 7

    2010 arXiv

  30. [38]

    Anti-dreambooth: Pro- tecting users from personalized text-to-image synthesis

    Thanh Van Le, Hao Phung, Thuan Hoang Nguyen, Quan Dao, Ngoc N Tran, and Anh Tran. Anti-dreambooth: Pro- tecting users from personalized text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2116–2127, 2023. 13

    2023

  31. [39]

    Boximator: Generat- ing rich and controllable motions for video synthesis

    Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guo- qiang Wei, Liping Yuan, and Hang Li. Boximator: Generat- ing rich and controllable motions for video synthesis. arXiv preprint arXiv:2402.01566, 2024. 2, 4

    2024 arXiv

  32. [40]

    Instancediffusion: Instance- level control for image generation

    Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Ro- hit Girdhar, and Ishan Misra. Instancediffusion: Instance- level control for image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6232–6242, 2024. 3, 4, 5, 6, 7, 8, 12, 13

    2024

  33. [41]

    Motionctrl: A unified and flexible motion controller for video generation

    Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Pa- pers, pages 1–11, 2024. 3

    2024

  34. [42]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

    2023

  35. [43]

    Boxdiff: Text-to-image synthesis with training-free box-constrained 10 diffusion

    Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wen- tian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free box-constrained 10 diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7452–7461, 2023. 3

    2023

  36. [44]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. 6, 7

    2024 arXiv

  37. [45]

    Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip- adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.06721,

    arXiv

  38. [46]

    Show-1: Marrying pixel and latent dif- fusion models for text-to-video generation

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent dif- fusion models for text-to-video generation. arXiv preprint arXiv:2309.15818, 2023. 1

    2023 arXiv

  39. [47]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 3, 4, 6

    2023

  40. [48]

    Migc: Multi-instance generation controller for text-to-image synthesis

    Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, and Yi Yang. Migc: Multi-instance generation controller for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6818– 6828, 2024. 3, 4, 5, 6, 7, 8, 12, 13 11 Global Ca...

    2024

  41. [49]

    ROICtrl-Bench ROICtrl-Bench contains 200 samples, divided into groups {1, 2, 3, 4, 5, 6-10, 11-15, 16-20, 21-25, 26-30} based on in- stance counts

    Detailed Evaluation Settings 6.1. ROICtrl-Bench ROICtrl-Bench contains 200 samples, divided into groups {1, 2, 3, 4, 5, 6-10, 11-15, 16-20, 21-25, 26-30} based on in- stance counts. Each group includes 20 examples randomly selected from the MS-COCO 2017 evaluation set [25]. Ha...

    2017

  42. [50]

    Qualitative Comparison We have demonstrated the qualitative comparison on ROICtrl-Bench in Sec

    Additional Experiments 7.1. Qualitative Comparison We have demonstrated the qualitative comparison on ROICtrl-Bench in Sec. 4.3 of the main paper. Therefore, in this section, we primarily present the qualitative compar- ison on InstDiff-Bench [40] and MIG-Bench [48]. Small-Siz...

  43. [51]

    Limitation Analysis The attribution leakage problem is largely addressed in ROICtrl, as we prioritize using instance captions in the learnable blending process

    Limitation and Future Works 8.1. Limitation Analysis The attribution leakage problem is largely addressed in ROICtrl, as we prioritize using instance captions in the learnable blending process. However, generating the same instance for highly overlapping bounding boxes remains...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.