Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketches

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SketchFlex claims rough region sketches plus auto-refined prompts and shape anchors produce more cohesive, intention-aligned images than text-to-image or region-to-image baselines.

desk verdict A plausible systems contribution with a real integration story, but the evaluation confounds its own components and the cognitive-load claim needs a direct measure. read the letter →

arxiv 2502.07556 v1 pith:OKO3HMQ6 submitted 2025-02-11 cs.HC cs.CV

Haichuan Lin , Yilin Ye , Jiazhi Xia , Wei Zeng This is my paper · ORCID
classification cs.HCcs.CV
keywords text-to-imagegenerationregion-basedsketchcontrolspatial-semanticcoherencepromptrecommendationdiffusionmodelsinteractiveimagedecompose-and-recomposehuman-AIco-creativetools
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces SketchFlex, an interactive system that lets novice users specify an image by drawing rough colored region sketches and typing a few words, then automatically fills in detailed, spatially coherent prompts and refines the rough shapes into precise anchors. The central claim is that this two-step intervention, semantic prompt completion plus shape refinement, makes generated images more cohesive and better matched to user intent than either pure text-to-image generation or standard region-based generation, while lowering the mental effort of prompt crafting. If true, it gives non-experts a practical way to control object placement, shape, and relationships in diffusion-model image generation without learning depth maps, canny references, or prompt-engineering skills. The paper reports a 12-participant study in which SketchFlex achieved higher IoU alignment (0.535 overall, versus 0.314 for text-to-image and 0.418 for region-to-image) and significantly higher subjective ratings on image quality, cohesiveness, and intention alignment.

What carries the argument

The load-bearing mechanism is semantic-space-guided prompt completion coupled with decompose-and-recompose shape anchoring. The semantic space is a structured vocabulary covering type, attribute, state, direction, relationship, lighting, and style, used to organize per-region prompts; a multimodal large language model fills it from the sketch and initial prompt, while retrieval from crowd-sourced image-annotation datasets supplies realistic attribute and relationship references. On the shape side, the rough sketch is decomposed into individual object regions, candidate refined shapes are generated for each object, filtered by IoU and CLIP score, and presented to the user for selection and spatial adjustment. The selected masks are recomposed into a joint mask, and the canny edges of that anchor are fed into a ControlNet-conditioned diffusion model, with a cross-attention update strengthening relationship prompts inside joint masks and suppressing them outside. This decompose-and-recompose loop is what converts rough scribbles into precise anchors while keeping iterative refinement local to individual objects.

What would settle it

A controlled ablation that gives the multimodal language model only the rough sketch and the semantic-space template, without the retrieved attribute and relationship examples, would settle whether the crowd-sourced data carries the coherence gain; the paper's own Figure 5 ablation compares full SketchFlex against no-prompt-recommendation and no-sketch-refinement conditions, but not retrieval alone. A second check would count missing-object and wrong-relationship errors on a fixed sketch set where thin strokes and inclusion relationships are deliberately included, testing whether the decompose-and-recompose shape anchoring can handle those cases even with perfect prompts.

Watch

Extended reading notes

Core claim

SketchFlex's core claim is that the two bottlenecks for novice text-to-image users, writing prompts that specify relationships between multiple objects and preparing precise spatial conditions, can be removed by a pipeline that first uses a multimodal large language model, guided by a curated semantic space and retrieved crowd-sourced object attributes and relationships, to turn a rough color-coded sketch and minimal prompt into complete region prompts. The pipeline then refines each foreground object through a decompose-and-recompose strategy: generate candidate single-object images, filter them by IoU and CLIP score, let the user select and adjust masks, and recompose the selected shapes into a joint mask whose canny edges condition the diffusion model. The paper argues that the resulting images are more cohesive and more aligned with user intentions than end-to-end text-to-image and region-based baselines, with the user study reporting IoU of 0.535 versus 0.314 and 0.418 respectively, and significantly higher subjective ratings on all three outcome dimensions.

Load-bearing premise

The whole pipeline depends on the multimodal language model correctly reading a color-coded rough sketch, and on the retrieved real-world attributes and relationships being relevant priors, so if the model misreads the sketch or the dataset examples are off, the recommended prompts can insert wrong objects or unnatural relationships instead of fixing them.

Editorial extensions

If this is right

  • Novice users can produce images with specific object arrangement and interaction without learning depth maps, canny references, or segmentation controls.
  • Iterative editing becomes local: changing one object's shape or position need not regenerate the whole composition.
  • Prompt recommendation reduces the cognitive load of per-region prompt writing, since users can rely on auto-filled attributes, states, and relationships.
  • The same backbone model can serve text-to-image, region-to-image, and SketchFlex conditions, isolating the effect of the interaction design from model quality.
  • The performance gap narrows on complex multi-object scenes, with Task 2 IoU of 0.456 versus 0.396 and a non-significant p-value of 0.19, indicating limits with three or more small objects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The pipeline's ceiling likely tracks the spatial reasoning ability of the underlying multimodal language model, so swapping in open-weight models would test whether the semantic-space scaffolding carries the benefit or whether the LLM's sketch reading is essential.
  • The decompose-and-recompose shape-anchoring step could be reused as a general front-end for other spatial conditioning backbones, not only canny-based ControlNet.
  • The failure cases named in Section 6.6, thin strokes, inclusion relationships, and three or more similar objects, imply the system's benefit is strongest for clear, well-separated objects; extending to overlapping or contained objects likely needs layer or depth control rather than more prompt tuning.
  • A larger user study with fixed layout constraints rather than open reference images could separate tool capability from user skill, since the reported variability in Task 2 suggests subjective alignment choices partly drive the IoU differences.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SketchFlex is an interactive system for rough-sketch region-based text-to-image generation. It combines sketch-aware prompt recommendation, in which GPT-4o completes a semantic space populated with attributes and relationships from Visual Genome and VAW, with a decompose-and-recompose sketch refinement pipeline that generates per-object shape candidates, lets users adjust their masks, and anchors the final generation through Canny/ControlNet. The paper reports a 12-participant user study comparing SketchFlex with a text-to-image baseline and a region-to-image baseline on closed-ended tasks (IoU and subjective ratings) and an open-ended creative task. The authors claim that SketchFlex produces more cohesive images, better matches user intentions, and significantly reduces cognitive load relative to the region-based baseline.

Significance. If the reported results hold, SketchFlex provides a practical and novel interaction pathway for novice users to control object placement, shape, and inter-object relationships in diffusion-based image generation. The release of code and data is a concrete strength, and the semantic-space formulation is a reasonable way to connect rough sketches to coherent prompts. The core ideas are plausible and the qualitative examples are compelling. The significance is currently conditional, however, because the evaluation does not isolate the contribution of the sketch-aware prompt recommendation, the harder-task IoU advantage is not statistically significant, and the abstract's cognitive-load claim is not supported by any direct workload measurement.

major comments (3)
  1. [§5.1.3, §5.2, §5.4.2] The R2I baseline is not an independent system: it is the same SketchFlex application with the advanced features disabled. The close-ended IoU task additionally rewards the manual single-object adjustment stage, because participants can move and resize masks to match the reference image, and this can improve IoU even if the prompt recommendation contributes nothing. The paper's named central contribution, sketch-aware prompt recommendation, is never evaluated in isolation: the ablations in Figs. 5 and 11 are qualitative only, with no metrics or significance tests. Please add a condition that isolates prompt recommendation from the manual shape-adjustment controls, or report quantitative ablations on the same backbone and tasks.
  2. [§5.2, Table 3] On the harder, more representative Task 2, SketchFlex's IoU (0.456) is not significantly better than the R2I baseline (0.396, p = 0.19). The overall headline advantage therefore rests mainly on Task 1, where the fixed two-object setup is easier and where manual mask adjustment can trivially improve IoU. Report effect sizes and confidence intervals for each pairwise comparison, and discuss the statistical power of the n = 12 design when making broad claims about superiority.
  3. [Abstract, §5.4.1] The abstract's claim that SketchFlex "significantly reduc[es] cognitive load" is not supported by the reported measurements. Section 5 contains no NASA-TLX or other validated workload instrument; the evidence consists of feature-specific Likert items and interview comments about time savings, which conflate perceived helpfulness with cognitive workload. Either add a direct workload measure or revise the abstract and conclusions to remove the cognitive-load claim.
minor comments (5)
  1. [§2.3] There is a typo in "reseach" in the opening sentence of Section 2.3.
  2. [§5.1.1] The participant experience description is internally inconsistent: it says 2 participants had more than one year of experience and 1 participant had over a year of experience. Please clarify the intended categories.
  3. [§5.1.4, §5.2] Section 5.1.4 describes Task 2 as involving four main objects, while Section 5.2 says Task 2 features three smaller objects. Please reconcile the object count.
  4. [§5.4.2, Figure 11] The feature-influence examples in Figure 11 are informative, but the figure would benefit from explicit labels indicating which row corresponds to which task and which changes were made between columns; this would make the qualitative ablation easier to interpret.
  5. [§6.6] The acknowledged failure cases (three or more similar objects, thin strokes, inclusion relationships) directly bound the central claim about multi-object coherence; please state in the Discussion how these limitations affect the scope of the claimed advantage.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the system is evaluated empirically against external baselines and no claimed prediction reduces to a fitted input.

full rationale

SketchFlex is an interactive system paper, not a derivation of predictive quantities from fitted parameters. The central claims are empirical: SketchFlex produces more cohesive images and better matches user intentions than two baselines, as measured in a 12-participant study (Sec. 5). The quantitative IoU scores in Table 3 are computed against reference images using SAM and DINO segmentation, which are external to the system and not used to fit any SketchFlex component. The R2I baseline is described in Sec. 5.1.3 as the same web application with SketchFlex's advanced features disabled ("in the R2I generation condition, the advanced features of SketchFlex are disabled"). This is a standard ablation-style comparison rather than a circular construction: the outcome metric still measures alignment to an independent reference image, and the disabled features are the system's own interventions, not the metric's definition. The prompt recommendation module (Sec. 4.1) uses GPT-4o plus retrieved attributes from Visual Genome and VAW; the sketch refinement module (Sec. 4.2) uses decomposition, FAST SAM, and Canny/ControlNet. None of these are defined in terms of the study's outcome measures, and no equation in the paper reduces to a fitted parameter or to the evaluation metric. The qualitative ablations in Figs. 5 and 11 lack statistical tests, and the abstract's "significantly reducing cognitive load" claim relies on informal participant comments rather than a validated instrument; these are evaluation-strength limitations, not circularity. Some references are to prior work by the same authors (e.g., PlantoGraphy, IntentTuner, and the chart-embedding paper), but they appear only in related-work and motivation contexts and are not load-bearing for the paper's reported results. No self-citation chain is used to justify the system's effectiveness. Therefore the central claim is self-contained against external comparison, and no circular step can be exhibited.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central mechanism rests on domain assumptions about user input, dataset priors, MLLM reliability, and shape anchoring rather than on mathematical axioms. No invented physical entities are introduced. Hand-tuned parameters include lambda_mi, the IoU/CLIP filtering weights, the k=10 retrieval sample count, and the top-four candidate list; none are fitted to an external benchmark.

free parameters (4)
  • mask amplification scalar lambda_mi = not reported
    Equations 2 and 3 use lambda_mi to scale text embeddings inside masked regions; the value is hand-chosen and not given, yet it controls the strength of shape anchoring and negative-prompt exclusion.
  • IoU/CLIP weighted-sum coefficients for single-object filtering = not reported
    Section 4.2.1 selects the top four generated single-object images by a weighted sum of IoU and CLIP scores; the weights are not specified and directly determine which shape candidates users can choose.
  • k (number of retrieved semantic examples) = 10
    Section 4.1.2 randomly samples k=10 attribute/relationship examples from Visual Genome/VAW as MLLM references; this choice affects prompt diversity and coherence.
  • candidate count for single-object generation = 4
    Section 4.2.1 shows users the top four generated images per object; this ceiling shapes the interaction and the achievable alignment.
assumptions (5)
  • domain assumption Rough region sketches are a sufficient and preferable spatial input for novice users.
    Formative study with 8 users (Sec 3.2.1) is interpreted as establishing this; the paper does not test alternative input modalities.
  • domain assumption Visual Genome and VAW crowd-sourced attributes and relationships are representative priors for coherent image generation.
    Sec 4.1.2 uses them as reference data for MLLM prompt completion without evaluating their coverage or noise.
  • domain assumption GPT-4o can perform reliable chain-of-thought spatial reasoning over color-mask sketches and produce useful region prompts.
    Sec 4.1.2 and appendix prompt; no systematic evaluation of MLLM failure rate is reported.
  • domain assumption Canny edge extraction plus ControlNet shape anchoring preserves user-selected object shapes in the final image.
    Sec 4.2.2; contradicted by failure cases in Sec 6.6 (thin strokes, missing objects, inclusion relationships).
  • domain assumption Dense Diffusion is a representative state-of-the-art region-based baseline for this comparison.
    Sec 5.1.2; the user study does not compare against other region-based methods such as Omost.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketches." pith.science (2026). https://pith.science/paper/OKO3HMQ6

@misc{pith2026250207556,
  author       = {Pith},
  title        = {Pith review of: SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketches},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OKO3HMQ6}},
  note         = {Machine review of arXiv:2502.07556}
}
read the original abstract

Text-to-image models can generate visually appealing images from text descriptions. Efforts have been devoted to improving model controls with prompt tuning and spatial conditioning. However, our formative study highlights the challenges for non-expert users in crafting appropriate prompts and specifying fine-grained spatial conditions (e.g., depth or canny references) to generate semantically cohesive images, especially when multiple objects are involved. In response, we introduce SketchFlex, an interactive system designed to improve the flexibility of spatially conditioned image generation using rough region sketches. The system automatically infers user prompts with rational descriptions within a semantic space enriched by crowd-sourced object attributes and relationships. Additionally, SketchFlex refines users' rough sketches into canny-based shape anchors, ensuring the generation quality and alignment of user intentions. Experimental results demonstrate that SketchFlex achieves more cohesive image generations than end-to-end models, meanwhile significantly reducing cognitive load and better matching user intentions compared to region-based generation baseline.

Figures

Figures reproduced from arXiv: 2502.07556 by the authors.

Figure 1
Figure 1. Failure cases of existing methods for rough sketch based image generation: a) missing object for the green sketch, b) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. SketchFlex mainly consists of three components: (1) sketch-aware prompt recommendation that support users in crafting effective prompts for the rough sketch; (2) object shape refinement through single object decomposition and generation; and (3) spatial adjustment and anchoring of object shapes. with these techniques, it is clear that prompt tuning would be even more difficult for novice users. 3.2.3 Iterative gener… view at source ↗
Figure 3
Figure 3. Our sketch-aware prompt recommendation first builds a semantic space through data-driven analysis of key semantic [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Spatial-condition sketch refinement can help novice users refine their sketch by generating more realistic and accurate [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Ablation study shows that prompt recommendation avoids common issues like missing objects and unrealistic [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: SketchFlex interface consists of (a) Canvas view, (b) Prompt Recommend view, (c) Sketch Refine view and (d) Result view. the wrong areas, as shown in Equations 2-3. In Equation 2, the cross￾attention map that correlates text and image patches is updated such that the t…
Figure 7
Figure 7. Figure 7: Outcome satisfaction survey indicates that [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Outcome examples of Task 1 and Task2 show that while region-based generation offers better spatial control than [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Outcome examples of open-ended task show that users can freely create sophisticated images with varying degree of [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: User ratings of main features of SketchFlex, including Prompt Recommendation, Single Object Decomposition and Single Object Adjustment. making the outcomes much more aligned with the reference images. P1 remarked, "I can choose the desired object shape and fix it in t…
Figure 11
Figure 11. Figure 11: Examples showing single feature influence on the results, including Prompt Recommendation, Single Object [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Overall usability ratings of system compared with two baselines. [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: The comparison between novice and expert participants in painting reveals that experts produce more accurate and [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Design

    cs.HC 2025-08 conditional novelty 5.0 of 10

    GenTune improves AI image refinement by tracing image regions back to prompt labels and allowing element-level, semantic-guided edits.

Reference graph

Works this paper leans on

73 extracted references · 48 canonical work pages · cited by 1 Pith paper

  1. [1]

    2005. Krita. https://krita.org/

    work page 2005

  2. [2]

    2022. Civitai. https:https://civitai.com. https:https://civitai.com

    work page 2022

  3. [3]

    Adobe Firefly

    2023. Adobe Firefly. https://www.adobe.com/uk/products/firefly.html

    work page 2023

  4. [4]

    colorful-realistic-xl-v1-sdxl

    2023. colorful-realistic-xl-v1-sdxl. https://huggingface.co/John6666/colorful- realistic-xl-v1-sdxl

    work page 2023

  5. [5]

    Midjourney

    2023. Midjourney. https://www.midjourney.com/

    work page 2023

  6. [6]

    Stable Diffusion

    2023. Stable Diffusion. https://github.com/AUTOMATIC1111/stable-diffusion- webui

    work page 2023

  7. [7]

    InternVL2-Llama3-76B

    2024. InternVL2-Llama3-76B. https://huggingface.co/OpenGVLab/InternVL2- Llama3-76B/

    work page 2024

  8. [8]

    Qwen2-VL

    2024. Qwen2-VL. https://github.com/QwenLM/Qwen2-VL/

    work page 2024

Show all 73 references
  1. [9]

    Elad Aharoni-Mack, Yakov Shambik, and Dani Lischinski. 2017. Pigment-based recoloring of watercolor paintings. In Proceedings of the Symposium on Non- Photorealistic Animation and Rendering. 1–11. https://doi.org/10.1145/3092919. 3092926

    2017 doi

  2. [10]

    Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. 2023. MultiDiffusion: fusing diffusion paths for controlled image generation. In Proceedings of the International Conference on Machine Learning . Article 74, 16 pages

    2023

  3. [11]

    Josiah D Boucher, Gillian Smith, and Yunus Doğan Telliel. 2024. Is Resistance Futile?: Early Career Game Developers, Generative AI, and Ethical Skepticism. In Proc. ACM CHI. Article 173, 13 pages. https://doi.org/10.1145/3613904.3641889

    2024

  4. [12]

    Stephen Brade, Bryan Wang, Mauricio Sousa, Sageev Oore, and Tovi Grossman

  5. [13]

    Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. 2018. Coco-stuff: Thing and stuff classes in context. In Proc. CVPR. 1209–1218

    2018

  6. [14]

    J Canny. 1986. A Computational Approach to Edge Detection.IEEE Trans. Pattern Anal. Mach. Intell. (1986), 679–698. https://doi.org/10.1109/TPAMI.1986.4767851

    1986

  7. [15]

    Tingfeng Cao, Chengyu Wang, Bingyan Liu, Ziheng Wu, Jinhui Zhu, and Jun Huang. 2023. Beautifulprompt: Towards automatic prompt engineering for text- to-image synthesis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing: Industry Track . 1–11

    2023

  8. [16]

    Minghao Chen, Iro Laina, and Andrea Vedaldi. 2024. Training-free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision . 5343–5353

    2024

  9. [18]

    John Joon Young Chung and Eytan Adar. 2023. PromptPaint: Steering Text-to- Image Generation Through Paint Medium-like Interactions. In Proc. ACM UIST. Article 6, 17 pages. https://doi.org/10.1145/3586183.3606777

    2023

  10. [20]

    Nicholas Davis. 2015. An Enactive Approach to Facilitate Interactive Machine Learning for Co-Creative Agents. In Proceedings of the ACM SIGCHI Conference on Creativity and Cognition . 345–346. https://doi.org/10.1145/2757226.2764773

    2015

  11. [21]

    Yuki Endo. 2023. Masked-attention diffusion guidance for spatially controlling text-to-image generation. The Visual Computer (2023), 6033–6045. https://doi. org/10.1007/s00371-023-03151-y

    2023 doi

  12. [22]

    Yingchaojie Feng, Xingbo Wang, Kam Kwai Wong, Sijia Wang, Yuhong Lu, Min- feng Zhu, Baicheng Wang, and Wei Chen. 2023. PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation. IEEE Trans. Vis. Comput. Graph. (2023), 295–305. https://doi.org/10.1109/TVCG.2023.3327168

    2023

  13. [23]

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2023. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532 (2023)

    2023 arXiv

  14. [24]

    Yuhan Guo, Hanning Shao, Can Liu, Kai Xu, and Xiaoru Yuan. 2024. PrompTHis: Visualizing the Process and Influence of Prompt Editing during Text-to-Image Creation. IEEE Trans. Vis. Comput. Graph. (2024), 1–12

    2024

  15. [25]

    Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. 2024. Optimizing prompts for text-to-image generation. In Proceedings of the International Conference on Neural Information Processing Systems. Article 2923, 17 pages. https://doi.org/10.5555/ 3666122.3669045

    2024

  16. [26]

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. In Proc. EMNLP. 7514–7528. https://doi.org/10.18653/v1/2021.emnlp-main.595

    2021 doi

  17. [28]

    James Hutson and Peter Cotroneo. 2023. Generative AI tools in art education: Exploring prompt engineering and iterative processes for enhanced creativity. Metaverse 4, 1 (2023), 1–14. https://doi.org/10.54517/m.v4i1.2164

    2023 doi

  18. [29]

    Jiang, Lauren Brown, Jessica Cheng, Mehtab Khan, Abhishek Gupta, Deja Workman, Alex Hanna, Johnathan Flowers, and Timnit Gebru

    Harry H. Jiang, Lauren Brown, Jessica Cheng, Mehtab Khan, Abhishek Gupta, Deja Workman, Alex Hanna, Johnathan Flowers, and Timnit Gebru. 2023. AI Art and its Impact on Artists. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. 363–374. https://doi.org/...

    2023

  19. [30]

    Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu. 2023. Dense text-to-image generation with attention modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 7701–7711

    2023

  20. [31]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment Anything. arXiv:2304.02643 (2023)

    2023 arXiv

  21. [32]

    Hyung-Kwon Ko, Gwanmo Park, Hyeon Jeon, Jaemin Jo, Juho Kim, and Jinwook Seo. 2023. Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works. In Proceedings of the International Conference on Intelligent User Interfaces. 919–933. https://doi.org/10.1145/3...

    2023

  22. [33]

    Akio Kodaira, Chenfeng Xu, Toshiki Hazama, Takanori Yoshimoto, Kohei Ohno, Shogo Mitsuhori, Soichi Sugano, Hanying Cho, Zhijian Liu, and Kurt Keutzer

  23. [34]

    Subhadeep Koley, Ayan Kumar Bhunia, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, and Yi-Zhe Song. 2023. Picture that sketch: Photorealistic image generation from abstract sketches. In Proc. CVPR. 6850–6861

    2023

  24. [35]

    arXiv preprint arXiv:2312.12491 (2023)

    Streamdiffusion: A pipeline-level solution for real-time interactive genera- tion. arXiv preprint arXiv:2312.12491 (2023)

    2023 arXiv

  25. [36]

    Katrin Lasinger, René Ranftl, Konrad Schindler, and Vladlen Koltun. 2019. To- wards robust monocular depth estimation: Mixing datasets for zero-shot cross- dataset transfer. arXiv preprint arXiv:1907.01341 (2019)

    2019 arXiv

  26. [37]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotatio...

    2017 doi

  27. [38]

    Jaerin Lee, Daniel Sungho Jung, Kanggeon Lee, and Kyoung Mu Lee. 2024. Stream- MultiDiffusion: Real-Time Interactive Generation with Region-Based Semantic Control. arXiv preprint arXiv:2403.09055 (2024)

    2024 arXiv

  28. [39]

    Tomas Lawton, Kazjon Grace, and Francisco J Ibarrola. 2023. When is a Tool a Tool? User Perceptions of System Agency in Human–AI Co-Creative Drawing. In Proceedings of the 2023 ACM Designing Interactive Systems Conference . 1978–1996. https://doi.org/10.1145/3563657.3595977

    2023

  29. [40]

    Zejian Li, Ying Zhang, Shengzhe Zhou, Qi Liu, Jiesi Zhang, Haoran Xu, Shuyao Chen, Xiaoyu Chen, and Lingyun Sun. 2024. RealtimeGen: An Intervenable AI Image Generation System for Commercial Digital Art Asset Creators. Interna- tional Journal of Human–Computer Interaction (2024...

    2024

  30. [41]

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. 2023. Gligen: Open-set grounded text-to-image generation. In Proc. CVPR. 22511–22521

    2023

  31. [42]

    Vivian Liu and Lydia B Chilton. 2022. Design guidelines for prompt engineering text-to-image generative models. In Proc. ACM CHI. Article 384, 23 pages. https: //doi.org/10.1145/3491102.3501825 SketchFlex CHI ’25, April 26-May 1, 2025, Yokohama, Japan

    2022

  32. [43]

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al . 2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023)

    2023 arXiv

  33. [44]

    Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. 2023. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378 (2023)

    2023 arXiv

  34. [45]

    Vivian Liu, Jo Vermeulen, George Fitzmaurice, and Justin Matejka. 2023. 3DALL- E: Integrating text-to-image AI in 3D design workflows. In Proceedings of the ACM designing interactive systems conference. 1955–1977. https://doi.org/10.1145/ 3563657.3596098

    2023

  35. [46]

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. 2023. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08453 (2023)

    2023 arXiv

  36. [47]

    Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. 2023. Lcm-lora: A universal stable-diffusion acceleration module. arXiv preprint arXiv:2311.05556 (2023)

    2023 arXiv

  37. [48]

    Jonas Oppenlaender. 2023. A taxonomy of prompt modifiers for text-to-image generation. Behaviour & Information Technology (2023), 1–14. https://doi.org/10. 1080/0144929X.2023.2286532

    2023

  38. [49]

    Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, et al. 2024. DOCCI: Descriptions of Connected and Contrasting Images. arXiv preprint arXiv:2404.19753 (2024)

    2024 arXiv

  39. [50]

    Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. 2021. Learning to predict visual attributes in the wild. In Proc. CVPR. 13018–13028

    2021

  40. [51]

    Jonas Oppenlaender, Rhema Linder, and Johanna Silvennoinen. 2023. Prompting ai art: An investigation into the creative skill of prompt engineering. arXiv preprint arXiv:2303.13534 (2023)

    2023 arXiv

  41. [52]

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen

  42. [53]

    Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng

  43. [54]

    arXiv preprint arXiv:2305.03495 (2023)

    Automatic prompt optimization with gradient descent and beam search. arXiv preprint arXiv:2305.03495 (2023)

    2023 arXiv

  44. [55]

    Yang Shi, Tian Gao, Xiaohan Jiao, and Nan Cao. 2023. Understanding design collaboration between designers and artificial intelligence: a systematic literature review. Proceedings of the ACM on Human-Computer Interaction 7, Article 368 (2023), 35 pages. https://doi.org/10.1145/3610217

    2023 doi

  45. [56]

    Omost Team. 2024. Omost GitHub Page. https://github.com/lllyasviel/Omost

    2024

  46. [57]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proc. CVPR. 10684–10695

    2022

  47. [58]

    Sara Mah- davi, Raphael Gontijo-Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Lit, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mah- davi, Raphael Gontijo-Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. 2024. Photorealistic text-to-image di...

    2024

  48. [59]

    Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. 2024. Instancediffusion: Instance-level control for image generation. In Proc. CVPR. 6232–6242

    2024

  49. [60]

    Xinyi Wang, Wanrong Zhu, and William Yang Wang. 2023. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. arXiv preprint arXiv:2301.11916 (2023)

    2023 arXiv

  50. [61]

    An Adapt-or-Die Type of Situation

    Veera Vimpari, Annakaisa Kultima, Perttu Hämäläinen, and Christian Guck- elsberger. 2023. “An Adapt-or-Die Type of Situation”: Perception, Adoption, and Use of Text-to-Image-Generation AI by Game Industry Professionals. Proc. ACM Hum.-Comput. Interact. 7, CHI PLAY, Article 379...

    2023 doi

  51. [62]

    Ruochen Wang, Ting Liu, Cho-Jui Hsieh, and Boqing Gong. 2024. On Discrete Prompt Optimization for Diffusion Models. InInternational Conference on Machine Learning

    2024

  52. [63]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2024. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the International Conference on Neural Information Processing Systems

    2024

  53. [64]

    Shishi Xiao, Suizi Huang, Yue Lin, Yilin Ye, and Wei Zeng. 2023. Let the chart spark: Embedding semantic context into chart with text-to-image generative model. IEEE Trans. Vis. Comput. Graph. (2023). https://doi.org/10.1109/TVCG. 2023.3326913

    2023

  54. [65]

    Zhijie Wang, Yuheng Huang, Da Song, Lei Ma, and Tianyi Zhang. 2024. PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement. In Proc. ACM CHI. Article 185, 21 pages. https://doi.org/10.1145/ 3613904.3642803

    2024

  55. [66]

    Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. 2022. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint arXiv:2210.14896 (2022)

    2022 arXiv

  56. [67]

    Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui

  57. [68]

    Xingchen Zeng, Ziyao Gao, Yilin Ye, and Wei Zeng. 2024. IntentTuner: An Interactive Framework for Integrating Human Intentions in Fine-tuning Text- to-Image Generative Models. In Proc. ACM CHI. 7514–7528. https://doi.org/10. 1145/3613904.3642165

    2024

  58. [69]

    Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. 2023. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision . 7452–7461

    2023

  59. [70]

    Yutong Xie, Zhaoying Pan, Jinge Ma, Luo Jie, and Qiaozhu Mei. 2023. A Prompt Log Analysis of Text-to-Image Generation Systems. In Proceedings of the ACM Web Conference 2023. 3892–3902. https://doi.org/10.1145/3543507.3587430

    2023

  60. [71]

    Guangcong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. 2023. Layoutdiffusion: Controllable diffusion model for layout-to-image generation. In Proc. CVPR. 22490–22499

    2023

  61. [72]

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017. Scene parsing through ade20k dataset. In Proc. CVPR. 633–641. A APPENDIX: PROMPT “Here is a sketch of an image.{𝑖𝑛𝑝𝑢𝑡 _𝑐𝑜𝑙𝑜𝑟 _𝑚𝑎𝑠𝑘}, while the rest of the white space is the backgroun...

    2017

  62. [74]

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023. Adding Conditional Control to Text-to-Image Diffusion Models. In IEEE International Conference on Computer Vision (ICCV). 3836–3847

    2023

  63. [75]

    Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. 2023. Fast segment anything. arXiv preprint arXiv:2306.12156 (2023)

    2023 arXiv

  64. [2022]

    arXiv preprint arXiv:2204.06125 (2022)

    Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06125 (2022)

    2022 arXiv

  65. [2024]

    In International Conference on Machine Learning

    Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs. In International Conference on Machine Learning

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.