Pith. sign in

REVIEW 4 major objections 7 minor 62 references

Leveraging Multimodal LLM for Inspirational User Interface Search

T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a zero-shot multimodal large language model can extract designer-relevant semantics directly from mobile UI screenshots, and that a search system built on those semantics significantly outperforms existing UI…

desk verdict A useful, well-evaluated system for semantic UI search, but the headline claim overreaches because the user study compares a rich interface against a bare-bones baseline, so the gains are not clearly attributable to the MLLM semantics themselves. read the letter →

arxiv 2501.17799 v3 pith:QIDSMRHC submitted 2025-01-29 cs.HC cs.IR

Seokhyeon Park , Yumin Song , Soohyun Lee , Jaeyoung Kim , Jinwook Seo This is my paper · ORCID
classification cs.HCcs.IR
keywords UIretrievalsemanticsearchmultimodallargelanguagemodelmobiledesigninspirationalzero-shotclassificationsemanticstools
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a multimodal large language model, given only a mobile UI screenshot, can extract the semantics designers actually care about—target user, screen role, app category, and mood—and that a retrieval system built on those extractions makes inspirational UI search better than pixel- or keyword-based retrieval. The authors derive these semantics from interviews with designers, build the search system S&UI around them, and evaluate both extraction quality and the search experience. Designers currently search inspiration by visual similarity or keywords and lose context, while computational methods often require metadata like view hierarchies. If the claim holds, designers could search large screenshot collections by meaning, with no metadata, and see the extracted semantics as an explanation of why each result was returned.

What carries the argument

The central object is a structured zero-shot extraction pipeline for multimodal large language models: a prompt containing an assistant persona, task instruction, feature list, feature definitions and instructions, and a YAML response format, run through GPT-4o on raw mobile UI screenshots. This pipeline converts pixels into 14 semantics organized into Application, Screen, Composition, and Visual Design levels, producing text that is embedded with the text-embedding-3-large model. Retrieval then computes a weighted cosine similarity between a user's textual semantic query and the per-semantic embeddings in the database, with special queries for finding next and previous screens in a user flow.

What would settle it

A controlled ablation in which the same S&UI interface is run once with MLLM-extracted semantic embeddings and once with GUIClip embeddings, with designers unaware of which backend they are using; if the non-semantic backend receives equal ratings on relevance and usefulness, the central outperformance claim collapses. An independent replication on a new screenshot corpus where the MLLM's top-1 screen-category accuracy does not exceed the CLIP baseline would also refute the extraction claim.

Watch

Extended reading notes

Core claim

The central claim is that GPT-4o, prompted zero-shot with a structured feature list, produces accurate and useful mobile UI semantics from screenshots alone, and the semantic-based retrieval system S&UI built on those semantics outperforms the GUIClip baseline across relevance, diversity, reliability, usefulness, and serendipity in a comparative user study. In computational evaluations, GPT-4o reaches 59.21% top-1 accuracy on screen category classification and 58.33% on app category classification, versus 36.24% and 39.95% for the best CLIP-based baselines, and it generates screen descriptions with higher syntactic and lexical richness than the Screen2Words dataset. Designers rated most extracted semantics above 5 on a 7-point scale for relevance and comprehensiveness, with the highest serendipity for target user, similar app, next screen, and previous screen. The paper concludes that a semantic representation generated directly from images provides a more interpretable and contextually relevant inspirational search experience than embedding-only retrieval.

Load-bearing premise

The comparative user study assumes that the higher ratings for S&UI come from its semantic extraction rather than from the richer query interface it provides, since the baseline GUIClip interface had fewer search affordances.

Editorial extensions

If this is right

  • Designers can search UI inspiration by meaning—target user, screen role, mood—without view hierarchies or manual annotation.
  • The system surfaces the extracted semantics for each result, making retrieval explainable and supporting iterative refinement through an import feature.
  • The released S&UI dataset of MLLM-extracted semantics gives other researchers a resource for UI semantic analysis and retrieval experiments.
  • The combination of semantic extraction with future vision-language models or fine-tuning could improve weaker composition and visual design semantics.
  • The comparative result suggests that text-based semantic representations can beat embedding-only retrieval on designer-centered quality metrics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the comparative study did not ablate the interface, part of the measured advantage may come from S&UI's richer query affordances—dropdowns, weight sliders, detail panels—rather than from the semantic representation alone.
  • The same extraction approach could transfer to web or desktop UI search, and potentially to other visual design domains, if the semantic taxonomy were re-derived for that domain.
  • A testable extension is to feed the extracted semantics of a reference screen into a generative UI model to synthesize new design alternatives with the same target user, mood, and screen role.
  • The relative weakness at composition and visual design semantics suggests that delegating color, layout, and element detection to specialized vision models before LLM interpretation would likely improve retrieval quality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes a semantic-based UI inspiration search system (S&UI) that uses GPT-4o in a zero-shot manner to extract 14 designer-relevant semantic attributes from mobile UI screenshots, without relying on view hierarchies or other metadata. The semantics were identified through a formative study with six designers and organized into application, screen, composition, and visual design levels. The system embeds these extracted semantics and retrieves screens by weighted cosine similarity, with additional features for next/previous screen flows and iterative query refinement. The authors evaluate semantic extraction accuracy on Enrico, CLAY, and Screen2Words, assess semantic quality with ten designers, and compare S&UI against a GUIClip-based retrieval interface in a user study. They report that S&UI significantly outperforms the baseline on relevance, diversity, reliability, usefulness, and serendipity, and they release the extracted semantic dataset.

Significance. If the central claim holds, the paper makes a useful contribution: it demonstrates that an MLLM can extract high-level, designer-oriented semantics (target user, screen role, mood) directly from raw screenshots, and that these semantics can drive retrieval in a way that goes beyond visual similarity and keyword matching. The computational classification results on Enrico are the strongest part of the paper, showing large accuracy gaps over CLIP and GUIClip in zero-shot screen and app category classification. The structured YAML prompting strategy and the public dataset release are concrete assets for follow-up work. The paper is also honest in acknowledging limitations in composition-level and visual-level semantic extraction and in the current embedding-based retrieval approach. However, the headline comparative claim is currently supported only by a user study in which the rich S&UI interface is confounded with the underlying semantic representation, so the significance of the human evaluation is not yet established at the level the abstract claims.

major comments (4)
  1. [§6.3.1, Figure 8; §4.2.2; Appendix Figure 10] The comparative user study conflates the semantic representation with the interface. S&UI provides dropdown facet selectors, weight sliders, a screen-detail panel with extracted semantics, an Import button for iterative refinement, and dedicated next/previous-screen flow queries, while the GUIClip baseline only supports free-text and image queries. The significant gains in Relevance, Reliability, Usefulness, Diversity, and Serendipity could therefore be driven by query expressiveness, result explainability, or the direct mapping between flow tasks and S&UI's flow-query feature, rather than by the MLLM-extracted semantics themselves. To support the abstract's claim that the 'approach' significantly outperforms existing UI retrieval methods, the authors should either add an ablation that holds the interface constant (e.g., the same query panel fed by CLIP embeddings, or semantic queries run through the baseline interface) or explicitly reframe the result as a system-level comparison rather than an attribution to semantic extraction.
  2. [§5.2 and §5.3, Figure 6] Two computational claims lack the support needed for the conclusions drawn. First, the UI element prediction result (weighted F1 = 0.681) is reported without any baseline, so the reader cannot judge whether this is an improvement over existing UI element recognition methods; the claim that the model can 'identify various UI elements' is credible, but the comparative framing is not. Second, the screen-role richness analysis equates informativeness with syntactic dependency complexity and POS diversity, an unvalidated proxy. Higher syntactic complexity does not necessarily mean the description is more useful for design inspiration, and no human or task-based validation of description quality is provided. Please add a relevant baseline for element prediction and a more direct quality measure for screen descriptions, or temper the corresponding claims.
  3. [§3.3 and §6.2] There is a partial circularity in the support for the semantic taxonomy. The same designer community (six interviewees in §3) supplied the semantics that later shaped both the extraction prompt and the criteria by which the semantics were rated as relevant and useful by another ten designers (§6.2). The retrieval claim itself is grounded in external benchmarks, but the claim that these are 'the key UI semantics' for inspirational search is only validated within the community that helped define them. The paper should explicitly acknowledge this limitation and, if feasible, triangulate the taxonomy with a broader survey or an independent card-sorting study.
  4. [§7.5] The paper's own limitation section states that the retrieval method's reliance on weighted embedding similarity 'becomes less effective as query complexity increases.' This is directly relevant to the scope of the comparative claim: the user study tasks involve relatively simple queries (one domain plus an interaction sequence, or one mood keyword set), so the 'consistently outperformed' conclusion in §6.3.1 may not generalize to complex multi-faceted queries. The discussion should connect this admitted limitation to the boundary conditions of the headline claim, rather than presenting the user-study result as unqualified evidence for the superiority of the semantic approach.
minor comments (7)
  1. [§5.1, Table 4] The accuracy comparisons would be more convincing with confidence intervals, per-category error bars in Figure 5, and either significance tests or an explicit statement that these are descriptive comparisons on a fixed benchmark.
  2. [§6.1 and §6.3.1] For the Wilcoxon signed-rank tests, please report effect sizes or the underlying rating distributions per participant; with only 10 participants, individual differences could be substantial.
  3. [§6.3.1] The operationalization of Reliability is underspecified: it is defined as consistency across semantically similar queries, but the number of such queries, how semantic similarity of queries was ensured, and how consistency was scored are not reported.
  4. [§4.1.3] The 'up to 50% token reduction' for YAML over JSON cites a blog post; please replace this with a peer-reviewed source or temper the claim.
  5. [§7.4] The dataset release would be more reproducible with a versioned DOI, a license statement, and a clear description of whether raw screenshots are included or only semantic annotations.
  6. [References] Several references are incomplete or contain venue information only in the URL (e.g., [39] lacks a title and venue, [52] is listed as 'Just Accepted' without publication details); these should be completed.
  7. [Figure 2] The prompt illustration is dense and hard to read at print size; moving the full prompt template to an appendix and keeping a simplified version in the main text would improve clarity.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the MLLM extraction is zero-shot and benchmarked externally; the only self-citations are motivational and not load-bearing.

full rationale

The paper's derivation chain is self-contained in the sense that matters for circularity. The semantic taxonomy is an input to the extraction pipeline, not a fitted output, and the MLLM is used zero-shot, so the claimed semantic-understanding results are not equivalent to the prompt by construction. Computational evaluations are anchored to external datasets (Enrico, CLAY, Screen2Words) and to external baselines (CLIP variants and GUIClip), making the 'significantly outperforms' claim an empirical comparison rather than a renaming of inputs. The Section 6.2 quality ratings use different participants from the formative study and ask them to judge extracted semantics against screens, which does not force the observed relevance, comprehensiveness, or serendipity scores. The Section 6.3 comparative study does have an interface confound: S&UI's dropdowns, weight sliders, semantic detail panels, and import-based iteration are absent from the GUIClip baseline, so the outperformance may be partly attributable to interface affordances. That is a validity threat about attribution, not a circular reduction: no fitted parameter is renamed as a prediction, and no equation reduces to its own input. The self-citations in the paper, mainly [28] on embedding perception and [43] on CLIP outperforming traditional UI-specific models, are used for motivation and baseline selection, but the actual comparative evidence is generated in the present paper against external baselines, so these citations are not load-bearing. Section 7.5 explicitly lists limitations such as manual query input, weighted-embedding similarity limits, and single-screen focus, but those are correctness limitations rather than circular steps. Overall, the score of 2 reflects only the presence of minor non-load-bearing self-citations; there is no partial circularity in the central claim.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The system introduces no fitted parameters or new theoretical constructs. The main load-bearing assumptions are that proprietary API outputs are stable and trustworthy, that existing UI datasets provide valid ground truth, and that embedding-based cosine similarity aligns with designer relevance judgments. The screen-role richness proxy is a paper-specific assumption that has not been validated against semantic accuracy.

assumptions (5)
  • domain assumption GPT-4o zero-shot outputs are treated as reliable semantic groundings for retrieval and quality evaluation.
    Sections 4.1.4 and 5 use GPT-4o outputs as the basis for all downstream retrieval and evaluation without fine-tuning or verification beyond the reported metrics.
  • domain assumption Enrico, CLAY, Screen2Words, Mobbin, and SCapRepo are valid external ground truth and material for evaluation.
    Sections 5.1, 5.2, 5.3, and 6.2 treat the labels and screenshots in these datasets as objective benchmarks for semantic extraction and retrieval.
  • domain assumption Cosine similarity in text-embedding-3-large captures semantic relevance as perceived by designers.
    The retrieval method in Section 4.2.1 computes weighted cosine similarity over text embeddings and assumes this ranking reflects designer-perceived relevance.
  • ad hoc to paper Syntactic dependency complexity and POS diversity are meaningful proxies for screen-description informativeness.
    Section 5.3 uses these spacy-computed proxies to argue GPT-4o produces more informative screen descriptions than Screen2Words.
  • domain assumption The mapping between Enrico screen topics and the paper's screen categories is valid.
    Section 5.1 maps Enrico topics to the proposed screen categories before measuring classification accuracy, and the mapping is not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging Multimodal LLM for Inspirational User Interface Search." pith.science (2026). https://pith.science/paper/QIDSMRHC

@misc{pith2026250117799,
  author       = {Pith},
  title        = {Pith review of: Leveraging Multimodal LLM for Inspirational User Interface Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QIDSMRHC}},
  note         = {Machine review of arXiv:2501.17799}
}
read the original abstract

Inspirational search, the process of exploring designs to inform and inspire new creative work, is pivotal in mobile user interface (UI) design. However, exploring the vast space of UI references remains a challenge. Existing AI-based UI search methods often miss crucial semantics like target users or the mood of apps. Additionally, these models typically require metadata like view hierarchies, limiting their practical use. We used a multimodal large language model (MLLM) to extract and interpret semantics from mobile UI images. We identified key UI semantics through a formative study and developed a semantic-based UI search system. Through computational and human evaluations, we demonstrate that our approach significantly outperforms existing UI retrieval methods, offering UI designers a more enriched and contextually relevant search experience. We enhance the understanding of mobile UI design semantics and highlight MLLMs' potential in inspirational search, providing a rich dataset of UI semantics for future studies.

Figures

Figures reproduced from arXiv: 2501.17799 by the authors.

Figure 1
Figure 1. Overview of our approach for semantic-based UI search. Left: A multimodal large language model is employed to [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of prompt and semantic output for mobile UI semantic extraction using multimodal LLM. The left panel [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The query and retrieval method illustrates how user-specified semantics and weights are processed to retrieve relevant [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The S&UI system interface. The system enables designers to search UI screens using key UI semantics, such as app [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Comparison of Correct Screen and App Category Predictions: The top chart compares GPT-4o and GUIClip across 20 [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Box plots comparing the syntactic dependency complexity (left) and POS diversity scores of screen descriptions (right) [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Quality assessment results of extracted semantics by UI designers. The graph shows the rating distribution on a [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Comparison of user ratings between GUIClip and S&UI across five metrics: Relevance, Diversity, Reliability, Usefulness, [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Real examples of results from participant E3 for flow and style tasks for evaluating the S&UI and GUIClip systems. [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Baseline System Interface. The GUIClip-based retrieval system supports natural language text search queries and [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 21 canonical work pages

  1. [1]

    Xavier Amatriain. 2024. Prompt Design and Engineering: Introduction and Advanced Methods. arXiv:2401.14423 [cs.SE]

    arXiv 2024

  2. [2]

    Anthropic. 2023. Be clear, direct, and detailed - Anthropic — docs.anthropic.com. https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/be- clear-and-direct

    work page 2023

  3. [3]

    Anthropic. 2023. Prompt Engineering / Use XML Tags. https://docs.anthropic. com/en/docs/build-with-claude/prompt-engineering/use-xml-tags

    work page 2023

  4. [4]

    Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Carbune, Jason Lin, Jindong Chen, and Abhanshu Sharma

  5. [5]

    Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Aguera y Arcas. 2021. UIBert: Learning Generic Multimodal Representations for UI Understanding. arXiv:2107.13731 [cs.CV] https://arxiv.org/abs/2107.13731

    arXiv 2021

  6. [6]

    Romain Beaumont. 2022. Clip Retrieval: Easily compute clip embeddings and build a clip retrieval system with them. https://github.com/rom1504/clip- retrieval

    2022

  7. [7]

    Nathalie Bonnardel. 1999. Creativity in design activities: the role of analogies in a constrained cognitive environment. In Proceedings of the 3rd Conference on Creativity & Cognition (Loughborough, United Kingdom) (C&C ’99). Association for Computing Machinery, New York, NY, USA, 158–165. https://doi.org/10. 1145/317561.317589

    work page arXiv 1999

  8. [8]

    Sara Bunian, Kai Li, Chaima Jemmali, Casper Harteveld, Yun Fu, and Magy Seif Seif El-Nasr. 2021. VINS: Visual Search for Mobile User Interface Design. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 423, 14 pages. https://doi.org/10...

    arXiv 2021

Show all 62 references
  1. [9]

    Dow, and Christian D

    Joel Chan, Steven P. Dow, and Christian D. Schunn. 2015. Do the best design ideas (really) come from conceptually distant sources of inspiration? Design Studies 36 (2015), 31–58. https://doi.org/10.1016/j.destud.2014.08.001

    2015 doi

  2. [10]

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. arXiv:2401.10935 [cs.HC] https://arxiv.org/abs/2401.10935

    2024 arXiv

  3. [11]

    Design Council. [n. d.]. Double Diamond. https://www.designcouncil.org.uk/our- resources/the-double-diamond/

  4. [12]

    Nigel Cross. 2004. Expertise in design: an overview. Design Studies 25, 5 (2004), 427–441. https://doi.org/10.1016/j.destud.2004.06.002 Expertise in Design

    2004 doi

  5. [13]

    Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software an...

    2017

  6. [14]

    Harrison, and Anton Dereventsov

    Oluwole Fagbohun, Rachel M. Harrison, and Anton Dereventsov. 2024. An Empirical Categorization of Prompting Techniques for Large Language Models: A Practitioner’s Guide. arXiv:2402.14837 [cs.CL]

    2024 arXiv

  7. [15]

    Yuan Gao, Kunyu Shi, Pengkai Zhu, Edouard Belval, Oren Nuriel, Srikar Appalaraju, Shabnam Ghadar, Vijay Mahadevan, Zhuowen Tu, and Stefano Soatto. 2024. Enhancing Vision-Language Pre-training with Rich Supervisions. arXiv:2403.03346 [cs.CV]

    2024 arXiv

  8. [16]

    Milene Gonçalves, Carlos Cardoso, and Petra Badke-Schaub. 2014. What in- spires designers? Preferences on inspirational approaches during idea generation. Design Studies 35, 1 (2014), 29–53. https://doi.org/10.1016/j.destud.2013.09.001

    2014 doi

  9. [17]

    Google. 2024. Give clear and specific instructions. https://cloud.google.com/ vertex-ai/generative-ai/docs/learn/prompts/clear-instructions

    2024

  10. [18]

    Greg Guest, Arwen Bunce, and Laura Johnson. 2006. How Many Interviews Are Enough?: An Experiment with Data Saturation and Variability. Field Methods 18, 1 (2006), 59–82. https://doi.org/10.1177/1525822X05279903

    2006 doi

  11. [19]

    Herring, Chia-Chen Chang, Jesse Krantzler, and Brian P

    Scarlett R. Herring, Chia-Chen Chang, Jesse Krantzler, and Brian P. Bailey. 2009. Getting inspired! understanding how and why examples are used in creative design practice. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09...

    2009

  12. [20]

    Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd

  13. [21]

    Chih-Pei HU and Yan-Yi CHANG. 2017. John W. Creswell, Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Journal of Social and Administrative Sciences 4, 2 (Jun. 2017), 205–207. https://doi.org/10.1453/jsas. v4i2.1313

    2017 doi

  14. [22]

    Canny, and Jeffrey Nichols

    Forrest Huang, John F. Canny, and Jeffrey Nichols. 2019. Swire: Sketch-Based User Interface Retrieval. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–10...

    2019

  15. [23]

    Yue Jiang, Yuwen Lu, Jeffrey Nichols, Wolfgang Stuerzlinger, Chun Yu, Christof Lutteroth, Yang Li, Ranjitha Kumar, and Toby Jia-Jun Li. 2022. Computational Approaches for Understanding, Generating, and Adapting User Interfaces. In Extended Abstracts of the 2022 CHI Conference ...

    2022

  16. [24]

    Jong Wook Kim. 2021. CLIP/Prompt_Engineering_for_ImageNet.ipynb at main · openai/CLIP — github.com. https://github.com/openai/CLIP/blob/main/ notebooks/Prompt_Engineering_for_ImageNet.ipynb. CHI ’25, April 26-May 1, 2025, Yokohama, Japan Seokhyeon Park, Yumin Song, Soohyun Lee...

    2021

  17. [25]

    Janin Koch, Andrés Lucero, Lena Hegemann, and Antti Oulasvirta. 2019. May AI? Design Ideation with Cooperative Contextual Bandits. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machin...

    2019

  18. [26]

    Klem- mer

    Brian Lee, Savil Srivastava, Ranjitha Kumar, Ronen Brafman, and Scott R. Klem- mer. 2010. Designing with interactive example galleries. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10) . Association for Computing M...

    2010

  19. [27]

    Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Ju- lian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2Struct: Screenshot Parsing as Pretrain- ing for Visual Language Understanding. In Proceedings of ...

    2023

  20. [28]

    Soohyun Lee, Minsuk Chang, Seokhyeon Park, and Jinwook Seo. 2024. Assessing Graphical Perception of Image Embedding Models using Channel Effectiveness. In 2024 IEEE Visualization and Visual Analytics (VIS) . 226–230. https://doi.org/10. 1109/VIS55277.2024.00053

    2024

  21. [29]

    Leiva, Asutosh Hota, and Antti Oulasvirta

    Luis A. Leiva, Asutosh Hota, and Antti Oulasvirta. 2021. Enrico: A Dataset for Topic Modeling of Mobile UI Designs. In22nd International Conference on Human- Computer Interaction with Mobile Devices and Services (Oldenburg, Germany) (MobileHCI ’20). Association for Computing M...

    2021

  22. [30]

    Gang Li, Gilles Baechler, Manuel Tragut, and Yang Li. 2022. Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems . Association for Computing Machinery, New Orleans, LA, USA, 1...

    2022

  23. [31]

    Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. 2021. Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. In Pro- ceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing M...

    2021

  24. [32]

    Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. 2020. Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements. arXiv:2010.04295 [cs.LG]

    2020 arXiv

  25. [33]

    Junpeng Liu, Tianyue Ou, Yifan Song, Yuxiao Qu, Wai Lam, Chenyan Xiong, Wenhu Chen, Graham Neubig, and Xiang Yue. 2024. Harnessing Webpage UIs for Text-Rich Visual Understanding. (2024). arXiv:2410.13824 [cs.CV]

    2024 arXiv

  26. [34]

    Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar

    Thomas F. Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018. Learning Design Semantics for Mobile Apps. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (Berlin, Germany) (UIST ’18). Association for Comput...

    2018

  27. [35]

    Elya Livshitz. 2023. YAML vs. JSON: Which is more efficient for language mod- els? https://betterprogramming.pub/yaml-vs-json-which-is-more-efficient-for- language-models-5bc11dd0f6df

    2023

  28. [36]

    Yuwen Lu, Ziang Tong, Qinyi Zhao, Chengzhi Zhang, and Toby Jia-Jun Li. 2023. UI Layout Generation with LLMs Guided by UI Grammar. arXiv:2310.15455 [cs.HC]

    2023 arXiv

  29. [37]

    Yuwen Lu, Chengzhi Zhang, Iris Zhang, and Toby Jia-Jun Li. 2022. Bridg- ing the Gap Between UX Practitioners’ Work Practices and AI-Enabled De- sign Support Tools. In Extended Abstracts of the 2022 CHI Conference on Hu- man Factors in Computing Systems (New Orleans, LA, USA) (...

    2022

  30. [38]

    Kate Moran. 2016. Tone-of-voice words. https://www.nngroup.com/articles/tone- voice-words/

    2016

  31. [39]

    Janice M. Morse. 2000. Determining Sample Size. , 3-5 pages. https://doi.org/10. 1177/104973200129118183

    2000

  32. [40]

    OpenAI. 2024. Introducing vision to the fine-tuning API . https://openai.com/ index/introducing-vision-to-the-fine-tuning-api/

    2024

  33. [41]

    OpenAI. 2024. OpenAI GPT-4 API Documentation. https://platform.openai.com/ docs/models/gpt-4o

    2024

  34. [42]

    OpenAI. 2024. Prompt engineering. https://platform.openai.com/docs/guides/ prompt-engineering

    2024

  35. [43]

    Seokhyeon Park, Wonjae Kim, Young-Ho Kim, and Jinwook Seo. 2023. Compu- tational Approaches for App-to-App Retrieval and Design Consistency Check. arXiv:2309.10328 [cs.HC]

    2023 arXiv

  36. [44]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings...

    2021

  37. [45]

    Daniel Ritchie, Ankita Arvind Kejriwal, and Scott R. Klemmer. 2011. d.tour: style-based exploration of design example galleries. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology(Santa Barbara, California, USA) (UIST ’11). Association fo...

    2011

  38. [46]

    Yvonne Rogers, Helen Sharp, and Jennifer Preece. 2023. Interac- tion Design: Beyond Human-Computer Interaction, 6th Edition . Wiley. https://www.wiley.com/en-us/Interaction+Design%3A+Beyond+Human+ Computer+Interaction%2C+6th+Edition-p-00381113

    2023

  39. [47]

    Bailey, Cole Coats, and Kevin Hamilton

    Moushumi Sharmin, Brian P. Bailey, Cole Coats, and Kevin Hamilton. 2009. Understanding knowledge management practices for early design activity and its implications for reuse. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ...

    2009 doi

  40. [48]

    Gajos, and Steven P

    Pao Siangliulue, Joel Chan, Krzysztof Z. Gajos, and Steven P. Dow. 2015. Providing Timely Examples Improves the Quantity and Quality of Generated Ideas. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition (Glasgow, United Kingdom) (C&C ’15). Associatio...

    2015

  41. [49]

    Bryan Wang, Gang Li, and Yang Li. 2023. Enabling Conversational Interaction with Mobile UI using Large Language Models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, ...

    2023

  42. [50]

    Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021. Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA)(UIST ’21). Association for ...

    2021

  43. [51]

    Jinge Wang, Qing Ye, Li Liu, Nancy Lan Guo, and Gangqing Hu. 2024. Scientific figures interpreted by ChatGPT: strengths in plot recognition and limits in color perception. NPJ Precision Oncology 8, 1 (2024), 84. https://doi.org/10.1038/s41698- 024-00576-z

    2024 doi

  44. [52]

    Jialiang Wei, Anne-Lise Courbis, Thomas Lambolais, Binbin Xu, Pierre Louis Bernard, Gérard Dray, and Walid Maalej. 2024. GUing: A Mobile GUI Search Engine using a Vision-Language Model. ACM Trans. Softw. Eng. Methodol. (Nov. 2024). https://doi.org/10.1145/3702993 Just Accepted

    2024 doi

  45. [53]

    Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S. Yu. 2023. Multimodal Large Language Models: A Survey. In 2023 IEEE International Confer- ence on Big Data (BigData) . 2247–2256. https://doi.org/10.1109/BigData59044. 2023.10386743

    2023

  46. [54]

    Jason Wu, Yi-Hao Peng, Xin Yue Amanda Li, Amanda Swearngin, Jeffrey P Bigham, and Jeffrey Nichols. 2024. UIClip: A Data-driven Model for Assessing User Interface Design. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, ...

    2024

  47. [55]

    Jason Wu, Kashyap Todi, Joannes Chan, Brad A Myers, and Ben Lafreniere. 2024. FrameKit: A Tool for Authoring Adaptive UIs Using Keyframes. InProceedings of the 29th International Conference on Intelligent User Interfaces (Greenville, SC, USA) (IUI ’24). Association for Computi...

    2024

  48. [56]

    Ziming Wu, Qianyao Xu, Yiding Liu, Zhenhui Peng, Yingqing Xu, and Xiaojuan Ma. 2021. Exploring Designers’ Practice of Online Example Management for Supporting Mobile UI Design. In Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction (Toulouse &...

    2021

  49. [57]

    Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. 2023. ExpertPrompting: Instructing Large Language Models to be Distinguished Experts. arXiv:2305.14688 [cs.CL]

    2023 arXiv

  50. [58]

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A Survey on Multimodal Large Language Models. National Science Review (11 2024), nwae403. https://doi.org/10.1093/nsr/nwae403

    2024 doi

  51. [59]

    Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. 2025. Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs. In Computer Vision – ECCV 2024 , Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky...

    2025

  52. [60]

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2023. AppAgent: Multimodal Agents as Smartphone Users. arXiv:2312.13771 [cs.CV] Leveraging Multimodal LLM for Inspirational User Interface Search CHI ’25, April 26-May 1, 2025, Yokoham...

    2023 arXiv

  53. [2020]

    https: //doi.org/10.5281/zenodo.1212303

    spaCy: Industrial-strength Natural Language Processing in Python. https: //doi.org/10.5281/zenodo.1212303

    doi

  54. [2024]

    In Proceedings of the Thirty-Third International Joint Conference on Artificial Intel- ligence, IJCAI-24, Kate Larson (Ed.)

    ScreenAI: A Vision-Language Model for UI and Infographics Understanding. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intel- ligence, IJCAI-24, Kate Larson (Ed.). International Joint Conferences on Artificial Intelligence Organization, 3058–3...

    doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.