Pith. sign in

REVIEW 5 major objections 6 minor 46 references

Fusing UI Structure & Semantics for Feature-Oriented App Screen Retrieval & Clustering

T0 review · 5 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FRAME builds a spatial graph of UI components and uses spectral propagation plus Rips-complex weighting to make screen embeddings capture the features a screen affords, improving retrieval and clustering over CLIP and other baselines…

desk verdict FRAME is a genuinely novel training-free UI embedding approach with a solid MRR result, but the impossible HR@k table and undisclosed hyperparameters (w) need major revision before the retrieval claim is fully trustworthy. read the letter →

arxiv 2608.08880 v1 pith:PYUOVSCP submitted 2026-08-09 cs.SE

Arun Krishna Vajjala , Yanfu Yan , Ajay Krishna Vajjala , Shrunal Pothagoni , Denys Poshyvanyk , Kevin Moran This is my paper · ORCID
classification cs.SE
keywords screenretrievalclusteringUIembeddingneuro-symbolicVietoris-Ripscomplexpropagationvision-languagemodelsmobileunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FRAME tries to establish that app screens can be embedded so that similarity reflects the features a screen affords, rather than only its pixels or text. The paper argues that augmenting off-the-shelf vision-language embeddings with a symbolic UI component graph—detecting components, connecting them spatially, propagating embeddings along the graph, and weighting them by triangle areas from a Vietoris-Rips complex—produces embeddings that rank and cluster screens by function better than the base models. This matters because UI testing, bug report management, and design tools depend on retrieving or grouping screens that share functionality despite different visual styles. Across the Aurora, Avgust, and Enrico benchmarks, the paper reports up to 13% higher MRR (mean reciprocal rank) in screen retrieval and up to 7.6 percentage points higher clustering accuracy over strong baselines, with no additional training of the underlying models.

What carries the argument

The load-bearing mechanism is a UI component neighborhood graph combined with two operations: (1) embedding propagation through the first-order spectral filter $S' = (I_N + w D^{-1/2} A D^{-1/2})S$, which mixes each component's representation with its neighbors and approximates a localized graph filter; and (2) geometric weighting through the Vietoris-Rips complex, in which open balls of radius $\epsilon = 0.5$ around propagated embeddings form 2-simplices and the area of each triangle (Heron's formula) is used as the weight of the three involved components. Together these operations encode the symbolic prior that spatially grouped UI components correspond to screen features, so the final screen embedding preserves feature structure that flat pixel and text embeddings miss.

What would settle it

Run FRAME on a held-out corpus of screens at a different resolution or from a different platform (for example, iOS or web) without retuning the 300-pixel threshold or the Rips radius, and compare its MRR and clustering accuracy with CLIP; if the margin disappears while a per-dataset tuned version restores it, the claim that the fixed graph geometry generalizes is false.

Watch

Extended reading notes

Core claim

FRAME takes a screenshot, normalizes it to greyscale with doubled contrast, detects UI component bounding boxes, and builds a graph whose nodes carry a CLIP image embedding and a BERT text embedding. Edges connect components within 300 px Manhattan distance. It mixes each node's embedding with its neighbors through a first-order spectral filter, constructs a Vietoris-Rips complex on the propagated embeddings, and uses the area of each 2-simplex (triangle) as a weight so that clustered components dominate the screen representation. The weighted, propagated component embeddings are concatenated with a whole-screen CLIP embedding and projected to 116 dimensions. The paper's central claim is that this graph-and-geometry augmentation makes the resulting embedding feature-oriented: screens that support the same function become closer in embedding space even when their visual designs differ, improving retrieval ranking and clustering on three UI benchmarks.

Load-bearing premise

The load-bearing premise is that the single 300-pixel Manhattan edge threshold and the single Rips radius of 0.5, chosen by visual inspection on a small held-out subset of the RICO dataset, also hold for the Avgust, Aurora, and Enrico datasets so that the triangles they create correspond to real UI features.

Editorial extensions

If this is right

  • FRAME can be plugged into existing screen-retrieval-based UI tools, such as test transfer, duplicate bug report detection, and design search, without retraining the underlying vision-language model.
  • Because FRAME augments rather than replaces CLIP and BERT, the embedding quality should improve automatically as those base models improve.
  • The clustering results indicate that FRAME separates functional screen types more cleanly than CLIP, which suggests it can support app categorization and design-pattern discovery from unlabeled screens.
  • Only a screenshot is needed at inference time, so FRAME works in settings where Android view hierarchies or other runtime metadata are unavailable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Replacing CLIP with a newer vision-language model should change the size of FRAME's gains, since the method inherits component-level visual discrimination from the base model.
  • The 300-pixel edge threshold and $\epsilon=0.5$ Rips radius are tuned on RICO screens; at other resolutions or on web and desktop layouts these parameters would likely need re-tuning, and a sweep would show whether per-dataset tuning increases the margin over CLIP further.
  • A direct test of whether the geometric weighting carries the feature signal would compare FRAME with a variant that replaces triangle-area weights with uniform averaging over graph neighbors, since the paper does not ablate the Rips radius or edge threshold themselves.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes FRAME, a neuro-symbolic UI screen embedding technique that augments CLIP and BERT embeddings by (i) normalizing screenshots, (ii) building a UI component graph from detected bounding boxes, (iii) propagating embeddings through a spectral graph filter, (iv) weighting component relationships via Vietoris-Rips triangulation, and (v) applying PCA. FRAME is evaluated on screen retrieval (MRR, HR@k) and clustering (nine metrics) across three datasets (Aurora, Avgust, Enrico) against CLIP, BLIP, BERT, and Screen2Vec. The authors report consistent retrieval improvements, clustering improvements over CLIP, and an ablation study isolating each component.

Significance. If the results hold, FRAME provides a training-free, model-agnostic structural prior that could benefit UI testing, bug localization, and design search tools. The paper ships an archived replication package and source code, and the ablation design is systematic. However, the current manuscript contains internal metric inconsistencies and overclaimed significance that must be resolved before the empirical claims can be accepted.

major comments (5)
  1. [Section III-C, Table III] In Section III-C, HR@k is defined as 'the percentage of queries for which an approach retrieves at least one correct screen within the top-k screens.' Under this definition, HR@1 ≤ HR@5 ≤ HR@10 for every method, because a query satisfied at rank 1 is also satisfied at ranks 5 and 10. Table III, however, shows the opposite ordering in every row (e.g., Aurora FRAME: HR@1=0.5625, HR@5=0.4652, HR@10=0.4166; the same decreasing pattern appears for all baselines and all datasets). These values are mathematically impossible for the defined metric; the columns are likely a different statistic (e.g., precision-at-k or recall-at-k at that exact rank). The authors must recompute the correct hit rates, re-run the paired t-tests on those correct values, and update the retrieval claims in Section IV-A accordingly.
  2. [Section IV-A, Table III] Section IV-A states that 'most results across all three benchmarks reaching statistical significance,' and the introduction claims FRAME 'significantly outperforms baselines across three diverse datasets.' Yet Table III reports p-values on Enrico of 0.0176 (HR@1), 0.0954 (HR@5), 0.0687 (HR@10), and 0.0586 (MRR). Only the HR@1 result is below 0.05, so the Enrico results are not consistently significant. The significance claims and the abstract's blanket wording must be revised to reflect the actual statistical outcomes, or additional evidence must be provided.
  3. [Section II-C, Eq. (1)] Equation (1) defines the propagation update as S' = (I_N + w D^{-1/2} A D^{-1/2}) S, where w is 'a constant to balance the information from the original node/component with structural information from the neighbor nodes/components.' The manuscript never reports the value of w used in the experiments or any tuning procedure for it. Because this equation is the core of the propagation phase, the missing value prevents an independent re-instantiation of FRAME from the text and should be added to the experimental configuration (e.g., in Section III or the replication package).
  4. [Section III-D, Table IV] The clustering evaluation in Section III-D and Table IV compares FRAME only to CLIP, and the results are reported from a single run with no variance estimates or significance tests. Absolute clustering accuracy is very low even for FRAME (0.0217 on Aurora, 0.0543 on Enrico, 0.1669 on Avgust). Given the paper's claim of superior clustering, the authors should report multiple K-means runs with different seeds (or other forms of variance), add a significance analysis, and compare against at least one additional baseline (e.g., BLIP or Screen2Vec) so that the clustering claim is not tied to a single comparison.
  5. [Section II-B2 and II-D] The 300-px Manhattan edge threshold in Section II-B2 and the Rips radius epsilon=0.5 in Section II-D are selected by visual inspection on a small held-out subset of RICO, yet they are applied unchanged to Avgust, Aurora, and Enrico. The manuscript should report a sensitivity analysis of the retrieval (and clustering) results with respect to these two parameters, at least over a small grid around the chosen values, to demonstrate that the reported gains are not artifacts of dataset-specific tuning.
minor comments (6)
  1. [Section II-D, Eq. (2)] Heron's formula in Eq. (2) is typeset with a bare 'p' instead of a square-root symbol; the notation A, B, C for edge lengths also conflicts with the adjacency matrix A in Eq. (1). The formula should be restated cleanly.
  2. [Table III] In the Avgust HR@10 row, the entry '0.00150MRR' has the MRR label run into the p-value; this formatting error should be fixed.
  3. [Abstract and Section I] The abstract and introduction state that FRAME achieves '7.6 percentage points higher accuracy in clustering'; Table IV shows per-dataset gains of 2.0, 4.7, and 16.0 percentage points (Aurora, Enrico, Avgust), so the 7.6 figure appears to be an average and should be labeled as such.
  4. [Section III-F and IV-D] Section III-F states that the qualitative study uses a single randomly selected test image from Avgust; this makes the qualitative comparison in Section IV-D anecdotal. The paper should either present several examples or explicitly describe this as an illustrative case, not evidence of general behavior.
  5. [Throughout] The acronym expansion uses 'ReinForced' but the standard spelling is 'Reinforced'; please correct the acronym expansion in the abstract and Section I.
  6. [Section IV-C, Table VI] The sentence 'The NG-C (contrast only) variant is the least effective configuration, ranking last on all eight dataset–metric combinations' is correct for Table VI, but the subsequent discussion of G-NC vs. NG-NC on Avgust could be clarified by reporting the tied HR@1 values (0.8784) explicitly in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FRAME's gains are measured against external baselines and supported by component ablations; the only self-citation is not load-bearing.

full rationale

FRAME's central claim is that augmenting CLIP/BERT embeddings with a UI component graph, spectral propagation, and Rips weighting improves feature-oriented retrieval and clustering. The evaluation compares FRAME to externally sourced baselines (CLIP, BLIP, BERT, Screen2Vec) on external datasets (Enrico, and Avgust/Aurora built in prior work) using label agreement, not any quantity that FRAME is fitted to reproduce. The component ablations in Tables V and VI compare full FRAME against variants that omit propagation, weighting, preprocessing, or the global CLIP embedding, and these variants are not constructed to enforce the headline outcome; they independently show each piece contributes. The graph edge threshold (300px) and Rips epsilon (0.5) are tuned on held-out RICO subsets via visual inspection and triangle-count heuristics, not on the retrieval/clustering labels that are later predicted, and the target triangle count is not the reported evaluation metric. The only mild self-citation is to Krishna Vajjala et al. [33] for the ATE weighting and triangle-count calibration (Section II-D); this is a component-level reuse, not an assertion that FRAME outperforms baselines by construction, so it is not load-bearing circularity. Separate correctness caveats do not affect the circularity verdict: Table III reports HR@k values that decrease as k increases, which contradicts the paper's own definition of hit rate, and the propagation weight w in Eq. (1) is not reported, hampering reproduction. These are metric and reproducibility concerns, not instances of a prediction reducing to its inputs, so the circularity score remains 0.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

FRAME's contribution is the synthesis of off-the-shelf VLM embeddings with graph propagation and topological weighting. The free parameters are numerous and partly hand-tuned: edge threshold, Rips radius, propagation weight (unreported), PCA dimension, and contrast factor. The method assumes component detection is accurate and that geometric triangle area encodes feature dominance. No fundamentally new entity is introduced.

free parameters (5)
  • UI graph edge distance threshold = 300 px (Manhattan distance)
    Section II-B2: chosen via grid search on a held-out RICO dev set and visual inspection by two authors; defines graph neighborhoods that represent screen features.
  • Vietoris-Rips radius epsilon = 0.5
    Section II-D: selected from a 0.1-1.0 parameter study targeting about 100 2-simplices, assumed to match an average of 20-100 features per screen; visually inspected by two authors.
  • Propagation balance weight w = not reported
    Section II-C, Eq. 1: balances original versus neighbor information in S'=(I_N + w D^-1/2 A D^-1/2)S; the value is never specified in the paper.
  • PCA output dimensionality = 116
    Section II-E: full 1,792-dim embeddings are reduced to 116 without stating the retained variance or selection criterion.
  • Contrast enhancement factor = 2x
    Section II-A2: Pillow ImageEnhance doubles contrast; no alternative values or tuning are reported.
assumptions (6)
  • domain assumption UI component detection via UIED yields accurate bounding boxes for all interactive and meaningful components.
    Section II-B1: the component graph, propagation, and weighting all depend on detected bounds; no detection accuracy is reported on the three datasets.
  • domain assumption CLIP and BERT embeddings are informative representations of individual UI components.
    Section II-B2: FRAME concatenates CLIP image and BERT text embeddings per node; this assumes off-the-shelf VLM embeddings capture UI element semantics.
  • ad hoc to paper The Vietoris-Rips 2-simplex area is a valid proxy for functional importance of UI component groups.
    Section II-D: the paper asserts that components consuming more screen real-estate are more functionally dominant; this heuristic is stated, not derived or validated externally.
  • domain assumption A single 300 px Manhattan threshold and epsilon=0.5 generalize across the three evaluation datasets.
    Sections II-B2 and II-D: hyperparameters tuned on a held-out RICO dev set are applied unchanged to Avgust, Aurora, and Enrico.
  • domain assumption The ground-truth labels in Aurora, Avgust, and Enrico are reliable proxies for feature-level screen similarity.
    Section III-A: retrieval and clustering evaluation treats the provided labels as the ground truth for semantic similarity.
  • standard math The first-order spectral graph filter (GCN propagation) is appropriate for UI component graphs constructed by Euclidean distance.
    Section II-C: Eq. 1 follows Kipf and Welling; the axiom is that UI component neighborhoods behave like the graphs GCN was designed for.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fusing UI Structure & Semantics for Feature-Oriented App Screen Retrieval & Clustering." pith.science (2026). https://pith.science/paper/PYUOVSCP

@misc{pith2026260808880,
  author       = {Pith},
  title        = {Pith review of: Fusing UI Structure & Semantics for Feature-Oriented App Screen Retrieval & Clustering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PYUOVSCP}},
  note         = {Machine review of arXiv:2608.08880}
}
read the original abstract

User Interface (UI) programming is challenging due to the complex abstraction gap between code and graphical software representations. To bridge this gap, UI programming tools often rely on screen retrieval and clustering, which require accurate similarity measures based on overlapping features. However, computing feature-oriented similarity is difficult because screens with similar functionality often exhibit design variations. To address this, we propose FRAME (ReinForced UseR InterfAce Screen EMbedding with Graphical Structural ComprEhension), a multi-modal, neuro-symbolic embedding technique. FRAME constructs symbolic, graph-based representations of UI components to encode salient relationships and capture feature patterns across different screens. It leverages large vision-language models for visual and lexical encoding, alongside a novel UI-specific computational geometry algorithm that enables weighted embedding propagation. Across three benchmarks, FRAME outperforms strong baselines by up to 13% MRR in search and 7.6 percentage points in clustering accuracy. A comprehensive ablation study further confirms the benefit of each component, demonstrating FRAME's potential for enhancing automated UI design and testing tools.

Figures

Figures reproduced from arXiv: 2608.08880 by the authors.

Figure 1
Figure 1. Example UI Component Relationships & Corresponding Features [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the FRAME Screen Embedding Workflow connections between nodes are constructed. 3 In the Em￾bedding Propagation phase FRAME shares information across connected UI graph components using an approximation of the spectral filter of the constructed UI graph. 4 In the Geometric Embedding Weighting phase, groups of compo￾nents embeddings are weighted by calculating the Vietoris￾Rips complex [23], a technique ad… view at source ↗
Figure 3
Figure 3. Three Images with Similar CLIP Embeddings [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Visualization of Low-Dimensional Simplices [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: CLIP Embedding Clustering – Avgust Dataset FRAME Embedding Clustering [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: FRAME Embedding Clustering – Avgust Dataset Visualizations in Figs. 6 and 7 show that CLIP creates tighter, more homogeneous clusters that sacrifice accuracy due to a lack of structural granularity. In contrast, FRAME effectively identifies and groups functionally simi…
Figure 8
Figure 8. Figure 8: Comparison of the Top-3 Most Similar Screens for Each Embedding [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 27 canonical work pages

  1. [1]

    Challenges of HCI design and implementation,

    B. Myers, “Challenges of HCI design and implementation,”interactions, vol. 1, no. 1, pp. 73–83, Jan. 1994. [Online]. Available: http: //doi.acm.org.proxy.wm.edu/10.1145/174800.174808

    work page arXiv 1994

  2. [2]

    Survey on user interface programming,

    B. A. Myers and M. B. Rosson, “Survey on user interface programming,” inProceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI ’92. New York, NY , USA: Association for Computing Machinery, 1992, p. 195–202. [Online]. Available: https://doi.org/10.1145/142750.142789

    arXiv 1992

  3. [3]

    Barriers in front-end web devel- opment,

    D. I. Samudio and T. D. LaToza, “Barriers in front-end web devel- opment,” in2022 IEEE Symposium on Visual Languages and Human- Centric Computing (VL/HCC), 2022, pp. 1–11

    work page 2022

  4. [4]

    Designing for designers: an analysis of design practice in the real world,

    M. B. Rosson, S. Maass, and W. A. Kellogg, “Designing for designers: an analysis of design practice in the real world,” inProceedings of the SIGCHI/GI Conference on Human Factors in Computing Systems and Graphics Interface, ser. CHI ’87. New York, NY , USA: Association for Computing Machinery, 1986, p. 137–142. [Online]. Available: https://doi.org/10.1145...

    work page arXiv 1986

  5. [5]

    A. B. Tucker,Computer Science Handbook, Second Edition. Boca Raton, FL, USA: CRC Press, 2004

    work page 2004

  6. [6]

    Automated reporting of GUI design violations for mobile apps,

    K. Moran, B. Li, C. Bernal-C ´ardenas, D. Jelf, and D. Poshyvanyk, “Automated reporting of GUI design violations for mobile apps,” inProceedings of the 40th International Conference on Software Engineering, ser. ICSE ’18. New York, NY , USA: ACM, 2018, p. 165–175. [Online]. Available: https://doi.org/10.1145/3180155.3180246

    arXiv 2018

  7. [8]

    Aurora: Navigating UI tarpits via automated neural screen understanding,

    S. A. Khan, W. Wang, Y . Ren, B. Zhu, J. Shi, A. McGowan, W. Lam, and K. Moran, “Aurora: Navigating UI tarpits via automated neural screen understanding,” in2024 IEEE International Conference on Software Testing, Verification and Validation, ser. ICST’16, Apr. 2024

    work page 2024

  8. [9]

    Detecting non-crashing functional bugs in Android apps via deep-state differential analysis,

    J. Wang, Y . Jiang, T. Su, S. Li, C. Xu, J. Lu, and Z. Su, “Detecting non-crashing functional bugs in Android apps via deep-state differential analysis,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2022. New York, NY , USA: Association for Computing Ma...

    arXiv 2022

Show all 46 references
  1. [10]

    Efficiency matters: Speeding up automated testing with GUI rendering inference,

    S. Feng, M. Xie, and C. Chen, “Efficiency matters: Speeding up automated testing with GUI rendering inference,” inProceedings of the 45th International Conference on Software Engineering, ser. ICSE ’23. IEEE Press, 2023, p. 906–918. [Online]. Available: https://doi.org/10.1109...

    2023

  2. [11]

    Semantic GUI scene learning and video alignment for detecting duplicate video-based bug reports,

    Y . Yan, N. Cooper, O. Chaparro, K. Moran, and D. Poshyvanyk, “Semantic GUI scene learning and video alignment for detecting duplicate video-based bug reports,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY , U...

    2024

  3. [12]

    Toward the automated localization of buggy mobile app UIs from bug descriptions,

    A. Saha, Y . Song, J. Mahmud, Y . Zhou, K. Moran, and O. Chaparro, “Toward the automated localization of buggy mobile app UIs from bug descriptions,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’24), 2024

    2024

  4. [13]

    Machine learning-based prototyping of graphical user interfaces for mobile apps,

    K. Moran, C. Bernal-C ´ardenas, M. Curcio, R. Bonett, and D. Poshy- vanyk, “Machine learning-based prototyping of graphical user interfaces for mobile apps,”IEEE Transactions on Software Engineering, vol. 46, no. 2, pp. 196–221, 2020

    2020

  5. [14]

    Guifetch: supporting app design and development through GUI search,

    F. Behrang, S. P. Reiss, and A. Orso, “Guifetch: supporting app design and development through GUI search,” inProceedings of the 5th International Conference on Mobile Software Engineering and Systems, ser. MOBILESoft ’18. New York, NY , USA: Association for Computing Machiner...

    2018

  6. [15]

    Guievo: Automated evolution of mobile app UIs,

    S. Salma, S. M. H. Mansur, Y . Zhang, and K. Moran, “Guievo: Automated evolution of mobile app UIs,” inProceedings of the 21st International Conference on Mining Software Repositories, ser. MSR ’24. New York, NY , USA: Association for Computing Machinery, 2024, p. 335–347. [On...

    2024 doi

  7. [16]

    Rico: A mobile app dataset for building data-driven design applications,

    B. Deka, Z. Huang, C. Franzen, J. Hibschman, D. Afergan, Y . Li, J. Nichols, and R. Kumar, “Rico: A mobile app dataset for building data-driven design applications,” inProceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’17. New Yo...

    2017

  8. [17]

    Screen2Vec: Semantic embedding of GUI screens and GUI components,

    T. J.-J. Li, L. Popowski, T. Mitchell, and B. A. Myers, “Screen2Vec: Semantic embedding of GUI screens and GUI components,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, ser. CHI ’21. ACM, May 2021. [Online]. Available: http://dx.doi.org/10.1...

    2021

  9. [18]

    VUT: Versatile UI transformer for multi-modal multi-task user interface modeling,

    Y . Li, G. Li, X. Zhou, M. Dehghani, and A. Gritsenko, “VUT: Versatile UI transformer for multi-modal multi-task user interface modeling,” 2021

    2021

  10. [19]

    UIBert: Learning generic multimodal representations for UI understanding,

    C. Bai, X. Zang, Y . Xu, S. Sunkara, A. Rastogi, J. Chen, and B. A. y Arcas, “UIBert: Learning generic multimodal representations for UI understanding,” 2021

    2021

  11. [20]

    Enrico: A high-quality dataset for topic modeling of mobile UI designs,

    L. A. Leiva, A. Hota, and A. Oulasvirta, “Enrico: A high-quality dataset for topic modeling of mobile UI designs,” inProc. MobileHCI Adjunct, 2020

    2020

  12. [21]

    FRAME replication package,

    “FRAME replication package,” Zenodo, https://doi.org/10.5281/zenodo. 21386573, 2026

    2026 doi

  13. [22]

    FRAME source code repository,

    “FRAME source code repository,” https://github.com/SageSELab/ FRAME, 2026

    2026

  14. [23]

    Rips complex,

    C. Maria, P. Dlotko, V . Rouvreau, and M. Glisse, “Rips complex,” inGUDHI User and Reference Manual, 3rd ed. GUDHI Editorial Board, 2023. [Online]. Available: https://gudhi.inria.fr/doc/3.9.0/group rips complex.html

    2023

  15. [24]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervi- sion,” 2021

    2021

  16. [25]

    Pillow (pil fork) documentation,

    A. Clark, “Pillow (pil fork) documentation,” 2015. [Online]. Available: https://buildmedia.readthedocs.org/media/pdf/pillow/latest/pillow.pdf

    2015

  17. [26]

    UIED: UI element detection — detecting UI elements from UI screenshots or drawings,

    M. Xie, “UIED: UI element detection — detecting UI elements from UI screenshots or drawings,” https://github.com/MulongXie/UIED, 2021. [Online]. Available: https://github.com/MulongXie/UIED

    2021

  18. [27]

    Android developers: Accessibility,

    Google, “Android developers: Accessibility,” https://developer.android. com/guide/topics/ui/accessibility, 2022

    2022

  19. [28]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” 2019

    2019

  20. [29]

    Huggingface CLIP library,

    “Huggingface CLIP library,” 2022. [Online]. Available: https:// huggingface.co/docs/transformers/en/model doc/clip

    2022

  21. [30]

    Pytesseract,

    “Pytesseract,” 2022. [Online]. Available: https://github.com/h/pytesseract

    2022

  22. [31]

    Semi-supervised classification with graph convolutional networks,

    T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. [Online]. Available: htt...

    2017

  23. [32]

    Convolutional neural networks on graphs with fast localized spectral filtering,

    M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,” inProceedings of the 30th International Conference on Neural Information Processing Systems, ser. NIPS’16. Red Hook, NY , USA: Curran Associates I...

    2016

  24. [33]

    Vietoris-rips complex: A new direction for cross-domain cold-start recommendation,

    A. Krishna Vajjala, D. Meher, S. Pothagoni, Z. Zhu, and D. Rosenblum, “Vietoris-rips complex: A new direction for cross-domain cold-start recommendation,” inProceedings of the SIAM International Conference on Data Mining (SDM), 2024

    2024

  25. [34]

    Heron’s formula,

    E. Weisstein, “Heron’s formula,”MathWorld, 2003

    2003

  26. [35]

    On lines and planes of closest fit to systems of points in space,

    K. Pearson, “On lines and planes of closest fit to systems of points in space,”The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, vol. 2, no. 11, pp. 559–572, 1901

    1901

  27. [36]

    Guigle: A GUI search engine for Android apps,

    C. Bernal-C ´ardenas, K. Moran, M. Tufano, Z. Liu, L. Nan, Z. Shi, and D. Poshyvanyk, “Guigle: A GUI search engine for Android apps,” in 2019 IEEE/ACM 41st International Conference on Software Engineer- ing: Companion Proceedings (ICSE-Companion), 2019, pp. 71–74

    2019

  28. [37]

    It takes two to tango: Combining visual and textual information for detecting duplicate video-based bug reports,

    N. Cooper, C. Bernal-C ´ardenas, O. Chaparro, K. Moran, and D. Poshyvanyk, “It takes two to tango: Combining visual and textual information for detecting duplicate video-based bug reports,” inProceedings of the 43rd International Conference on Software Engineering, ser. ICSE ’...

    2021

  29. [38]

    Screenai: A vision- language model for ui and infographics understanding,

    G. Baechler, S. Sunkara, M. Wang, F. Zubach, H. Mansoor, V . Etter, V . C˘arbune, J. Lin, J. Chen, and A. Sharma, “Screenai: A vision- language model for ui and infographics understanding,”arXiv preprint arXiv:2402.04615, 2024

    2024 arXiv

  30. [39]

    Graph4GUI: Graph neural networks for representing graphical user interfaces,

    Y . Jiang, C. Zhou, V . Garg, and A. Oulasvirta, “Graph4GUI: Graph neural networks for representing graphical user interfaces,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, ser. CHI ’24. ACM, May 2024, p. 1–18. [Online]. Available: http://dx.doi....

    2024

  31. [40]

    BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” inICML, 2022

    2022

  32. [41]

    UIClip: A data-driven model for assessing user interface design,

    J. Wu, Y .-H. Peng, A. Li, A. Swearngin, J. P. Bigham, and J. Nichols, “UIClip: A data-driven model for assessing user interface design,”arXiv preprint arXiv:2404.12500, 2024

    2024 arXiv

  33. [42]

    Generating automatic feedback on UI mockups with large language models,

    P. Duan, J. Warner, Y . Li, and B. Hartmann, “Generating automatic feedback on UI mockups with large language models,”arXiv [cs.HC], 2024

    2024

  34. [43]

    On using GUI interaction data to improve text retrieval-based bug localization,

    J. Mahmud, N. De Silva, S. A. Khan, S. H. Mostafavi, S. M. H. Mansur, O. Chaparro, A. A. Marcus, and K. Moran, “On using GUI interaction data to improve text retrieval-based bug localization,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering,...

    2024

  35. [44]

    Robust detection of Android UI similarity,

    J. Mao, J. Bian, H. Ma, Y . Jia, Z. Liang, and X. Jiang, “Robust detection of Android UI similarity,” in2018 IEEE International Conference on Communications (ICC), 2018, pp. 1–6

    2018

  36. [45]

    Predicting and explaining mobile UI tappability with vision modeling and saliency analysis,

    E. Schoop, X. Zhou, G. Li, Z. Chen, B. Hartmann, and Y . Li, “Predicting and explaining mobile UI tappability with vision modeling and saliency analysis,” inProceedings of the 2022 CHI Conference on Human Factors in Computing Systems, ser. CHI ’22. New York, NY , USA: Associat...

    2022

  37. [46]

    Screen parsing: Towards reverse engineering of UI models from screenshots,

    J. Wu, X. Zhang, J. Nichols, and J. P. Bigham, “Screen parsing: Towards reverse engineering of UI models from screenshots,” in The 34th Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’21. New York, NY , USA: Association for Computing Machinery, 2021,...

    2021

  38. [47]

    Spotlight: Mobile UI understanding using vision- language models with a focus,

    G. Li and Y . Li, “Spotlight: Mobile UI understanding using vision- language models with a focus,” 2023

    2023

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.