Pith. sign in

REVIEW 4 major objections 5 minor 47 references

MLLM-Based UI2Code Automation Guided by UI Layout Information

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LayoutCoder turns a webpage screenshot into HTML/CSS by first building a layout tree, then asking an MLLM to code each block and fusing the snippets deterministically, beating the strongest baseline by 10.14% BLEU and 3.95% CLIP on average.

desk verdict Solid system, confounded evaluation: the layout-tree contribution isn't isolated from output-budget effects. read the letter →

arxiv 2506.10376 v1 pith:D2NPHG2C submitted 2025-06-12 cs.SE cs.HC

Fan Wu , Cuiyun Gao , Shuqing Li , Xin-Cheng Wen , Qing Liao This is my paper · ORCID
classification cs.SEcs.HC
keywords UI-to-codemultimodallargelanguagemodelslayouttreewebpagescreenshotcodegenerationUIelementgroupingSnap2Codedivfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the reason multimodal LLMs fail at UI2Code is not a lack of code-generation ability but a lack of layout comprehension, and that this weakness can be fixed without retraining. Its proposal, LayoutCoder, detects UI elements, groups structurally similar ones, and recursively divides the screenshot into a layout tree; an MLLM then generates code for each atomic block, and a deterministic fusion algorithm reassembles the snippets into nested flexbox divs. On a new benchmark of 350 real-world sites (Snap2Code, split into seen and unseen) plus the Design2Code benchmark, the authors report average improvements of 10.14% in BLEU and 3.95% in CLIP over the strongest baseline, and a 24.12 BLEU-point gap on the harder seen set. The significance would be a training-free, token-economical path to preserving complex webpage layouts in generated code.

What carries the argument

The load-bearing object is the UI layout tree: a recursive decomposition whose nodes are row containers, column containers, or atomic leaf blocks, each node storing its bounding box, child list, spatial ratio, and eventual code snippet. It is built by projecting detected element blocks onto both axes, sorting the resulting blank gaps by size (an implementation of the Gestalt proximity principle), and dividing the largest gap first, with similar elements pre-grouped so repeated structures are not over-segmented. The tree is what allows the task to be split into easy per-block code generation plus a deterministic fusion pass: Algorithm 2 traverses the tree and emits nested div elements with flex proportions, so the final HTML mirrors the inferred layout rather than relying on the MLLM to reconstruct a whole page at once.

What would settle it

Measure whether the inferred layout tree actually matches the page's DOM structure: run LayoutCoder's parsing on Snap2Code pages and compute a tree-edit distance between each inferred layout tree and the corresponding DOM tree. If many complex pages yield trees that disagree with the DOM nesting, the claimed layout preservation cannot be attributed to the tree; if the agreement is high, the mechanism is doing its stated work.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a projection-based UI layout tree can serve as a structural scaffold that makes an unmodified MLLM produce layout-faithful code. The pipeline first builds an element-relation graph from detected bounding boxes, merges repetitive aligned regions into groups, then recursively splits the screen by the largest blank gaps, recording row/column container nodes and atomic leaf regions. Each atomic image block is cropped and handed to the MLLM with a prompt that forbids fixed sizes and demands a standalone div; the snippets are then woven together by a rule-based fusion that maps each tree node to a flex div with a proportional flex value. The paper reports that this beats the best closed-source baseline on five of six dataset/metric combinations, with the largest gains on pages whose DOM depth, tag count, and aspect ratio are an order of magnitude beyond Design2Code. It also reports that the deterministic fusion avoids the token-window failures seen in a divide-and-conquer baseline, reaching 100% generation success on the new datasets while cutting time and token cost.

Load-bearing premise

The entire argument rests on the layout tree being a faithful picture of the page's true visual layout: the tree is built from detected bounding boxes and blank-gap divisions, and if that tree misrepresents a page, the fused code cannot preserve layout no matter how good each code snippet is.

Editorial extensions

If this is right

  • If the reported gains hold, layout guidance of this kind removes the need to fine-tune a model on UI data to get layout-faithful code; the same MLLM works on unseen sites.
  • Because fusion is deterministic and the MLLM only sees atomic blocks, generated pages are no longer bounded by the model's context window, so arbitrarily long and tall pages can be produced.
  • The pipeline cuts token use, time, and dollar cost relative to MLLM-based divide-and-conquer fusion, which makes repeated UI-to-code passes more practical.
  • With the Snap2Code benchmark split by registration date, future UI2Code systems can be tested for leakage rather than relying only on datasets the model may have memorized.
  • The complexity-binned results suggest the method's advantage grows with page height and tag count, so the approach is aimed exactly at the real-world pages where direct MLLM prompting collapses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: compare the inferred layout tree against the page's actual DOM tree; if tree fidelity explains the BLEU/CLIP gains, then a better parser (or DOM ground truth) should push scores further, whereas if it does not, the gains come from block-wise cropping itself.
  • The same tree could be reused as a structured prompt or as a constraint during MLLM decoding, not just as a cropping-and-fusion scaffold, which would likely reduce snippet inconsistency further.
  • The grouping heuristic assumes left-aligned, evenly spaced elements, so pages using centered or masonry layouts may force over-segmentation; adapting the grouping criteria could extend the method to more design languages.
  • If BLEU/CLIP are only proxies, the human-evaluation gap (LayoutCoder near 4 out of 5 where Claude(SR) sits near 1.6 on complex pages) is the more consequential evidence, and a targeted usability study with developers would tell whether the generated code is actually maintainable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LayoutCoder, a pipeline that converts a webpage screenshot into HTML/CSS by (i) detecting and grouping UI elements with bounding boxes and heuristic alignment/spacing rules, (ii) recursively dividing the image into a layout tree via 2D block projection and gap sorting, and (iii) generating per-block code snippets with GPT-4o and fusing them into a nested flexbox div tree with a non-MLLM deterministic fusion algorithm (Algorithm 2). The authors introduce Snap2Code, a 350-page benchmark with seen and unseen splits, and compare against nine baselines on Design2Code and Snap2Code using BLEU and CLIP scores, plus ablations, human evaluation, and a cost comparison with DCGen. They claim average improvements of 10.14% in BLEU and 3.95% in CLIP over the best baseline, Claude(SR).

Significance. If the central claim were established, the paper would be a useful contribution: it offers a practical divide-and-conquer decomposition for MLLM-based UI2Code, introduces a new benchmark containing real images and an unseen split, includes a cost and success-rate comparison with DCGen, and uses human evaluation to support visual fidelity. The deterministic tree-based fusion is an interesting alternative to MLLM-based fusion. I find no circularity in the evaluation because the layout tree is derived from the screenshot and measured against external reference code and rendered screenshots. However, the empirical evidence as reported does not isolate the contribution of the layout tree, because the main comparison is confounded with output-length budget and divide-and-conquer; the significance is therefore conditional on additional control experiments.

major comments (4)
  1. [§5.1, Table 3] The headline numbers are not relative percentages. The average 10.14% BLEU improvement is the mean of absolute score-point differences (Design2Code: 4.93−2.66=2.27; Snap2Code(Seen): 26.07−1.95=24.12; Snap2Code(Unseen): 6.46−2.45=4.01), not a relative gain. Likewise, the 3.95% CLIP claim is the signed mean of point differences and includes a negative gap on Design2Code (81.58 vs 82.74), where Claude(SR) is better. The text should report point differences or relative percentages consistently; as written, the abstract and Section 5.1 overstate the average improvement and hide a loss on one dataset.
  2. [§4.3 and §5.1] The main comparison is confounded. LayoutCoder is a multi-block pipeline with effectively unlimited output length, while GPT-4o and Claude baselines are capped at max_tokens=4096. Table 2 shows Snap2Code(Seen) reference lengths have a mean of 4379 tokens and a maximum of 53682 tokens, so a large fraction of baseline outputs are truncated. BLEU's brevity penalty then suppresses baseline scores for reasons unrelated to layout understanding. The 24.12-point Seen BLEU gap therefore cannot be attributed to layout-tree guidance. At minimum, the closed-source baselines should be run without the 4096 cap or with a comparable token budget, and the paper should report the truncation rate.
  3. [§5.2, Table 4] No ablation removes the layout tree. The three variants remove UI grouping, gap sorting, and the snippet-generation prompt, but all still use the projection-based recursive division and the Algorithm 2 tree-based fusion. Since LayoutCoder differs from the baselines in two major dimensions—per-block MLLM generation and tree-based fusion—the observed gains could come from divide-and-conquer plus a larger output budget alone. A control that uses the same per-block GPT-4o generation and fuses snippets by reading order or a fixed grid, without the projection tree, is needed to support the causal claim that layout information drives the improvement.
  4. [§3.3–§3.5] The paper does not validate that the extracted layout tree corresponds to the true page structure. The tree is built from UIED bounding boxes, alignment and spacing thresholds in CanAddToGroup, and a recursive-division stopping criterion, all of which involve free parameters, and Algorithm 2 renders the tree as nested flexbox divs. If the tree misrepresents a page, the fused output cannot preserve the layout regardless of snippet quality. The authors should add a direct evaluation of tree quality (e.g., comparison against the ground-truth DOM hierarchy) and a sensitivity analysis for the key thresholds; without this, the mechanism underlying the empirical gains remains unverified.
minor comments (5)
  1. [§5.2.2, Table 4] The variant labeled 'w/o Recursive Division' actually removes only gap sorting, not recursive division; the label should be 'w/o Gap Sorting', and the reported '0.19%' and '6.30%' improvements are point differences, not relative percentages.
  2. [§3.3 and §3.4.2] There are typographical errors in the section headings: 'Constuction' should be 'Construction' and 'Recusive Division' should be 'Recursive Division'.
  3. [§4.1 and §5.4] The number of human evaluators is inconsistent: Section 4.1 says three participants, while Section 5.4 says four participants; this discrepancy should be resolved.
  4. [§6.3, Tables 7–8] The DCGen comparison reports a 50-sample subset for Snap2Code(Seen) and excludes failed DCGen cases, but the sample sizes in Table 8 (labeled 223, 30, and 91) are not explained in the surrounding text; please state these sample sizes explicitly.
  5. [§5.1, Table 3] In the table, some numeric cells appear concatenated (e.g., the Claude(SR) row on Design2Code shows '2.6682.74'); please fix the formatting so values are clearly separated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: LayoutCoder's layout tree is computed from the input screenshot and the headline scores are measured against external references, while the only author-overlap citation (DCGen) is empirically benchmarked rather than used as load-bearing evidence.

full rationale

The paper's central claim is that the layout-tree-guided pipeline improves BLEU and CLIP scores relative to nine baselines on Design2Code and the new Snap2Code benchmark. The derivation chain is self-contained with respect to the evaluation: the UI layout tree is constructed from UIED bounding boxes and 2D block projections of the input screenshot (Sections 3.3 and 3.4), not from the reference HTML, the reference screenshot, or the BLEU/CLIP metrics, so the reported scores are not defined in terms of the method's own outputs. The code-fusion algorithm (Algorithm 2) is a deterministic rendering of that tree, and its quality is checked against external ground-truth code and rendered screenshots. The only self-citation overlap is DCGen (reference [32], co-authored by Shuqing Li), but the paper re-runs DCGen empirically and compares against it in Section 6.3 and Table 8, and no uniqueness or correctness theorem is imported from that citation. The skeptical concern that the baseline token cap and the absence of a tree-removal ablation confound the causal attribution of the improvement is a legitimate experimental-design risk, but it is not a case where a prediction reduces by construction to a fitted input or to an unverified self-citation. Accordingly, no circularity step is identified and the score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on assumptions about the accuracy of the UIED detector, the validity of proximity-based recursive division, the expressiveness of row/column flexbox templates, and the reliability of BLEU/CLIP as evaluation proxies. Three implementation parameters (grouping tolerances, node portion computation, and division stopping criteria) are unspecified in the preprint, making exact reproduction dependent on the authors' code.

free parameters (3)
  • alignment and spacing tolerances in CanAddToGroup = unspecified
    The grouping criterion in Section 3.3.2 requires 'consistent' spacing and left alignment; the implementation must choose numeric tolerances, which affect group boundaries and the resulting layout tree.
  • layout tree node 'portion' (flex ratio) = unspecified
    Algorithm 2 reads node.get('portion', 1) to set the flex style of each div; Section 3.4.3 says the attribute table records 'spatial ratio' but does not define how the ratio is computed from bounding boxes, so the relative sizes of fused sections are not reproducible from the text.
  • recursive division stopping criterion = unspecified
    Section 3.4.2 splits 'until no further divisions are possible' without specifying minimum gap or region size thresholds, controlling the granularity of atomic regions and hence the number of MLLM calls.
assumptions (5)
  • domain assumption UIED provides accurate and complete element bounding boxes on real-world webpages
    Section 3.3.1 relies on UIED output to build the relation graph and projection intervals; missed or merged elements propagate to the layout tree.
  • domain assumption Largest gaps correspond to layout boundaries (Gestalt proximity)
    Section 3.4.2 sorts projection intervals by descending distance and treats the largest as the next division line, assuming spatial semantics follow visual proximity.
  • domain assumption Any real-world webpage layout can be represented as a row/column tree of nested flexbox divs
    Algorithm 2 converts the layout tree into nested divs with flex styles; layouts using absolute positioning, overlapping elements, or complex CSS grids may not map to this representation.
  • domain assumption MLLMs generate accurate code snippets for small atomic image blocks
    Section 3.5.1 splits the page into crops and relies on GPT-4o to produce valid div-delimited snippets for each crop before deterministic fusion.
  • domain assumption BLEU and CLIP scores are valid proxies for UI2Code quality
    Section 4.4 uses BLEU on code and CLIP on rendered images; Section 6.4.3 acknowledges this as a threat to validity, noting the metrics may not capture UI2Code nuances.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MLLM-Based UI2Code Automation Guided by UI Layout Information." pith.science (2026). https://pith.science/paper/D2NPHG2C

@misc{pith2026250610376,
  author       = {Pith},
  title        = {Pith review of: MLLM-Based UI2Code Automation Guided by UI Layout Information},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D2NPHG2C}},
  note         = {Machine review of arXiv:2506.10376}
}
read the original abstract

Converting user interfaces into code (UI2Code) is a crucial step in website development, which is time-consuming and labor-intensive. The automation of UI2Code is essential to streamline this task, beneficial for improving the development efficiency. There exist deep learning-based methods for the task; however, they heavily rely on a large amount of labeled training data and struggle with generalizing to real-world, unseen web page designs. The advent of Multimodal Large Language Models (MLLMs) presents potential for alleviating the issue, but they are difficult to comprehend the complex layouts in UIs and generate the accurate code with layout preserved. To address these issues, we propose LayoutCoder, a novel MLLM-based framework generating UI code from real-world webpage images, which includes three key modules: (1) Element Relation Construction, which aims at capturing UI layout by identifying and grouping components with similar structures; (2) UI Layout Parsing, which aims at generating UI layout trees for guiding the subsequent code generation process; and (3) Layout-Guided Code Fusion, which aims at producing the accurate code with layout preserved. For evaluation, we build a new benchmark dataset which involves 350 real-world websites named Snap2Code, divided into seen and unseen parts for mitigating the data leakage issue, besides the popular dataset Design2Code. Extensive evaluation shows the superior performance of LayoutCoder over the state-of-the-art approaches. Compared with the best-performing baseline, LayoutCoder improves 10.14% in the BLEU score and 3.95% in the CLIP score on average across all datasets.

Figures

Figures reproduced from arXiv: 2506.10376 by the authors.

Figure 1
Figure 1. Limitations of MLLMs. Websites play a pivotal role in human digital life, serving as a channel for information retrieval and a platform for business transactions, social entertainment, and education. As of 2024, there are approximately 1.1 billion active websites worldwide, with around 252,000 new websites emerging daily [31]. Converting UI designs into code is a crucial step in web development, known as UI2Code. Ho… view at source ↗
Figure 2
Figure 2. Examples illustrating the difference between screenshots(a), hand-drawn sketches(b) and design [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The architecture of LayoutCoder. evaluated based on both textual and visual similarity. Specifically, 𝐶𝑔 should resemble 𝐶𝑜 in terms of HTML’s nested structure and tags, while 𝐼𝑔 should visually align with 𝐼𝑜 . 3.2 Overview The proposed LayoutCoder framework consists of three major modules: Element Relation Con￾struction, UI Layout Parsing, and Layout-Guided Code Fusion [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Illustration of segmentation by SAM(b) and LayoutCoder(c). This module addresses the challenge of accurately segmenting the layout of UI images. Before designing this module, we conducted some preliminary studies on layout segmentation using existing image segmentation…
Figure 5
Figure 5. Figure 5: An example of UIED and 2D Block Projection processing a webpage image. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Two examples from Design2Code(a) and Snap2Code(b). [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Box plots showing the distribution of BLEU and CLIP scores for LayoutCoder and baseline methods. [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: The performance of LayoutCoder across the four complexity features for both datasets. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: An example illustrating the difference between the web pages generated by LayoutCoder and GPT-4o. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Examples illustrating the differences between the web pages generated by LayoutCoder and its [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 21 canonical work pages

  1. [1]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al. 2024. Flamingo: a visual language model for few-shot learning. InProceedings of the 36th International Conference on Neural Information Processing Systems(New Orleans, LA, USA) (NIPS ’22). Curran Associates Inc., Red Hook, NY, USA, Article 1723, 21 pages

    work page 2024

  2. [2]

    Anthropic. 2024. Claude 3.5. https://www.anthropic.com. Accessed: 2024-10-30

    work page 2024

  3. [3]

    Batuhan Aşıroğlu, Büşta Rümeysa Mete, Eyyüp Yıldız, Yağız Nalçakan, Alper Sezen, Mustafa Dağtekin, and Tolga Ensari. 2019. Automatic HTML code generation from mock-up images using machine learning techniques. In2019 Scientific Meeting on Electrical-Electronics & Biomedical Engineering and Computer Science (EBBT). Ieee, 1–4

    work page 2019

  4. [4]

    2018.Turn your whiteboard sketches to working code in seconds with sketch2code

    Microsoft Azure. 2018.Turn your whiteboard sketches to working code in seconds with sketch2code. https://azure. microsoft.com/en-us/blog/turn-your-whiteboard-sketches-to-working-code-in-seconds-with-sketch2code/ Accessed: 2024-10-30

    work page 2018

  5. [5]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966 [cs.CV] https://arxiv.org/abs/2308.12966

    arXiv 2023

  6. [6]

    Daniel Baulé, Christiane Gresse von Wangenheim, Aldo von Wangenheim, Jean C. R. Hauck, and Edson C. Vargas Júnior. 2021. Automatic code generation from sketches of mobile applications in end-user development using Deep Learning. arXiv:2103.05704 [cs.HC] https://arxiv.org/abs/2103.05704

    work page Pith review arXiv 2021

  7. [7]

    Tony Beltramelli. 2018. pix2code: Generating Code from a Graphical User Interface Screenshot. InProceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems(Paris, France)(EICS ’18). Association for Computing Machinery, New York, NY, USA, Article 3, 6 pages. doi:10.1145/3220134.3220135

    arXiv 2018

  8. [8]

    Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI design image to GUI skeleton: a neural machine translator to bootstrap mobile GUI implementation. InProceedings of the 40th International Conference on Software Engineering(Gothenburg, Sweden)(ICSE ’18). Association for Computing Machinery, New York, NY, USA, 665–676. doi:10....

    arXiv 2018

Show all 47 references
  1. [9]

    Wen-Yin Chen, Pavol Podstreleny, Wen-Huang Cheng, Yung-Yao Chen, and Kai-Lung Hua. 2022. Code generation from a graphical user interface via attention-based encoder–decoder model.Multimedia Systems28, 1 (2022), 121–130

    2022

  2. [10]

    André Armstrong Janino Cizotto, Rodrigo Clemente Thom de Souza, Viviana Cocco Mariani, and Leandro dos Santos Coelho. 2023. Web pages from mockup design based on convolutional neural network and class activation mapping. 82, 25 (March 2023), 38771–38797. doi:10.1007/s11042-023-15108-3

    2023 doi

  3. [11]

    Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. InProceedings of the 30th Annual ACM Symposium on User Interface Software and...

    2017

  4. [12]

    Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise(KDD’96). AAAI Press, 226–231

    1996

  5. [13]

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2022. CLIPScore: A Reference-free Evaluation Metric for Image Captioning. arXiv:2104.08718 [cs.CV] https://arxiv.org/abs/2104.08718

    2022 arXiv

  6. [14]

    Vanita Jain, Piyush Agrawal, Subham Banga, Rishabh Kapoor, and Shashwat Gulyani. 2019. Sketch2Code: Transforma- tion of Sketches to UI in Real-time Using Deep Neural Network. arXiv:1910.08930 [cs.CV] https://arxiv.org/abs/1910. 08930

    2019 arXiv

  7. [15]

    Zhaoyun Jiang, Jiaqi Guo, Shizhao Sun, Huayu Deng, Zhongkai Wu, Vuksan Mijovic, Zijiang James Yang, Jian-Guang Lou, and Dongmei Zhang. 2023. LayoutFormer++: Conditional Graphic Layout Generation via Constraint Serialization and Decoding Space Restriction . In2023 IEEE/CVF Conf...

    2023

  8. [16]

    Zhaoyun Jiang, Shizhao Sun, Jihua Zhu, Jian-Guang Lou, and Dongmei Zhang. 2022. Coarse-to-Fine Generative Modeling for Graphic Layouts.Proceedings of the AAAI Conference on Artificial Intelligence36, 1 (Jun. 2022), 1096–1103. doi:10.1609/aaai.v36i1.19994

    2022 doi

  9. [17]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment Anything. arXiv:2304.02643 [cs.CV] https://arxiv.org/abs/2304.02643

    2023 arXiv

  10. [18]

    Hugo Laurençon, Léo Tronchon, and Victor Sanh. 2024. Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset. arXiv:2403.09029 [cs.HC] https://arxiv.org/abs/2403.09029

    2024 arXiv

  11. [19]

    Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, et al. 2023. Pix2Struct: screenshot parsing as pretraining for visual language understanding. InProceedings of the 40th International Conference on Machine Learning(Honolulu, Hawaii, USA)(ICML’23). JMLR.org, Article...

    2023

  12. [20]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv:2301.12597 [cs.CV] https://arxiv.org/abs/2301.12597 Proc. ACM Softw. Eng., Vol. 2, No. ISSTA, Article ISST...

    2023 arXiv

  13. [21]

    Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. 2019. LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators. arXiv:1901.06767 [cs.CV] https://arxiv.org/abs/1901.06767

    2019 arXiv

  14. [22]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning.ArXivabs/2304.08485 (2023). https://api.semanticscholar.org/CorpusID:258179774

    2023 arXiv

  15. [23]

    Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps.IEEE Transactions on Software Engineering 46 (2018), 196–221. https://api.semanticscholar.org/Corpus...

    2018

  16. [24]

    Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI (T).2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE)(2015), 248–259. https://api.semanticscholar.org/CorpusID:7499368

    2015

  17. [25]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, et al . 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv.org/abs/2303.08774

    2024 arXiv

  18. [26]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. InProceedings of the 40th Annual Meeting on Association for Computational Linguistics(Philadelphia, Pennsylvania)(ACL ’02). Association for Comput...

    2002

  19. [27]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. InInternational conference on machine learnin...

    2021

  20. [28]

    Alex Robinson. 2019. Sketch2code: Generating a website from a paper mockup. arXiv:1905.13750 [cs.CV] https: //arxiv.org/abs/1905.13750

    2019 arXiv

  21. [29]

    Andy Rutledge. 2009. Gestalt Principles - 3: Proximity, Uniform Connectedness, and Good Continuation. https: //andyrutledge.com/gestalt-principles-3.html. Accessed: 2024-10-30

    2009

  22. [30]

    Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. 2024. Design2Code: How Far Are We From Automating Front-End Engineering?arXiv preprint arXiv:2403.03163(2024)

    2024 arXiv

  23. [31]

    Digital Silk. 2024. How Many Websites Are There In 2024? https://www.digitalsilk.com/digital-trends/how-many- websites-are-there/. Accessed: 2024-10-31

    2024

  24. [32]

    Yuxuan Wan, Chaozheng Wang, Yi Dong, Wenxuan Wang, Shuqing Li, Yintong Huo, and Michael R. Lyu. 2024. Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach. arXiv:2406.16386 [cs.SE] https://arxiv.org/abs/2406.16386

    2024 arXiv

  25. [33]

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. 2023. Cogvlm: Visual expert for pretrained language models.arXiv preprint arXiv:2311.03079(2023)

    2023 arXiv

  26. [34]

    Fan Wu. 2025. MLLM-Based UI2Code Automation Guided by UI Layout Information. https://github.com/ay7u1009/ LayoutCoder/. Accessed: 2025-04-05

    2025

  27. [35]

    Jason Wu, Xiaoyi Zhang, Jeff Nichols, and Jeffrey P Bigham. 2021. Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots. InThe 34th Annual ACM Symposium on User Interface Software and Technology(Virtual Event, USA)(UIST ’21). Association for Computing Machi...

    2021

  28. [36]

    Shuhong Xiao, Yunnong Chen, Jiazhi Li, Liuqing Chen, Lingyun Sun, and Tingting Zhou. 2024. Prototype2Code: End-to- end Front-end Code Generation from UI Design Prototypes. arXiv:2405.04975 [cs.SE] https://arxiv.org/abs/2405.04975

    2024 arXiv

  29. [37]

    Shuhong Xiao, Yunnong Chen, Yaxuan Song, Liuqing Chen, Lingyun Sun, Yankun Zhen, and Yanfang Chang. 2024. UI Semantic Group Detection: Grouping UI Elements with Similar Semantics in Mobile Graphical User Interface. arXiv:2403.04984 [cs.SE] https://arxiv.org/abs/2403.04984

    2024 arXiv

  30. [38]

    Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. 2020. UIED: a hybrid tool for GUI element detection(ESEC/FSE 2020). Association for Computing Machinery, New York, NY, USA, 1655–1659. doi:10.1145/3368089.3417940

    2020

  31. [39]

    Mulong Xie, Zhenchang Xing, Sidong Feng, Xiwei Xu, Liming Zhu, and Chunyang Chen. 2022. Psychologically- inspired, unsupervised inference of perceptual groups of GUI widgets from GUI images(ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA, 332–343. doi:10...

    2022

  32. [40]

    Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. 2018. Graph R-CNN for Scene Graph Generation. InProceedings of the European Conference on Computer Vision (ECCV)

    2018

  33. [41]

    Ni, and Heung-Yeung Shum

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung-Yeung Shum. 2022. DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection. arXiv:2203.03605 [cs.CV] https: //arxiv.org/abs/2203.03605

    2022 arXiv

  34. [42]

    Junyi Zhang, Jiaqi Guo, Shizhao Sun, Jian-Guang Lou, and D. Zhang. 2023. LayoutDiffusion: Improving Graphic Layout Generation by Discrete Diffusion Probabilistic Models.2023 IEEE/CVF International Conference on Computer Vision (ICCV)(2023), 7192–7202. https://api.semanticschol...

    2023

  35. [43]

    Wei Zhang, Shangmin Luan, Liqin Tian, and Nima Jafari Navimipour. 2022. A Rapid Combined Model for Automatic Generating Web UI Codes.Wirel. Commun. Mob. Comput.2022 (Jan. 2022), 10 pages. doi:10.1155/2022/4415479 Proc. ACM Softw. Eng., Vol. 2, No. ISSTA, Article ISSTA050. Publ...

    2022 doi

  36. [44]

    Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, et al. 2021. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels.. InCHI, Yoshifumi Kitamura, Aaron Quigley, Katherine Isbister, Takeo Igarashi, Pernille Bjørn, and Steven Mark Drucker (Eds.)....

    2021

  37. [45]

    Shuyu Zheng, Ziniu Hu, and Yun Ma. 2019. FaceOff: Assisting the Manifestation Design of Web Graphical User Interface. InProceedings of the Twelfth ACM International Conference on Web Search and Data Mining(Melbourne VIC, Australia) (WSDM ’19). Association for Computing Machine...

    2019

  38. [46]

    Xu Zhong, Jianbin Tang, and Antonio Jimeno-Yepes. 2019. PubLayNet: Largest Dataset Ever for Document Layout Analysis.2019 International Conference on Document Analysis and Recognition (ICDAR)(2019), 1015–1022. https: //api.semanticscholar.org/CorpusID:201124789

    2019

  39. [47]

    Ting Zhou, Yanjie Zhao, Xinyi Hou, Xiaoyu Sun, Kai Chen, and Haoyu Wang. 2024. Bridging Design and Development with Automated Declarative UI Code Generation. arXiv:2409.11667 [cs.SE] https://arxiv.org/abs/2409.11667 Received 2024-10-31; accepted 2025-03-31 Proc. ACM Softw. Eng...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.