Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FullFront shows top MLLMs fall far short of human experts across the full front-end workflow.

desk verdict Useful benchmark with a solid perception gap result; the code-generation numbers rest on unverified LLM-generated ground truth, so those claims are softer than they appear. read the letter →

arxiv 2505.17399 v2 pith:5HKDSMBZ submitted 2025-05-23 cs.CL

Haoyu Sun , Huichen Will Wang , Jiawei Gu , Linjie Li , Yu Cheng This is my paper
classification cs.CL
keywords MLLMbenchmarkfront-endengineeringwebpagecodegenerationperceptionQAinteractionauthoringHTMLreconstructionmultimodalevaluationtext-to-imagedesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FullFront is a benchmark that evaluates multimodal large language models on the three stages of front-end engineering: turning a text brief into a visual design, perceiving the layout and elements of a rendered webpage, and generating or refining the HTML/CSS/JavaScript that implements it. The paper's central claim is that current MLLMs are far from expert-level across this full pipeline: the best model, Claude 3.7 Sonnet, averages below 55% accuracy on perception QA while human experts exceed 95%, and closed models score 8 or higher on human-rated visual fidelity for code generation while open models fall below 6.25. The benchmark reconstructs real-world webpages into clean, standardized HTML through a two-stage LLM pipeline, with a category-based image scheme that avoids placeholders and copyright issues. If the measurement is accepted, it gives the field a single instrument for tracking progress from idea to design to code, and it identifies fine-grained perception of alignment, size, and spacing as the main bottleneck.

What carries the argument

The load-bearing mechanism is the two-stage HTML reconstruction pipeline combined with a multi-metric evaluation. Starting from a real webpage screenshot, OmniParser extracts element boxes and content, GPT-4o writes an initial HTML-v1, and Claude 3.7 Sonnet refines it into HTML-v2; that HTML-v2 and its rendered screenshot serve as ground truth. Image content is mapped to 15 predefined categories with fixed non-copyrighted URLs, forcing models to perceive image type and set sizes and positions themselves. Evaluation uses CLIP Score, a Gemini Visual Score across ten fine-grained criteria, and a DOM-based Code Score that combines structural Longest Common Subsequence similarity with implementation-rate-adjusted similarity for text, images, and forms. This machinery converts real pages into clean, dependency-free references while preserving visual diversity that other datasets lack.

What would settle it

Take a random sample of FullFront's HTML-v2 reference pages and have front-end engineers manually reconstruct them from the original real-world screenshots; if model rankings or absolute scores substantially change when measured against this human-verified gold set, the benchmark's claim to measure real front-end reproduction would be refuted. A cheaper check is to audit a few dozen reference pages for layout errors against the original screenshots.

Watch

Extended reading notes

Core claim

The central discovery is that MLLM competence does not transfer smoothly across the front-end workflow. Three tasks map directly to the pipeline: Webpage Design for conceptualization, Webpage Perception QA for comprehension, and Webpage Code Generation for implementation, with the last split into Image to Code, Text to Code, Interaction Authoring, and Code Refinement. Across ten models, proprietary systems clearly outperform open-source ones on code generation, but perception QA shows no such gap, and error analysis concentrates failures in alignment (21.5%), size (19.5%), spacing (15.5%), and precise positioning (18.5%). Counterintuitively, models that answer perception questions incorrectly can still place those same elements correctly when generating code, suggesting perception and implementation use different internal routes. The paper concludes that even the best models are not yet reliable enough to act as front-end engineers without tool support, because they still produce abnormal image sizes, blank pages, and isolated interaction elements.

Load-bearing premise

The load-bearing premise is that the HTML-v2 pages used as ground truth are accurate reconstructions of the original real-world webpages, but they were produced automatically by GPT-4o and Claude 3.7 Sonnet and are not stated to have been manually verified, so all code-generation scores measure fidelity to an AI-generated reference.

Editorial extensions

If this is right

  • Current MLLMs cannot be trusted to reproduce fine-grained webpage layouts from screenshots without human review, since even top models make alignment, sizing, and spacing errors.
  • Open-source models are not yet competitive with proprietary ones on code generation, so cost-efficient automation of front-end implementation remains out of reach.
  • Interaction authoring is the hardest code subtask; models can often reproduce static pages but fail to implement dynamic behaviors or produce isolated interaction elements rather than complete pages.
  • Because perception QA rankings and code-generation rankings diverge, improving a model's perception accuracy will not automatically improve its code output, and separate evaluation of both is necessary.
  • The benchmark and its mini subset provide a reusable way to track progress across the whole workflow with relatively cheap iterative testing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: give models the perception QA questions while they generate code and check whether correcting their answers raises visual fidelity; the paper's observed perception-generation divergence suggests it may not.
  • If fine-grained perception is the binding constraint, tool-augmented systems that verify alignment, size, and spacing programmatically against the DOM could close much of the measured gap without retraining the base models.
  • The category-based image URL scheme could become a standard control in screenshot-to-code benchmarks, making image handling comparable across datasets.
  • Because the ground-truth HTML-v2 is itself LLM-generated, absolute code scores should be read as fidelity to an AI reconstruction; rankings could shift if a human-verified reference set were introduced.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. FullFront proposes a three-part benchmark for multimodal LLMs in front-end engineering: 50 Webpage Design problems, 1800 Webpage Perception QA questions (Real-world, Synthetic, Multi-window), and 400 Webpage Code Generation problems (Image-to-Code, Text-to-Code, Interaction Authoring, Code Refinement). The data are derived from real webpages through an OmniParser/GPT-4o/Claude 3.7 Sonnet pipeline, and models are evaluated with CLIP score, a Gemini-based visual score, a DOM-based Code Score, and human evaluation. The authors report that even the best MLLMs average below 55% accuracy on perception QA while human experts exceed 95%, that closed-source models generally outperform open-source models on code generation, and that image handling, layout fidelity, and interaction implementation remain weak.

Significance. If the result holds, FullFront would be a useful benchmark spanning the conceptualization, comprehension, and implementation phases of front-end work. Strengths include the breadth of the task suite, the two-stage LLM pipeline that converts real screenshots into clean standardized HTML, the category-based image strategy that avoids generic placeholders, the combination of automatic and human metrics, and the explicit error analyses in Section 5 and Tables 6-7. The perception-QA finding appears robust because the questions were manually reviewed and human accuracy is very high. However, the code-generation claims currently rest on unverified AI-generated references, on metric details that are not fully specified, and on human evaluations whose protocol is not documented. The headline gap for implementation quality is therefore not yet securely established.

major comments (3)
  1. [Section 3.1] The code-generation tasks treat the HTML-v2 produced by Claude 3.7 Sonnet as ground truth, but no manual verification of these pages against the original real-world screenshots is reported. The paper states manual review for the QA questions, Text-to-Code descriptions, and interaction code, but not for the HTML-v2 references used in Image-to-Code, Text-to-Code, and Code Refinement. Because GPT-4o and Claude 3.7 Sonnet are among the evaluated models, the scores in Table 3 may partly measure how well each model reproduces an artifact produced by the same model families. I request a validation study: human experts should check a random sample of HTML-v2 pages against the source screenshots, report the rate and types of layout/style discrepancies, and either correct the references or re-analyze the scores on a verified subset.
  2. [Tables 2 and 4; Section 5.3] No evaluation protocol is given for the human numbers. In Table 2, the 'Human Expert' rows (0.97, 0.96, 0.94) are central to the claim that MLLMs fall far below human performance, yet the paper does not state how many experts answered how many questions, whether they saw the original screenshots, or how disagreements were adjudicated. Section 5.3 mentions 'three human experts' and Figure 6 reports win/tie/lose percentages, but the rubric, selection procedure, and inter-annotator agreement are not described for Table 4 or Figure 6. Without this methodology, the human-gap quantification is not reliable. Please provide the full protocol and per-expert scores.
  3. [Appendix C.2, Eq. (7)] The Code Score is defined as a weighted sum of structural similarity and adjusted content-type similarities, using weights Wstruct, Wtext_content, Wtext_style, Wimage, and Wform, with a matching threshold of 0.9. However, the weight values are never reported, and the threshold is only mentioned in passing. Since the Code Score is one of the three headline metrics in Table 3, the benchmark is not reproducible or comparable without these constants. Please list the exact weights and include a sensitivity analysis showing, for example, that perturbing Wstruct by plus or minus 0.1 does not change the model ranking.
minor comments (6)
  1. [Section 4.1] The text says 'six proprietary models' but then lists seven: Claude 3.7 Sonnet, Gemini 2.5 Flash, GPT-4o, o4-mini, GPT-4.1, o1, and Gemini 2.5 Pro. Please correct the count or the list.
  2. [Table 6] The column headers contain typos: 'Blacnk' should be 'Blank', 'Isonlation' should be 'Isolation', and 'Instruct' in the model name should be 'Instruct'.
  3. [Table 3] The mini-dataset block uses the header 'Enh' for what is called 'Ref' (Code Refinement) in the full-dataset block. Please standardize the naming.
  4. [References] Reference [37] has a malformed URL: 'https://https://www.anthropic.com/news/claude-3-7-sonnet' should be a single 'https://'.
  5. [Appendix A] The FullFront-mini description mentions '20 Code Authoring' samples, which should be 'Interaction Authoring' to match the task name used elsewhere.
  6. [Tables 2 and 3] No confidence intervals, repeated runs, or significance tests are reported, and several subtasks in FullFront-mini have only 10 samples. Adding variance estimates or at least acknowledging this limitation would strengthen the comparisons.

Circularity Check

1 steps flagged · score 4.0 of 10

Code-generation ground truth is LLM-generated (GPT-4o + Claude 3.7 Sonnet) and never human-verified, so part of the measured gap is fidelity to artifacts of the same model families being benchmarked.

  1. self definitional [Section 3.1 (Webpage Code Generation); evaluated in Tables 3–5]
    "This two-stage process ... starts with a real-world webpage screenshot and its OmniParser-extracted element information. GPT-4o generates an initial HTML-v1, which Claude 3.7 Sonnet then refines—adjusting styles, positions, alignments, and layouts—into a higher-quality, more complex HTML-v2. This HTML-v2 and its rendered page serve as ground truth."

    The paper defines the ground truth for all four code-generation subtasks as HTML-v2 produced by GPT-4o (HTML-v1) and refined by Claude 3.7 Sonnet, and the rendered page of that HTML-v2. The models evaluated in Section 4.1 include the same two model families (Claude 3.7 Sonnet and GPT-4o). The Code Score and Gemini Visual Score compare each model's output to this reference; therefore, the code-generation scores measure similarity to LLM-generated artifacts rather than to independently human-verified page fidelity. The paper explicitly reports manual review for QA questions, Text-to-Code descriptions, and interaction code, but nowhere states that HTML-v2 itself was human-verified against the original real-world page.

full rationale

FullFront is an empirical benchmark rather than a formal derivation, and much of its evidence is independent: the Webpage Perception QA results compare MLLMs against human experts on manually reviewed questions, and human evaluations (Table 4, Figure 6) are external judgments. No load-bearing self-citation chain or fitted-parameter-as-prediction pattern is present. The only circular tendency is in the code-generation branch: ground-truth HTML-v2 is written by GPT-4o and refined by Claude 3.7 Sonnet, and the benchmark then scores Claude 3.7 Sonnet, GPT-4o, and other models by fidelity to that artifact. This makes part of the code-generation ranking a self-comparison with model-family outputs and weakens the claim that absolute scores (e.g., Gemini Visual Score 8.93 for Claude 3.7 Sonnet on Image-to-Code) are calibrated to true webpages. The perception-QA limitation finding remains valid, so the overall circularity is partial rather than total.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

No new physical or conceptual entities are introduced. The fifteen image categories and ten interaction types are taxonomies for benchmark construction, not inventions with independent empirical handles. The listed free parameters and axioms are the load-bearing design choices that shape all reported scores.

free parameters (3)
  • Code Score aggregation weights = not reported
    Equation (7) combines structural, text-content, text-style, image, and form similarities with weights Wstruct, Wtext_content, Wtext_style, Wimage, Wform that are 'predefined' but never given numerically; the ranking of models depends on these choices.
  • Element matching threshold theta_match = 0.9
    Used in both LCS structural similarity and content-type element matching (Appendix C.2); chosen by hand, not justified.
  • Number of image categories = 15
    The category-based image strategy defines 15 classes (People, Animal, Food, etc.); the granularity of these categories affects both the ground truth generation and the evaluation of image handling.
assumptions (6)
  • domain assumption OmniParser provides accurate bounding boxes and OCR text for web screenshots.
    The pipeline uses OmniParser-extracted element information to help GPT-4o generate HTML-v1 and to help GPT-4o generate QA questions (Section 3.1). If the box and OCR data are unreliable, both the benchmark data and interaction detection are affected.
  • domain assumption GPT-4o and Claude 3.7 Sonnet produce HTML that is a faithful, clean representation of real webpages.
    HTML-v1 and HTML-v2 serve as the ground truth for all code-generation tasks (Section 3.1). No manual audit of the final HTML-v2 is reported, so the benchmark assumes these AI-generated pages are correct.
  • domain assumption Gemini 2.5 Flash can score visual similarity reliably across the ten rubric criteria.
    Gemini Visual Score is the central metric for design and code generation (Appendix C.1); the paper does not report inter-judge agreement with human scores beyond aggregate averages.
  • domain assumption The 15 image categories cover the image content that appears in real webpages.
    Image-to-Code evaluation requires models to classify every image into one of the 15 categories and use the category URL (Appendix B.1). Misclassification is penalized, so the taxonomy must be adequate for real web content.
  • domain assumption Human experts answering QA questions are a valid gold standard.
    Human expert accuracy above 95% is used to calibrate the QA tasks (Table 2); if the experts (likely the authors) were biased by knowing the GPT-4o-generated answer key, the human baseline would be inflated.
  • domain assumption Rendering MLLM-generated HTML in a headless browser produces a faithful visual for scoring.
    The visual metrics compare rendered screenshots of generated HTML against rendered ground truth; the paper does not detail the rendering environment or browser consistency.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow." pith.science (2026). https://pith.science/paper/5HKDSMBZ

@misc{pith2026250517399,
  author       = {Pith},
  title        = {Pith review of: FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5HKDSMBZ}},
  note         = {Machine review of arXiv:2505.17399}
}
read the original abstract

Front-end engineering involves a complex workflow where engineers conceptualize designs, translate them into code, and iteratively refine the implementation. While recent benchmarks primarily focus on converting visual designs to code, we present FullFront, a benchmark designed to evaluate Multimodal Large Language Models (MLLMs) \textbf{across the full front-end development pipeline}. FullFront assesses three fundamental tasks that map directly to the front-end engineering pipeline: Webpage Design (conceptualization phase), Webpage Perception QA (comprehension of visual organization and elements), and Webpage Code Generation (implementation phase). Unlike existing benchmarks that use either scraped websites with bloated code or oversimplified LLM-generated HTML, FullFront employs a novel, two-stage process to transform real-world webpages into clean, standardized HTML while maintaining diverse visual designs and avoiding copyright issues. Extensive testing of state-of-the-art MLLMs reveals significant limitations in page perception, code generation (particularly for image handling and layout), and interaction implementation. Our results quantitatively demonstrate performance disparities across models and tasks, and highlight a substantial gap between current MLLM capabilities and human expert performance in front-end engineering. The FullFront benchmark and code are available in https://github.com/Mikivishy/FullFront.

Figures

Figures reproduced from arXiv: 2505.17399 by the authors.

Figure 1
Figure 1. Mapping the full front-end engineering workflow to FullFront’s benchmark tasks: (1) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the eight subtasks FullFront covers and our data construction pipeline. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Comparison of the images used in our FullFront for webpage code generation tasks with [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: MLLM Errors in Webpage Perception QA. (a) Distribution of error types for 200 questions. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Three common errors in Webpage Code Generation. (a) Abnormal Image Sizes, where [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Human evaluation comparing MLLMs-generated and Real-World webpages. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The 15 predefined image categories used in FullFront for standardized image representation. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Comparative Webpage Designs: Ground Truth (“Label Webpage”) vs. gemini-2.0-flash [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Qualitative Comparison of Webpage Code Generation by MLLMs. This figure illustrates [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Rendered outputs from various MLLMs for the “NAVI PILLAY” webpage section. Despite [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation

    cs.SE 2026-08 conditional novelty 6.0 of 10

    On a new 1,440-screenshot fill-in-the-blank benchmark, five frontier multimodal models default to repeated UI patterns instead of visual deviations, with mean bias rates from 69.8% on card widths to 80.2% on font sizes.

  2. LH-Bench: Skill-Grounded Evaluation of Long-Horizon Agents on Subjective Enterprise Tasks

    cs.AI 2026-03 conditional novelty 6.0 of 10

    Expert-authored dual-use skills make LLM judges more reliable on subjective long-horizon enterprise agent tasks than LLM-authored rubrics, with human preferences confirming the main ranking boundary.

Reference graph

Works this paper leans on

74 extracted references · 46 canonical work pages · cited by 2 Pith papers

  1. [1]

    Iw-bench: Evaluating large multimodal models for converting image-to-web,

    H. Guo, W. Zhang, J. Chen, Y . Gu, J. Yang, J. Du, B. Hui, T. Liu, J. Ma, C. Zhou et al., “Iw-bench: Evaluating large multimodal models for converting image-to-web,” arXiv preprint arXiv:2409.18980, 2024

    arXiv 2024

  2. [2]

    Webcode2m: A real-world dataset for code generation from webpage designs,

    Y . Gui, Z. Li, Y . Wan, Y . Shi, H. Zhang, B. Chen, Y . Su, D. Chen, S. Wu, X. Zhou et al., “Webcode2m: A real-world dataset for code generation from webpage designs,” in Proceedings of the ACM on Web Conference 2025, 2025, pp. 1834–1845

    work page 2025

  3. [3]

    Webquest: A benchmark for multimodal qa on web page sequences,

    M. Wang, S. Sunkara, G. Baechler, J. Lin, Y . Zhu, F. Zubach, L. Shu, and J. Chen, “Webquest: A benchmark for multimodal qa on web page sequences,” arXiv preprint arXiv:2409.13711, 2024

    arXiv 2024

  4. [4]

    Webqa: Multihop and multimodal qa,

    Y . Chang, M. Narang, H. Suzuki, G. Cao, J. Gao, and Y . Bisk, “Webqa: Multihop and multimodal qa,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16 495–16 504

    work page 2022

  5. [5]

    Worldgui: Dynamic testing for comprehensive desktop gui automation,

    H. H. Zhao, D. Gao, and M. Z. Shou, “Worldgui: Dynamic testing for comprehensive desktop gui automation,” arXiv preprint arXiv:2502.08047, 2025

    arXiv 2025

  6. [6]

    Os-copilot: Towards generalist computer agents with self-improvement,

    Z. Wu, C. Han, Z. Ding, Z. Weng, Z. Liu, S. Yao, T. Yu, and L. Kong, “Os-copilot: Towards generalist computer agents with self-improvement,” arXiv preprint arXiv:2402.07456, 2024

    arXiv 2024

  7. [7]

    Cradle: Empowering foundation agents towards general computer control,

    W. Tan, W. Zhang, X. Xu, H. Xia, Z. Ding, B. Li, B. Zhou, J. Yue, J. Jiang, Y . Li et al., “Cradle: Empowering foundation agents towards general computer control,” arXiv preprint arXiv:2403.03186, 2024

    arXiv 2024

  8. [8]

    Mobile-agent-e: Self-evolving mobile assistant for complex tasks,

    Z. Wang, H. Xu, J. Wang, X. Zhang, M. Yan, J. Zhang, F. Huang, and H. Ji, “Mobile-agent-e: Self-evolving mobile assistant for complex tasks,”arXiv preprint arXiv:2501.11733, 2025

    arXiv 2025

Show all 74 references
  1. [9]

    Gpt-4v (ision) is a generalist web agent, if grounded,

    B. Zheng, B. Gou, J. Kil, H. Sun, and Y . Su, “Gpt-4v (ision) is a generalist web agent, if grounded,” arXiv preprint arXiv:2401.01614, 2024

    2024 arXiv

  2. [10]

    Assistantbench: Can web agents solve realistic and time-consuming tasks?

    O. Yoran, S. J. Amouyal, C. Malaviya, B. Bogin, O. Press, and J. Berant, “Assistantbench: Can web agents solve realistic and time-consuming tasks?” arXiv preprint arXiv:2407.15711, 2024

    2024 arXiv

  3. [11]

    Seeclick: Harnessing gui grounding for advanced visual gui agents,

    K. Cheng, Q. Sun, Y . Chu, F. Xu, Y . Li, J. Zhang, and Z. Wu, “Seeclick: Harnessing gui grounding for advanced visual gui agents,” arXiv preprint arXiv:2401.10935, 2024

    2024 arXiv

  4. [12]

    Mind2web: Towards a generalist agent for the web,

    X. Deng, Y . Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y . Su, “Mind2web: Towards a generalist agent for the web,”Advances in Neural Information Processing Systems, vol. 36, pp. 28 091–28 114, 2023

    2023

  5. [13]

    Winclick: Gui grounding with multi- modal large language models,

    Z. Hui, Y . Li, T. Chen, C. Banbury, K. Koishidaet al., “Winclick: Gui grounding with multi- modal large language models,” arXiv preprint arXiv:2503.04730, 2025

    2025 arXiv

  6. [14]

    Gui- world: A dataset for gui-oriented multimodal llm-based agents,

    D. Chen, Y . Huang, S. Wu, J. Tang, L. Chen, Y . Bai, Z. He, C. Wang, H. Zhou, Y . Liet al., “Gui- world: A dataset for gui-oriented multimodal llm-based agents,” arXiv e-prints, pp. arXiv–2406, 2024

    2024

  7. [15]

    Guicourse: From general vision language models to versatile gui agents,

    W. Chen, J. Cui, J. Hu, Y . Qin, J. Fang, Y . Zhao, C. Wang, J. Liu, G. Chen, Y . Huo et al., “Guicourse: From general vision language models to versatile gui agents,” arXiv preprint arXiv:2406.11317, 2024

    2024 arXiv

  8. [16]

    Websrc: a dataset for web-based structural reading comprehension,

    X. Chen, Z. Zhao, L. Chen, D. Zhang, J. Ji, A. Luo, Y . Xiong, and K. Yu, “Websrc: a dataset for web-based structural reading comprehension,” arXiv preprint arXiv:2101.09465, 2021

    2021 arXiv

  9. [17]

    Webwalker: Benchmarking llms in web traversal,

    J. Wu, W. Yin, Y . Jiang, Z. Wang, Z. Xi, R. Fang, L. Zhang, Y . He, D. Zhou, P. Xie et al., “Webwalker: Benchmarking llms in web traversal,”arXiv preprint arXiv:2501.07572, 2025

    2025 arXiv

  10. [18]

    Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark,

    Y . Hao, J. Gu, H. W. Wang, L. Li, Z. Yang, L. Wang, and Y . Cheng, “Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark,” arXiv preprint arXiv:2501.05444, 2025

    2025 arXiv

  11. [19]

    Screen2words: Automatic mobile ui summarization with multimodal learning,

    B. Wang, G. Li, X. Zhou, Z. Chen, T. Grossman, and Y . Li, “Screen2words: Automatic mobile ui summarization with multimodal learning,” in The 34th Annual ACM Symposium on User Interface Software and Technology, 2021, pp. 498–510

    2021

  12. [20]

    Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding?

    J. Liu, Y . Song, B. Y . Lin, W. Lam, G. Neubig, Y . Li, and X. Yue, “Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding?” arXiv preprint arXiv:2404.05955, 2024. 10

    2024 arXiv

  13. [21]

    Design2code: Benchmarking multimodal code generation for automated front-end engineering,

    C. Si, Y . Zhang, R. Li, Z. Yang, R. Liu, and D. Yang, “Design2code: Benchmarking multimodal code generation for automated front-end engineering,” arXiv preprint arXiv:2403.03163, 2024

    2024 arXiv

  14. [22]

    Web2code: A large-scale webpage-to-code dataset and evaluation framework for multimodal llms,

    S. Yun, H. Lin, R. Thushara, M. Q. Bhat, Y . Wang, Z. Jiang, M. Deng, J. Wang, T. Tao, J. Liet al., “Web2code: A large-scale webpage-to-code dataset and evaluation framework for multimodal llms,” arXiv preprint arXiv:2406.20098, 2024

    2024 arXiv

  15. [23]

    Unlocking the conversion of web screenshots into html code with the websight dataset,

    H. Laurençon, L. Tronchon, and V . Sanh, “Unlocking the conversion of web screenshots into html code with the websight dataset,” arXiv preprint arXiv:2403.09029, 2024

    2024 arXiv

  16. [24]

    pix2code: Generating code from a graphical user interface screenshot,

    T. Beltramelli, “pix2code: Generating code from a graphical user interface screenshot,” in Proceedings of the ACM SIGCHI symposium on engineering interactive computing systems, 2018, pp. 1–6

    2018

  17. [25]

    Mrweb: An exploration of gener- ating multi-page resource-aware web code from ui designs,

    Y . Wan, Y . Dong, J. Xiao, Y . Huo, W. Wang, and M. R. Lyu, “Mrweb: An exploration of gener- ating multi-page resource-aware web code from ui designs,” arXiv preprint arXiv:2412.15310, 2024

    2024 arXiv

  18. [26]

    Interaction2code: How far are we from automatic interactive webpage generation?

    J. Xiao, Y . Wan, Y . Huo, Z. Xu, and M. R. Lyu, “Interaction2code: How far are we from automatic interactive webpage generation?” arXiv preprint arXiv:2411.03292, 2024

    2024

  19. [27]

    Navigating the digital world as humans do: Universal visual grounding for gui agents,

    B. Gou, R. Wang, B. Zheng, Y . Xie, C. Chang, Y . Shu, H. Sun, and Y . Su, “Navigating the digital world as humans do: Universal visual grounding for gui agents,” arXiv preprint arXiv:2410.05243, 2024

    2024 arXiv

  20. [28]

    Hello gpt-4o,

    OpenAI, “Hello gpt-4o,” https://openai.com/index/hello-gpt-4o/

  21. [29]

    Omniparser for pure vision based gui agent,

    Y . Lu, J. Yang, Y . Shen, and A. Awadallah, “Omniparser for pure vision based gui agent,” 2024. [Online]. Available: https://arxiv.org/abs/2408.00203

    2024 arXiv

  22. [30]

    Common crawl datasets,

    C. Crawl, “Common crawl datasets,” 2025, accessed: 2025-05-01. [Online]. Available: https://data.commoncrawl.org/

    2025

  23. [31]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervi- sion,” in International conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  24. [32]

    “mdiff,” https://github.com/m-matelski/mdiff

  25. [33]

    Qwen2. 5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tanget al., “Qwen2. 5-vl technical report,” arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  26. [34]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling,

    Z. Chen, W. Wang, Y . Cao, Y . Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu et al., “Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling,” arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  27. [35]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,

    J. Zhu, W. Wang, Z. Chen, Z. Liu, S. Ye, L. Gu, Y . Duan, H. Tian, W. Su, J. Shao et al., “Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,” arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  28. [36]

    Llava-onevision: Easy visual task transfer,

    B. Li, Y . Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y . Li, Z. Liu, and C. Li, “Llava-onevision: Easy visual task transfer,” arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  29. [37]

    Claude 3.7 sonnet and claude code,

    Anthropic, “Claude 3.7 sonnet and claude code,” https://https://www.anthropic.com/news/ claude-3-7-sonnet

  30. [38]

    Gemini 2.5 flash,

    G. Deepmind, “Gemini 2.5 flash,” https://deepmind.google/technologies/gemini/flash/

  31. [39]

    Introducing openai o3 and o4-mini,

    OpenAI, “Introducing openai o3 and o4-mini,” https://openai.com/index/ introducing-o3-and-o4-mini/

  32. [40]

    Introducing gpt-4.1 in the api,

    ——, “Introducing gpt-4.1 in the api,” https://openai.com/index/gpt-4-1/

  33. [41]

    Learning to reason with llms,

    ——, “Learning to reason with llms,” https://openai.com/index/learning-to-reason-with-llms/

  34. [42]

    Gemini 2.5: Our most intelligent ai model,

    G. Deepmind, “Gemini 2.5: Our most intelligent ai model,” https://blog.google/technology/ google-deepmind/gemini-model-thinking-updates-march-2025/

    2025

  35. [43]

    Experiment with gemini 2.0 flash native image genera- tion,

    K. Kampf and N. Brichtova, “Experiment with gemini 2.0 flash native image genera- tion,” https://developers.googleblog.com/en/experiment-with-gemini-20-flash-native-image- generation/. 11 A FullFront-mini FullFront-mini Dataset To facilitate rapid iterative evaluation of MLLMs...

  36. [44]

    Requires aria-expanded attribute toggling and specific dropdown selectors

    Click to Display Dropdown (Interaction_click_1): An element, when clicked, reveals a dropdown menu whose content, position, and style are contextually adapted. Requires aria-expanded attribute toggling and specific dropdown selectors

  37. [45]

    checkbox

    Click to Toggle Checkbox (Interaction_click_2): A clickable checkbox that toggles its checked/unchecked state. Must use an <input type="checkbox"> or an element with role=’checkbox’, displaying a checked state after interaction

  38. [46]

    Click to Change Background Color (Interaction_click_3): An element significantly changes its background color upon being clicked, with the new color being distinct and detectable

  39. [47]

    The modal must match specific selectors like .modal or [aria-modal=’true’]

    Click to Display Modal/Dialog (Interaction_click_4): Clicking an element triggers a modal window or dialog box with contextually generated content and styling. The modal must match specific selectors like .modal or [aria-modal=’true’]

  40. [48]

    The tooltip must adhere to specified class names or attributes (e.g., .tooltip, [role=’tooltip’])

    Click to Display Tooltip (Interaction_click_5): An element, when clicked, displays a tooltip providing additional information. The tooltip must adhere to specified class names or attributes (e.g., .tooltip, [role=’tooltip’])

  41. [49]

    Click to Display Text Input (Interaction_click_6): Clicking an element reveals a text box or input area for user entry, appropriately sized and positioned

  42. [50]

    Hover to Display Dropdown (Interaction_hover_1): A dropdown menu appears when the mouse hovers over an element, with adaptive content and styling

  43. [51]

    Hover to Bold Text ( Interaction_hover_2): Text within an element becomes bold (fontWeight ≥600 or ‘bold‘/‘bolder‘) upon mouse hover

  44. [52]

    underline

    Hover to Underline Text ( Interaction_hover_3): Text within an element gains an underline when hovered over, with the computed textDecoration including “underline”

  45. [53]

    Hover to Display Tooltip (Interaction_hover_4): A tooltip with additional information appears when the mouse hovers over an element, conforming to specific class or attribute requirements. 13 The models must determine the correct interaction type from the visual cues and imple...

  46. [54]

    It also considers if these reproduced elements are styled identically to the original in terms of appearance (e.g., color, shape, visual effects)

    Element Reproduction (Score: 0-10): This criterion assesses whether all key visual elements present in the ground-truth design (e.g., textual content, images, buttons, icons, input fields) are fully reproduced in the generated webpage. It also considers if these reproduced ele...

  47. [55]

    • Score 10: All elements maintain exact proportions and sizes relative to each other and the overall page, as in the original

    Proportion and Size Consistency (Score: 0-10): This evaluates if the relative and absolute sizes and proportions of all elements (including text blocks, images, buttons, and containers) in the generated page match those in the ground-truth design, thereby maintaining the inten...

  48. [56]

    Layout and Typography Fidelity (Score: 0-10): This focuses on the faithful replication of the overall page structure and typographic choices. It examines the placement and styling of major layout components such as headers, footers, navigation bars, sidebars, content grids, an...

  49. [57]

    • Score 10: All elements exhibit perfect alignment and spacing as per the original design

    Alignment and Spacing Accuracy (Score: 0-10): This criterion measures the precision of element alignment (e.g., left, right, center, justified, relative to other elements) and the consistency of spacing (margins, padding, gutters) both within and between elements, compared to ...

  50. [58]

    Visual Hierarchy Clarity (Score: 0-10): This assesses if the generated webpage success- fully maintains the same visual hierarchy as the original design. This means that the relative prominence and order of importance of different elements (achieved through size, color, contra...

  51. [59]

    • Score 10: All colors, including background, text, and element colors, are identical to the original design

    Color Consistency (Score: 0-10): This evaluates the match of the overall color scheme, including primary, secondary, and accent colors, as well as specific hues, saturation, and brightness levels used throughout the generated webpage, compared to the ground-truth. • Score 10: ...

  52. [60]

    look and feel

    Style Consistency (Score: 0-10): This criterion judges whether the overall aesthetic style of the generated webpage (e.g., modern, minimalistic, brutalist, skeuomorphic, playful) aligns with the intended style of the original design. This is a more holistic assessment of the “...

  53. [61]

    • Score 10: All text attributes (font, size, spacing, color, alignment, etc.) are identical to the original

    Text Style Consistency (Score: 0-10): This focuses specifically on the typographic at- tributes of textual content, such as font family, size, weight, style (italic, bold), color, line height, letter spacing, paragraph spacing, and text alignment, ensuring they are consistent ...

  54. [62]

    • Score 10: All main textual content is identical to the original

    Text Content Accuracy (Score: 0-10): This evaluates if the primary textual content (headings, body text, labels, captions) displayed on the generated webpage accurately reproduces the text from the original design, without omissions, additions, or substantial alterations. • Sc...

  55. [63]

    • Score 10: The generated page perfectly represents the same content, information, and intent as the original

    Overall Content Representation (Score: 0-10): This is a holistic measure of whether the generated webpage effectively conveys the same core information, message, purpose, and user intent as the original design, considering all visual and textual elements collectively. • Score ...

  56. [64]

    Structural Similarity Both Hgen and Href are parsed into DOM trees. We then extract sequences of HTML tags, Sgen and Sref respectively, representing the structural hierarchy of the doc- uments (as implemented in extract_structure and structure_to_sequence). The structural simi...

  57. [65]

    For each type, corresponding elements are identified and compared

    Content-Type Similarity This assesses similarity for three key content types: text, images, and forms. For each type, corresponding elements are identified and compared. Element Matching For each content type c ∈ {text, image, form}, we extract all elements of that type from H...

  58. [66]

    Content Similarity ( Simtext_content(egen, eref )): Calculated using Python’s SequenceMatcher on the textual content extracted

  59. [67]

    Each property p has a weight wp

    Style Similarity (Simtext_style(egen, eref )): Computed by comparing key CSS prop- erties (e.g., color, font-size, font-weight, background-color, etc.). Each property p has a weight wp. The style similarity is a weighted average of individual property similarities. Numerical p...

  60. [68]

    URL Similarity (0.6 weight): Based on comparing extracted category information from the image ‘src‘ attribute (e.g., ‘Animal‘ from ‘.../Animal.jpg‘) or filenames if the category pattern doesn’t match

  61. [69]

    Style Similarity (0.3 weight): Calculated similarly to text styles, using image-specific CSS properties (e.g., width, height, border-radius, as per self.style_weights[’image’])

  62. [70]

    The average Simimage is calculated over all matched image elements

    Alt Text Similarity (0.1 weight): Comparing the ‘alt‘ attributes using SequenceMatcher. The average Simimage is calculated over all matched image elements. • Form Elements (c = form): Similarity (Simf orm(egen, eref )) depends on the specific form element type (e.g., ‘input‘, ...

  63. [71]

    Attribute Similarity: Compares critical HTML attributes specific to the form ele- ment type (e.g., ‘type‘, ‘name‘, ‘value‘, ‘placeholder‘ for ‘input‘ elements) using SequenceMatcher

  64. [72]

    Style Similarity: Calculated using form-specific CSS properties (e.g., width, height, background-color)

  65. [73]

    Label Webpage

    Text Content Similarity (only for elements like ‘button‘, ‘label‘, ‘option‘): Compares textual content using SequenceMatcher. The average Simf ormis calculated over all matched form elements. 17 Figure 8: Comparative Webpage Designs: Ground Truth (“Label Webpage”) vs. gemini-2...

  66. [74]

    Label Webpage

    Final Code Score Aggregation The final Code Score ( Score code) is a weighted sum of the structural similarity and the adjusted content-type similarities: Score code = Wstruct · Simstruct + Wtext_content · Sim′ text_content + Wtext_style · Sim′ text_style + Wimage · Sim′ image...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.