Pith. sign in

REVIEW 3 major objections 6 minor 57 references

WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read WebMMU, a new multilingual benchmark, shows that current multimodal models handle basic page reading but fail at reasoning, grounding, structure-preserving code edits, and multilingual design-to-code.

desk verdict A genuinely useful multilingual web benchmark with one load-bearing caveat: the Mockup2Code judge correlates only moderately with humans, so the headline claims about nested-layout failure need either stronger validation or softer wording. read the letter →

arxiv 2508.16763 v1 pith:EKORI2YE submitted 2025-08-22 cs.CV

Rabiul Awal , Mahsa Massoud , Aarash Feizi , Zichao Li , Suyuchen Wang , Christopher Pal , Aishwarya Agrawal , David Vazquez
show 5 more authors
Siva Reddy Juan A. Rodriguez Perouz Taslakian Spandana Gella Sai Rajeswar
This is my paper · ORCID
classification cs.CV
keywords multimodallargelanguagemodelswebunderstandingbenchmarkvisualquestionansweringdesign-to-codegenerationcodeeditingmultilingualevaluationUIgroundingagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces WebMMU, a benchmark that brings together three web tasks that prior work treats separately: visual question answering about real website screenshots, editing HTML/CSS/JavaScript code, and turning mockups into webpage code. The dataset is expert-annotated, covers four languages (English, Spanish, German, French) and 20 domains, and contains 6,102 WebQA questions, 1,602 code-editing cases, and 436 mockup-to-code instances. Its central claim is that current multimodal large language models do well at basic visual extraction but fail when a task needs multi-step reasoning, precise spatial grounding, structure-preserving code changes, or multilingual layout generation. If the claim is right, WebMMU gives the field a single testbed for web agents and web-development assistants, and the failure pattern it documents becomes a concrete target for future model-building.

What carries the argument

The central object is WebMMU itself, a unified benchmark whose annotations tie three tasks to the same real web pages: WebQA questions demand reasoning and coordinate-level grounding, Mockup2Code pairs simple and nested UI sketches with reference code, and Web Code Editing couples screenshots and source code with user-requested changes. Evaluation runs on GPT-4o-as-judge (binary correctness for WebQA, 1-5 layout fidelity for Mockup2Code, 1-5 functional correctness for code editing) alongside BLEU and TreeBLEU for structural code matching. The shared page-level starting point is what lets the benchmark compare performance across tasks and attribute gaps to model capability rather than to dataset curation.

What would settle it

Take a fresh sample of the paper's mockup-to-code outputs for nested layouts and have professional web developers rank them by whether the code reproduces the mockup's hierarchy and spacing. If the human ordering disagrees with the GPT-4o judge's 1-5 score ordering at the level implied by the reported rank correlation of about 0.43 on a larger sample, then the claimed failure on UI hierarchy is an artifact of the judge rather than a model limitation.

Watch

Extended reading notes

Core claim

On the paper's own terms, WebMMU is a benchmark built from 2,059 expert-annotated real webpage screenshots in English, Spanish, German, and French, covering 20 domains and three tasks: WebQA (6,102 questions), Mockup2Code (436 instances), and Web Code Editing (1,602 edits). The central discovery is a consistent capability gap: models are near-competent at extracting visible facts from a page, but their accuracy collapses on multi-step reasoning (most below 50%, some near 0), on agentic action and coordinate grounding (almost always in single digits), on code edits that preserve function and structure (syntactically plausible but incomplete, and no model produced a directly applicable patch), and on mockup-to-code for nested layouts and multilingual content.

Load-bearing premise

The conclusions rest on the automatic GPT-4o judge's scores matching what expert human raters would say about correctness and layout fidelity; agreement is high for WebQA and code editing (89% and 91%) but only moderate for mockup-to-code (average rank correlation about 0.43), so the nested-layout findings are the part of the paper most exposed to judge error.

Editorial extensions

If this is right

  • WebQA's agentic-action scores, which top out near 10% even for closed models, place precise spatial grounding of clickable elements as the least solved requirement for web agents.
  • The fact that no tested model produced a directly usable patch means automated web-code editing will need human oversight until models learn to emit valid diffs.
  • Mockup2Code results separate simple from nested layouts, so UI-hierarchy comprehension is a measurable bottleneck rather than a general design-to-code problem.
  • Multilingual versions of every task score below their English counterparts for most models, making cross-lingual robustness an independent axis of difficulty.
  • WebMMU lets future work compare models on all three tasks under one protocol, which fragmented benchmarks cannot.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct comparison of the same models on WebMMU's real pages versus synthetic pages would test whether real-world layout variability, not raw difficulty, is what breaks models.
  • Because action questions are answered from a single screenshot with no environment, WebMMU measures action prediction, not successful interaction; a live-browser follow-up is needed to know whether grounding scores predict task completion.
  • The multilingual drop may partly come from OCR and font-rendering differences across languages; isolating perception from reasoning would show where cross-lingual investment belongs.
  • Collecting pairwise human preferences on mockup-to-code outputs, rather than rating on a 1-5 scale, would likely produce a stabler judge and clarify the nested-layout gap.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. WebMMU is a benchmark for multimodal, multilingual website understanding and code generation, covering three tasks: website visual question answering (WebQA), mockup-to-code generation, and web code editing. The dataset is expert-annotated and spans English, Spanish, German, and French, drawn from 2,059 screenshots across 20 domains. The authors evaluate 17 open and closed multimodal LLMs and report that models perform well on basic visual extraction but struggle with multi-step reasoning, action grounding, structure-preserving code editing, and hierarchical design-to-code generation, with notable multilingual performance drops. The dataset is promised to be publicly released.

Significance. If the evaluation methodology is sound, WebMMU fills a real gap by unifying three web-centric tasks in a multilingual, expert-annotated benchmark. The construction has clear strengths: 100% quality-assurance review, professional annotators across several continents, coverage of 20 domains and four languages, and human-alignment measurements for two of the three tasks (89% agreement for WebQA and 91% for Web Code Editing). The broad model sweep, including open- and closed-source systems, gives the community a useful snapshot of current capabilities. The main weakness is the moderate human-judge agreement for Mockup2Code, which currently under-specifies the reliability of the central design-to-code claims; if that is addressed, the benchmark would be a credible resource for web-agent and UI-understanding research.

major comments (3)
  1. [5.4, Figure 3] The Mockup2Code conclusion that models fail on nested/hierarchical layouts is carried by the GPT4o-as-judge scores, but the reported human alignment is only Spearman 0.43 (layout 0.39, spacing 0.33, alignment 0.46) and Pearson 0.50. Calling this moderate agreement 'support for reliability' is not established on the presented evidence: on 100 samples, it leaves most of the ordinal variation in human ratings unexplained. Moreover, no human validation is reported separately for the simple-vs-complex/nested strata where the headline 'maintains hierarchy' claim lives. Please add stratified human-judge agreement for the complex/nested subset, report inter-annotator agreement, and either recalibrate the judge or weaken the claim that MLLMs specifically fail at hierarchy preservation.
  2. [5.1, Table 3] The text states that 'Agentic action is the hardest, with top models rarely surpassing 10% accuracy,' but Table 3 contradicts this. Claude3.5 Sonnet, Gemini-2.0-Flash, and OpenAI-o1 all exceed 10% agentic accuracy in French, German, and Spanish (e.g., Claude3.5 Sonnet obtains 14.6, 16.0, and 16.1 in those three languages). Furthermore, several smaller models show agentic accuracy that is not the lowest category: Phi3.5-VI-4b German has Agentic 10.9 vs Multi-step 6.9, Molmo-7b German has 9.6 vs 5.1, and Pixtral-12b German has 14.2 vs 12.8. Please correct the summary to match the table or clarify which subset 'top models' refers to.
  3. [5.3] The strong universal claim that 'all models ... fail to automatically generate valid patch files' and that 'none produced patch content directly usable without manual fixes' is reported without describing how patch validity was determined. Because this is a claim about all 17 evaluated models, please specify the validation protocol (e.g., whether patches were applied to the original source, compiled, rendered, or human-adjudicated), or soften the claim to match the evidence presented.
minor comments (6)
  1. [Table 2 / Figure 3] The evaluation dimensions for Mockup2Code are named inconsistently: Table 2 uses 'layout, spacing, grid' while Figure 3 and Section 5.4 use 'layout, spacing, alignment'. Please unify the terminology.
  2. [Table 3] The table caption should explicitly state the column order (Multi-step Reasoning, Agentic Action, General Visual Comprehension) and explain the icon legend; currently the ordering is only inferable from the caption after close reading.
  3. [Section 5.4] For the WebQA and Web Code Editing human-alignment checks (89% and 91%), please report the number of annotators, the sampling procedure across languages and difficulty strata, and the inter-annotator agreement, since a single agreement percentage against the LLM judge does not fully characterize reliability.
  4. [3.2.1] The category name 'Agentic Action' is potentially misleading because the task is single-turn action grounding without environment feedback; consider renaming it to 'Action Grounding' or 'UI Action Comprehension'.
  5. [References] The PairBench citation used to justify the GPT4o judge selection shares co-authors with this submission; the paper should disclose this overlap or provide independent evidence for the judge's reliability.
  6. [Table 1] The row label 'Web Code Editing886' is missing a separating space, and the table would benefit from explicit column totals for each task and language.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's expert-annotated data, evaluation prompts, and in-paper human-alignment checks keep the conclusions independent of their inputs.

full rationale

WebMMU's derivation chain is self-contained. The benchmark items (WebQA questions, Mockup2Code sketches, code-edit requests) are expert-annotated in Section 3 and Appendix A, and they do not depend on any evaluated model or on the paper's conclusions. Model performance is measured with standardized prompts and either literal metrics (BLEU/TreeBLEU for code editing) or an LLM judge whose selection is contextual rather than definitional: Section 5.4 reports human-alignment checks (89% agreement on WebQA, 91% on code editing, and moderate Spearman correlations for Mockup2Code), so the judge is validated independently of the paper's claims. The Feizi et al. (2025) citation for GPT4o-1120 is a self-citation, but it is not load-bearing because the paper supplies its own human-alignment evidence. The moderate Mockup2Code correlations (Spearman 0.43) are a measurement-validity caveat, not a circular reduction: the claim that models fail at nested layouts is an empirical output of the evaluation, not an input to it. No equation or dataset-construction step defines the conclusions in terms of themselves.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted in this benchmark paper. The main assumptions are that the expert annotations constitute ground truth and that the automated judge (GPT4o) is a reliable evaluator, the latter being only moderately supported for Mockup2Code. No new entities are postulated.

assumptions (3)
  • domain assumption Expert-annotated data (QA pairs, code edits, mockup sketches) constitutes accurate ground truth for the three tasks.
    The paper describes a multi-stage QA process with 127 annotators, but the correctness of the annotations is not independently verified in the paper.
  • domain assumption GPT4o-as-judge produces valid correctness and fidelity scores across all three tasks.
    Section 5.4 reports human alignment, but the Mockup2Code correlations are moderate (Spearman 0.43 average), meaning the judge's layout fidelity scores are only weakly validated.
  • domain assumption BLEU and TreeBLEU are meaningful metrics for structural correctness of code edits.
    The paper uses n-gram-based metrics for code, which are known to be imperfect for code structure; an LLM judge is used as a fallback, but its reliability for code is also assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation." pith.science (2026). https://pith.science/paper/EKORI2YE

@misc{pith2026250816763,
  author       = {Pith},
  title        = {Pith review of: WebMMU: A Benchmark for Multimodal Multilingual Website Understanding and Code Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EKORI2YE}},
  note         = {Machine review of arXiv:2508.16763}
}
read the original abstract

We present WebMMU, a multilingual benchmark that evaluates three core web tasks: (1) website visual question answering, (2) code editing involving HTML/CSS/JavaScript, and (3) mockup-to-code generation. Unlike prior benchmarks that treat these tasks separately, WebMMU unifies them using expert-annotated, real-world web data to assess models' abilities in complex multi-step reasoning, precise element grounding, and functional UI comprehension and coding. Our evaluation shows that while multimodal large language models (MLLMs) perform well on basic information extraction, they struggle with reasoning and grounding, editing code to preserve functionality, and generating design-to-code that maintains hierarchy and supports multilingual content. These findings reveal key limitations in current MLLMs and underscore the need for improved multimodal and cross-lingual reasoning to build future web agents capable of automating diverse web development tasks.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 10 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, S \'e bastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, and 110 others. 2024. Phi-3 technical report: A highly capable language model l...

    arXiv 2024

  2. [2]

    Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, and 23 others

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, and 23 others. 2024. https://arxiv.org/ab...

    arXiv 2024

  3. [3]

    AI Anthropic. 2024. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card, 3(6)

    work page 2024

  4. [4]

    Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor C a rbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. 2024. Screenai: A vision-language model for ui and infographics understanding. arXiv preprint arXiv:2402.04615

    arXiv 2024

  5. [5]

    Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and 1 others. 2021. Uibert: Learning generic multimodal representations for ui understanding. arXiv preprint arXiv:2107.13731

    arXiv 2021

  6. [6]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

    arXiv 2025

  7. [7]

    Somoy Subandhu Barua, Imam Mohammad Zulkarnain, Abhishek Roy, Md Golam Rabiul Alam, and Md Zia Uddin. 2022. Sketch2fullstack: Generating skeleton code of full stack website and application from sketch using deep learning and computer vision. arXiv preprint arXiv:2211.14607

    work page Pith review arXiv 2022

  8. [8]

    Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sa g nak Ta s rlar. 2023. https://www.adept.ai/blog/fuyu-8b Introducing our multimodal models

    2023

Show all 57 references
  1. [9]

    Tony Beltramelli. 2018. pix2code: Generating code from a graphical user interface screenshot. In Proceedings of the ACM SIGCHI symposium on engineering interactive computing systems, pages 1--6

    2018

  2. [10]

    Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Jane Anderson, and 1 others. 2023. Can it edit? evaluating the ability of large language models to follow code editing instruction...

    2023 arXiv

  3. [11]

    Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. 2022. Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16495--16504

    2022

  4. [12]

    Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, and 1 others. 2024 a . Gui-world: A dataset for gui-oriented multimodal llm-based agents. arXiv preprint arXiv:2406.10819

    2024 arXiv

  5. [13]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021 a . Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    2021 arXiv

  6. [14]

    Xingyu Chen, Zihan Zhao, Lu Chen, Jiabao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021 b . Websrc: A dataset for web-based structural reading comprehension. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4173--4185

    2021

  7. [15]

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024 b . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference...

    2024

  8. [16]

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935

    2024 arXiv

  9. [17]

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch,...

    2024 arXiv

  10. [18]

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2024. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36

    2024

  11. [19]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  12. [20]

    Aarash Feizi, Sai Rajeswar, Adriana Romero-Soriano, Reihaneh Rabbany, Spandana Gella, Valentina Zantedeschi, and Jo \ a o Monteiro. 2025. Pairbench: A systematic framework for selecting reliable judge vlms. arXiv preprint arXiv:2502.15210

    2025 arXiv

  13. [21]

    Yi Gui, Zhen Li, Yao Wan, Yemin Shi, Hongyu Zhang, Yi Su, Shaoling Dong, Xing Zhou, and Wenbin Jiang. 2024. Vision2ui: A real-world dataset with layout for code generation from ui designs. arXiv preprint arXiv:2404.06369

    2024 arXiv

  14. [22]

    Jiawei Guo, Ziming Li, Xueling Liu, Kaijing Ma, Tianyu Zheng, Zhouliang Yu, Ding Pan, Yizhi Li, Ruibo Liu, Yue Wang, and 1 others. 2024. Codeeditorbench: Evaluating code editing capability of large language models. arXiv preprint arXiv:2404.03543

    2024 arXiv

  15. [23]

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919

    2024 arXiv

  16. [24]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  17. [25]

    Vanita Jain, Piyush Agrawal, Subham Banga, Rishabh Kapoor, and Shashwat Gulyani. 2019. Sketch2code: transformation of sketches to ui in real-time using deep neural network. arXiv preprint arXiv:1910.08930

    2019 arXiv

  18. [26]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In ICLR

    2024

  19. [27]

    Jihyung Kil, Chan Hee Song, Boyuan Zheng, Xiang Deng, Yu Su, and Wei-Lun Chao. 2024. Dual-view visual contextualization for web navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14445--14454

    2024

  20. [28]

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649

    2024 arXiv

  21. [29]

    Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. 2023. Viescore: Towards explainable metrics for conditional image synthesis evaluation. arXiv preprint arXiv:2312.14867

    2023 arXiv

  22. [30]

    Hugo Lauren c on, L \'e o Tronchon, and Victor Sanh. 2024. Unlocking the conversion of web screenshots into html code with the websight dataset. arXiv preprint arXiv:2403.09029

    2024 arXiv

  23. [31]

    Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. 2023. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Confe...

    2023

  24. [32]

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and 1 others. 2024 a . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

    2024 arXiv

  25. [33]

    Kaixin Li, Qisheng Hu, Xu Zhao, Hui Chen, Yuxi Xie, Tiedong Liu, Qizhe Xie, and Junxian He. 2023. Instructcoder: Instruction tuning large language models for code editing. arXiv preprint arXiv:2310.20329

    2023 arXiv

  26. [34]

    Ryan Li, Yanzhe Zhang, and Diyi Yang. 2024 b . Sketch2code: Evaluating vision-language models for interactive web design prototyping. arXiv preprint arXiv:2410.16232

    2024 arXiv

  27. [35]

    Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. 2024. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05955

    2024 arXiv

  28. [36]

    Xing Han L \`u , Zden e k Kasner, and Siva Reddy. 2024. Weblinx: Real-world website navigation with multi-turn dialogue. arXiv preprint arXiv:2402.05930

    2024

  29. [37]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  30. [38]

    Guilherme Penedo, Hynek Kydl \' c ek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. https://openreview.net/forum?id=n6SCkn2QaG The fineweb datasets: Decanting the web for the finest text data at scale . In The Thirt...

    2024

  31. [39]

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, and 1 others. 2025. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326

    2025 arXiv

  32. [40]

    Juan Rodriguez, Xiangru Jian, Siba Smarak Panigrahi, Tianyu Zhang, Aarash Feizi, Abhay Puri, Akshay Kalkunte, François Savard, Ahmed Masry, Shravan Nayak, Rabiul Awal, Mahsa Massoud, Amirhossein Abaskohi, Zichao Li, Suyuchen Wang, Pierre-André Noël, Mats Leon Richter, Saverio ...

    2024 arXiv

  33. [41]

    Rodriguez, Abhay Puri, Shubham Agarwal, Issam H

    Juan A. Rodriguez, Abhay Puri, Shubham Agarwal, Issam H. Laradji, Pau Rodriguez, Sai Rajeswar, David Vazquez, Christopher Pal, and Marco Pedersoli. 2024 b . https://arxiv.org/abs/2312.11556 Starvector: Generating scalable vector graphics code from images and text . Preprint, a...

    2024 arXiv

  34. [42]

    Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Haotian Hui, Weichuan Liu, Zhiyuan Liu, and 1 others. 2024. Debugbench: Evaluating debugging capability of large language models. arXiv preprint arXiv:2401.04621

    2024 arXiv

  35. [43]

    Maria Wang, Srinivas Sunkara, Gilles Baechler, Jason Lin, Yun Zhu, Fedir Zubach, Lei Shu, and Jindong Chen. 2024 a . Webquest: A benchmark for multimodal qa on web page sequences. arXiv preprint arXiv:2409.13711

    2024 arXiv

  36. [44]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024 b . Qwen2-vl: Enhancing vision-language m...

    2024 arXiv

  37. [45]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024 c . Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neura...

    2024

  38. [46]

    Jason Wu, Xiaoyi Zhang, Jeff Nichols, and Jeffrey P Bigham. 2021. Screen parsing: Towards reverse engineering of ui models from screenshots. In The 34th Annual ACM Symposium on User Interface Software and Technology, pages 470--483

    2021

  39. [47]

    Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Deyu Zhou, Pengjun Xie, and Fei Huang. 2025. Webwalker: Benchmarking llms in web traversal. arXiv preprint arXiv:2501.07572

    2025 arXiv

  40. [48]

    Linhui Xu, Yawen Zhang, Yawen Li, Yawen Zhang, and Yawen Li. 2024. https://doi.org/10.1145/3664647.3681071 Hierarchical multimodal fine-grained modulation for visual grounding . In Proceedings of the 32nd ACM International Conference on Multimedia, pages 1--9. ACM

    2024

  41. [49]

    Ori Yoran, Samuel Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. 2024. Assistantbench: Can web agents solve realistic and time-consuming tasks? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8938--8968

    2024

  42. [50]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, and 1 others. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Co...

    2024

  43. [51]

    Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, and 1 others. 2024. Web2code: A large-scale webpage-to-code dataset and evaluation framework for multimodal llms. arXiv preprint arXiv:2406.20098

    2024 arXiv

  44. [52]

    Ziniu Zhang, Shulin Tian, Liangyu Chen, and Ziwei Liu. 2024. Mmina: Benchmarking multihop multimodal internet agents. arXiv preprint arXiv:2404.09992

    2024 arXiv

  45. [53]

    Gpt-4v (ision) is a generalist web agent, if grounded

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. In Forty-first International Conference on Machine Learning

  46. [54]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  47. [55]

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2023. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854

    2023 arXiv

  48. [56]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.