{"id":"742ccced-6d4b-4ef1-8a11-6e611b296156","arxiv_id":"2508.16763","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"WebMMU introduces a multilingual, three-task benchmark for website understanding and code generation, and finds current MLLMs underperform on reasoning, grounding, and functional code editing.","lead":"WebMMU is a new benchmark that tests AI models on three web tasks: answering questions about website screenshots, turning mockups into code, and editing existing web code. Evaluations show that even the best multimodal models often fail at reasoning about page structure, locating interactive elements, and producing correct multilingual code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Mockup2Code results rely on an LLM judge with only moderate human agreement (Spearman 0.43), so the paper's claim that models fail at nested/hierarchical design-to-code is not yet established.","rationale":"The reader's weakest-assumption analysis points to the right spot. The central claim has three pillars, and two of them (WebQA and code editing) have reasonable human-judge agreement (89% and 91%), but the Mockup2Code pillar rests on a judge with only 0.43 average Spearman correlation against humans. Because the abstract specifically foregrounds the design-to-code hierarchy failure, that pillar is load-bearing. My concern is not that the benchmark is useless; expert annotation, quality-control stages, multilingual coverage, and public release are real strengths. The issue is epistemic calibration: a moderate correlation may be enough to say the judge is usable for coarse comparisons, but it is not enough to support the specific claim that models fail on nested structures, which is precisely the new finding the paper emphasizes. I would therefore keep the conditional verdict: accept only after the Mockup2Code evaluation is validated on the complex-layout stratum and the reporting issues are fixed. A secondary check I noticed is an apparent internal inconsistency between Section 5.1 (\"top models rarely surpassing 10%\" on agentic action) and Table 3, where Qwen2.5VL-7B/72B show agentic scores around 20-41%; this does not change my main concern but reinforces that the paper needs a careful numbers pass. The Limitations section is also cut off mid-sentence, so the authors' own caveats are not fully available, another reason to handle the paper conditionally.","tokens_in":13463,"tokens_out":10302,"duration_ms":97651,"concrete_test":"Re-run the Mockup2Code human-evaluation protocol of Section 5.4 on the same 100 samples with at least three independent human raters using the paper's 1-5 rubric, and compute (i) inter-rater agreement and (ii) human-judge Spearman/Pearson correlations separately for the simple and complex layout strata. If human-judge agreement on the complex stratum is below 0.5, or if human raters do not reproduce the judge-reported simple-vs-complex gap in Figure 3, then the hierarchy-failure conclusion is unsupported and the task should be rescored with human judgments before drawing conclusions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.4 reports that for Mockup2Code the GPT4o-1120 judge achieves Spearman correlations of 0.39 (layout), 0.33 (spacing), and 0.46 (alignment), averaging 0.43, with a Pearson average of 0.50. The central claim that MLLMs fail at \"generating design-to-code that maintains hierarchy\" is carried by exactly this judge score, especially the simple-vs-complex gap in Figure 3. A Spearman correlation of 0.43 on 100 samples leaves most of the ordinal variation in human ratings unexplained; it is not strong enough to support fine-grained claims about layout hierarchy or to rank models by fidelity. The paper argues the correlation is acceptable because the task is subjective, but that does not establish the specific \"nested structure\" failure: no human validation is reported separately for the complex/nested strata where the headline claim lives. If the judge's complex-layout scores do not track human judgment, the central design-to-code conclusion is unsupported. This concern is specific to Mockup2Code; WebQA and code editing have much stronger human agreement (89% and 91%).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"WebMMU is a benchmark for multimodal, multilingual website understanding and code generation, covering three tasks: website visual question answering (WebQA), mockup-to-code generation, and web code editing. The dataset is expert-annotated and spans English, Spanish, German, and French, drawn from 2,059 screenshots across 20 domains. The authors evaluate 17 open and closed multimodal LLMs and report that models perform well on basic visual extraction but struggle with multi-step reasoning, action grounding, structure-preserving code editing, and hierarchical design-to-code generation, with notable multilingual performance drops. The dataset is promised to be publicly released.","tokens_in":13678,"tokens_out":11137,"duration_ms":101849,"significance":"If the evaluation methodology is sound, WebMMU fills a real gap by unifying three web-centric tasks in a multilingual, expert-annotated benchmark. The construction has clear strengths: 100% quality-assurance review, professional annotators across several continents, coverage of 20 domains and four languages, and human-alignment measurements for two of the three tasks (89% agreement for WebQA and 91% for Web Code Editing). The broad model sweep, including open- and closed-source systems, gives the community a useful snapshot of current capabilities. The main weakness is the moderate human-judge agreement for Mockup2Code, which currently under-specifies the reliability of the central design-to-code claims; if that is addressed, the benchmark would be a credible resource for web-agent and UI-understanding research.","major_comments":[{"comment":"The Mockup2Code conclusion that models fail on nested/hierarchical layouts is carried by the GPT4o-as-judge scores, but the reported human alignment is only Spearman 0.43 (layout 0.39, spacing 0.33, alignment 0.46) and Pearson 0.50. Calling this moderate agreement 'support for reliability' is not established on the presented evidence: on 100 samples, it leaves most of the ordinal variation in human ratings unexplained. Moreover, no human validation is reported separately for the simple-vs-complex/nested strata where the headline 'maintains hierarchy' claim lives. Please add stratified human-judge agreement for the complex/nested subset, report inter-annotator agreement, and either recalibrate the judge or weaken the claim that MLLMs specifically fail at hierarchy preservation.","section":"5.4, Figure 3"},{"comment":"The text states that 'Agentic action is the hardest, with top models rarely surpassing 10% accuracy,' but Table 3 contradicts this. Claude3.5 Sonnet, Gemini-2.0-Flash, and OpenAI-o1 all exceed 10% agentic accuracy in French, German, and Spanish (e.g., Claude3.5 Sonnet obtains 14.6, 16.0, and 16.1 in those three languages). Furthermore, several smaller models show agentic accuracy that is not the lowest category: Phi3.5-VI-4b German has Agentic 10.9 vs Multi-step 6.9, Molmo-7b German has 9.6 vs 5.1, and Pixtral-12b German has 14.2 vs 12.8. Please correct the summary to match the table or clarify which subset 'top models' refers to.","section":"5.1, Table 3"},{"comment":"The strong universal claim that 'all models ... fail to automatically generate valid patch files' and that 'none produced patch content directly usable without manual fixes' is reported without describing how patch validity was determined. Because this is a claim about all 17 evaluated models, please specify the validation protocol (e.g., whether patches were applied to the original source, compiled, rendered, or human-adjudicated), or soften the claim to match the evidence presented.","section":"5.3"}],"minor_comments":[{"comment":"The evaluation dimensions for Mockup2Code are named inconsistently: Table 2 uses 'layout, spacing, grid' while Figure 3 and Section 5.4 use 'layout, spacing, alignment'. Please unify the terminology.","section":"Table 2 / Figure 3"},{"comment":"The table caption should explicitly state the column order (Multi-step Reasoning, Agentic Action, General Visual Comprehension) and explain the icon legend; currently the ordering is only inferable from the caption after close reading.","section":"Table 3"},{"comment":"For the WebQA and Web Code Editing human-alignment checks (89% and 91%), please report the number of annotators, the sampling procedure across languages and difficulty strata, and the inter-annotator agreement, since a single agreement percentage against the LLM judge does not fully characterize reliability.","section":"Section 5.4"},{"comment":"The category name 'Agentic Action' is potentially misleading because the task is single-turn action grounding without environment feedback; consider renaming it to 'Action Grounding' or 'UI Action Comprehension'.","section":"3.2.1"},{"comment":"The PairBench citation used to justify the GPT4o judge selection shares co-authors with this submission; the paper should disclose this overlap or provide independent evidence for the judge's reliability.","section":"References"},{"comment":"The row label 'Web Code Editing886' is missing a separating space, and the table would benefit from explicit column totals for each task and language.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the moderate Mockup2Code judge correlation. If the authors can add stratified human validation or substantively temper the nested-layout claims, I would be inclined to accept after revision. Please also ensure the agentic-action summary and the 'no valid patch' claim are corrected or substantiated. The PairBench self-citation overlap is worth the editor's attention but is not, by itself, disqualifying."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"WebMMU is a solid contribution. It unifies three web tasks (VQA, mockup-to-code, and code editing) across four languages using expert-annotated, real-world data, and it does something no prior benchmark does: the code-editing task pairs screenshots with edit requests, forcing models to reason about visual context while modifying HTML/CSS/JS. The evaluation covers a wide range of open- and closed-source MLLMs, and the findings that models struggle with multi-step reasoning, grounding, and multilingual generation are credible given the strong human agreement on WebQA (89%) and code editing (91%). The dataset looks carefully constructed, with professional annotators and a multi-stage QA process. The soft spots are real but localized. The Mockup2Code evaluation rests on GPT4o-as-judge with only moderate human correlation (Spearman 0.43 average). The paper calls this acceptable because the task is subjective, but that does not justify using the judge to support fine-grained claims about nested-layout failure. No human validation is reported separately for the complex/nested strata, which is where the headline result lives. The simple-vs-complex gap in Figure 3 could be an artifact of judge bias. This is not fatal to the whole paper, but the authors should either collect human ratings on the complex examples or soften the hierarchy claim. There is also a text-vs-table inconsistency: Section 5.1 says agentic action rarely surpasses 10% accuracy, but Table 3 shows Qwen2.5VL-7B at 30.8% in English and several non-English agentic scores above 10%. The claim that agentic action is the hardest category still holds up (it is the lowest category in most languages), but the specific number is wrong. The limitations section also appears truncated in the manuscript; the authors should ensure the full caveats are visible. The PairBench citation for judge selection shares authors with this paper. That is not circular, because the judge is validated against human agreement here, but it is worth noting for transparency. Overall, WebMMU deserves serious refereeing. It is a real resource, and the central findings are plausible. I would send it to peer review with a request to fix the inconsistencies, add per-stratum human validation for Mockup2Code, and adjust the claims to match the evidence. If those issues are addressed, it will be a frequently cited dataset.","headline":"A genuinely useful multilingual web benchmark with one load-bearing caveat: the Mockup2Code judge correlates only moderately with humans, so the headline claims about nested-layout failure need either stronger validation or softer wording.","tokens_in":700,"tokens_out":830,"would_cite":true,"duration_ms":34689,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WebMMU, a new multilingual benchmark, shows that current multimodal models handle basic page reading but fail at reasoning, grounding, structure-preserving code edits, and multilingual design-to-code.","keywords":["multimodal large language models","web understanding benchmark","visual question answering","design-to-code generation","code editing","multilingual evaluation","UI grounding","web agents"],"falsifier":"Take a fresh sample of the paper's mockup-to-code outputs for nested layouts and have professional web developers rank them by whether the code reproduces the mockup's hierarchy and spacing. If the human ordering disagrees with the GPT-4o judge's 1-5 score ordering at the level implied by the reported rank correlation of about 0.43 on a larger sample, then the claimed failure on UI hierarchy is an artifact of the judge rather than a model limitation.","tokens_in":13281,"feed_emoji":"🌐","tokens_out":9437,"duration_ms":89112,"temperature":0.7,"pith_summary":"The paper introduces WebMMU, a benchmark that brings together three web tasks that prior work treats separately: visual question answering about real website screenshots, editing HTML/CSS/JavaScript code, and turning mockups into webpage code. The dataset is expert-annotated, covers four languages (English, Spanish, German, French) and 20 domains, and contains 6,102 WebQA questions, 1,602 code-editing cases, and 436 mockup-to-code instances. Its central claim is that current multimodal large language models do well at basic visual extraction but fail when a task needs multi-step reasoning, precise spatial grounding, structure-preserving code changes, or multilingual layout generation. If the claim is right, WebMMU gives the field a single testbed for web agents and web-development assistants, and the failure pattern it documents becomes a concrete target for future model-building.","feed_headline":"AI reads web pages but fails at reasoning and code edits","feed_subtitle":"Multilingual benchmark scores show models drop below 50% on multi-step reasoning, grounding, and layout-preserving edits.","key_machinery":"The central object is WebMMU itself, a unified benchmark whose annotations tie three tasks to the same real web pages: WebQA questions demand reasoning and coordinate-level grounding, Mockup2Code pairs simple and nested UI sketches with reference code, and Web Code Editing couples screenshots and source code with user-requested changes. Evaluation runs on GPT-4o-as-judge (binary correctness for WebQA, 1-5 layout fidelity for Mockup2Code, 1-5 functional correctness for code editing) alongside BLEU and TreeBLEU for structural code matching. The shared page-level starting point is what lets the benchmark compare performance across tasks and attribute gaps to model capability rather than to dataset curation.","core_discovery":"On the paper's own terms, WebMMU is a benchmark built from 2,059 expert-annotated real webpage screenshots in English, Spanish, German, and French, covering 20 domains and three tasks: WebQA (6,102 questions), Mockup2Code (436 instances), and Web Code Editing (1,602 edits). The central discovery is a consistent capability gap: models are near-competent at extracting visible facts from a page, but their accuracy collapses on multi-step reasoning (most below 50%, some near 0), on agentic action and coordinate grounding (almost always in single digits), on code edits that preserve function and structure (syntactically plausible but incomplete, and no model produced a directly applicable patch), and on mockup-to-code for nested layouts and multilingual content.","pith_inferences":["A direct comparison of the same models on WebMMU's real pages versus synthetic pages would test whether real-world layout variability, not raw difficulty, is what breaks models.","Because action questions are answered from a single screenshot with no environment, WebMMU measures action prediction, not successful interaction; a live-browser follow-up is needed to know whether grounding scores predict task completion.","The multilingual drop may partly come from OCR and font-rendering differences across languages; isolating perception from reasoning would show where cross-lingual investment belongs.","Collecting pairwise human preferences on mockup-to-code outputs, rather than rating on a 1-5 scale, would likely produce a stabler judge and clarify the nested-layout gap."],"forward_implications":["WebQA's agentic-action scores, which top out near 10% even for closed models, place precise spatial grounding of clickable elements as the least solved requirement for web agents.","The fact that no tested model produced a directly usable patch means automated web-code editing will need human oversight until models learn to emit valid diffs.","Mockup2Code results separate simple from nested layouts, so UI-hierarchy comprehension is a measurable bottleneck rather than a general design-to-code problem.","Multilingual versions of every task score below their English counterparts for most models, making cross-lingual robustness an independent axis of difficulty.","WebMMU lets future work compare models on all three tasks under one protocol, which fragmented benchmarks cannot."],"supporting_citations":[{"why":"Supplies the prior WebQA task that WebMMU contrasts with; its multihop QA frames WebMMU's reasoning questions over web content.","marker":"(Chang et al., 2022)"},{"why":"WebSRC is the prior web structural reading comprehension dataset that WebMMU extends beyond text retrieval toward grounding and multilingual UIs.","marker":"(Chen et al., 2021b)"},{"why":"VisualWebBench gives the page-understanding and grounding baseline that WebMMU says is still fragmented and English-centric.","marker":"(Liu et al., 2024)"},{"why":"Pix2Code establishes the design-to-code task that WebMMU claims to move from synthetic UIs to real web layouts.","marker":"(Beltramelli, 2018)"},{"why":"Sketch2Code motivates low-fidelity sketch-to-code, which WebMMU includes as its simple mockup setting.","marker":"(Jain et al., 2019)"},{"why":"Web2Code is the large-scale webpage-to-code dataset whose limited layout and language variety WebMMU claims to fix.","marker":"(Yun et al., 2024)"},{"why":"CodeEditorBench is the prior code-editing benchmark that WebMMU extends by adding visual context and HTML/CSS/JS specificity.","marker":"(Guo et al., 2024)"},{"why":"CanItEdit tests instruction-following for code edits, providing the editing paradigm WebMMU adapts to user-facing web code.","marker":"(Cassano et al., 2023)"},{"why":"Supports the choice of GPT-4o as the judge model by showing its alignment behavior, which backs the paper's evaluation strategy.","marker":"(Feizi et al., 2025)"},{"why":"Introduces the LLM-as-a-judge methodology that WebMMU's scoring protocol follows for all three tasks.","marker":"(Zheng et al., 2023)"}],"fun_headline_variants":["WebMMU: AI passes web reading, fails reasoning and edits","Multilingual web benchmark: models flunk reasoning and code edits","WebMMU: AI reads pages, but can't reason or edit code","AI can't edit code or reason: WebMMU multilingual test"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusions rest on the automatic GPT-4o judge's scores matching what expert human raters would say about correctness and layout fidelity; agreement is high for WebQA and code editing (89% and 91%) but only moderate for mockup-to-code (average rank correlation about 0.43), so the nested-layout findings are the part of the paper most exposed to judge error.","fun_headline_variants_meta":{"raw":{"variants":["WebMMU: AI passes web reading, fails reasoning and edits","Multilingual web benchmark: models flunk reasoning and code edits","WebMMU: AI reads pages, but can't reason or edit code","AI can't edit code or reason: WebMMU multilingual test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000844,"raw_usage":{"total_tokens":3632,"prompt_tokens":858,"completion_tokens":2774,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":474,"completion_tokens_details":{"reasoning_tokens":2698}},"tokens_in":474,"tokens_out":2774,"duration_ms":19883,"temperature":1.0,"reasoning_tokens":2698,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:09:37.302737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fresh sample of the paper's mockup-to-code outputs for nested layouts and have professional web developers rank them by whether the code reproduces the mockup's hierarchy and spacing. If the human ordering disagrees with the GPT-4o judge's 1-5 score ordering at the level implied by the reported rank correlation of about 0.43 on a larger sample, then the claimed failure on UI hierarchy is an artifact of the judge rather than a model limitation.","supporting_citations":[{"cited_title":"PairBench: Are Vision-Language Models Reliable at Comparing What They See?","cited_arxiv_id":"2502.15210","evidence_quote":"Supports the choice of GPT-4o as the judge model by showing its alignment behavior, which backs the paper's evaluation strategy."}],"review_version":1}