Pith. sign in

REVIEW 4 major objections 8 minor 5 cited by

MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents

T0 review · 4 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the main bottleneck for current GUI automation agents is precise visual grounding of on-screen elements rather than high-level planning, and that modular planner-plus-grounder systems are currently the most…

desk verdict A substantial multi-platform GUI benchmark worth having, with a confounded central finding and an EQA metric that needs an explicit task ordering. read the letter →

arxiv 2507.19478 v1 pith:XPQEDQ52 submitted 2025-07-25 cs.CV cs.CL

Xuehui Wang , Zhenyu Wu , JingJing Xie , Zichen Ding , Bowen Yang , Zehao Li , Zhaoyang Liu , Qingyun Li
show 20 more authors
Xuan Dong Zhe Chen Weiyun Wang Xiangyu Zhao Jixuan Chen Haodong Duan Tianbao Xie Chenyu Yang Shiqian Su Yue Yu Yuan Huang Yiqian Liu Xiao Zhang Yanting Zhang Xiangyu Yue Weijie Su Xizhou Zhu Wei Shen Jifeng Dai Wenhai Wang
This is my paper
classification cs.CVcs.CL
keywords GUIagentsvisualgroundingbenchmarkevaluationefficiencymetricmulti-platformtaskautomationmodularEQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MMBench-GUI is a benchmark that tries to establish where GUI automation agents actually fail. It organizes 8,123 tasks into four levels—reading screens, locating UI elements, automating single-application tasks, and coordinating multi-application workflows—across Windows, macOS, Linux, iOS, Android, and the Web, and it adds an Efficiency-Quality Area (EQA) metric that rewards finishing tasks in few steps rather than merely finishing them. Using this setup, the paper argues that accurate visual grounding is the critical determinant of task success for current agents: improving localization while keeping the planner fixed raises success by about 2.8×, whereas improving the planner while keeping localization fixed gives only 1.15×. It also argues that modular systems, where a general-purpose planner is paired with a specialized grounding module, currently outperform end-to-end native agents on automation tasks, and that every evaluated agent wastes many redundant steps even when it succeeds. The benchmark matters because it turns the vague complaint that agents cannot use computers into specific, measurable bottlenecks—grounding, efficiency, action-space coverage, and cross-context memory.

What carries the argument

The load-bearing machinery is the hierarchical benchmark itself, together with a new metric. The four levels are cumulative: L1 is multiple-choice GUI content understanding, L2 is element grounding scored by whether a predicted interaction point falls inside an annotated bounding box, L3 is single-application task automation in virtualized environments, and L4 is multi-application collaboration; this ladder lets performance drops be attributed to a specific competence. The Efficiency-Quality Area (EQA) metric is defined as the area under a step-wise recall curve: for an ordered set of tasks with step counts $t_i$ and success flags $s_i$, cumulative cost $T_k=\sum_{j=1}^k t_j$ is normalized by the global budget $T_{\max}=N t_{\max}$ to give $u_k$, and the score integrates the best cumulative success fraction $R(u)=\max_{k:u_k\le u} S_k/N$ over $u\in[0,1]$. EQA is the instrument that exposes how many steps agents waste, and it grounds the paper's call for early-stopping policies and cost-sensitive training.

What would settle it

An agent whose element-grounding accuracy is near zero but whose single-app automation success is high—or a planner swap that changes planning quality without changing success—would falsify the claim that grounding is the critical determinant. To test the metric's own reliability, permute the task order and check that EQA stays constant for identical trajectories.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a diagnosis: current GUI agents hit a perception wall, not a reasoning wall. In L2 element grounding, general-purpose vision-language models score near zero—GPT-4o averages 2.87% and Claude-3.7 averages 4.66%—while specialized grounders such as UGround-V1-7B reach 65.68%, and top grounding models like UI-TARS-72B-DPO reach 74.25%. The pattern carries into automation: GPT-4o alone reaches 6.13% success on L3, but GPT-4o paired with UGround-V1-7B reaches 17.50%, and the strongest system evaluated, GPT-4o + UI-TARS-1.5-7B, reaches 26.60%. Controlled comparisons are the paper's evidence for causation: with the planner fixed, upgrading the grounder multiplies success rate by 2.84×; with the grounder fixed, swapping in a stronger planner multiplies it by only 1.15×. The paper also claims that efficiency is a second bottleneck—EQA scores run well below success rates, successful trajectories carry redundant steps, and L4 cross-application success collapses to under 9%, caused less by perception than by missing actions such as tab switching and by memory loss across application boundaries.

Load-bearing premise

The headline diagnosis assumes that the L2 grounding score measures the same localization ability that L3 and L4 tasks actually require, so the observed correlation between grounding accuracy and automation success is causal rather than incidental.

Editorial extensions

If this is right

  • If grounding is the critical determinant, then the highest-leverage improvement for current agents is high-precision cross-platform element localization rather than a larger or stronger planner.
  • Benchmarks that report only success rate understate the efficiency gap; step-aware scores such as EQA should accompany success rate in future evaluations.
  • Modular planner-plus-grounder systems are a practical near-term architecture, and they will stay competitive until native agents close the localization gap.
  • Multi-application agents need explicit action-space support for tab and window switching plus persistent cross-context memory; without these, L4 success will remain in single digits.
  • Raising the step budget from 15 to 50 does not by itself fix grounding or planning failures, since success rates rise only modestly while redundant steps remain high.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A cheap screening test follows from the paper's correlation: because L2 grounding accuracy tracks L3 automation success, a new agent could be screened on L2 alone before expensive online evaluation, and the paper's data predict a monotone relationship.
  • The EQA reformulation in Equation 12 makes the score depend on the order of tasks through the global cumulative budget, but the paper never specifies an ordering rule; reproducing EQA comparisons therefore requires a canonical task order or a permutation-robust variant.
  • The 2.84× versus 1.15× asymmetry was measured with a small set of planners and grounders; if it persists across a wider model grid, grounding becomes a design law rather than an artefact of current baselines.
  • For deployment, the modular result suggests dynamic routing—letting a planner call an external grounding service only for hard localization cases—as a cheaper alternative to retraining the whole agent, though the paper does not test it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This paper introduces MMBench-GUI, a hierarchical benchmark for GUI automation agents with 8,123 tasks spanning four levels — L1 GUI content understanding (MCQA), L2 element grounding (bounding-box localization), L3 single-app task automation, and L4 multi-app collaboration — across Windows, macOS, Linux, iOS, Android, and Web. L1/L2 are offline; L3/L4 are online environments assembled from OSWorld, AndroidWorld, WebArena, and WindowsAgentArena, plus 70 newly authored macOS tasks. The paper proposes an Efficiency–Quality Area (EQA) metric that rewards online task success achieved with fewer steps, evaluates ten-plus open and closed models, and reports five findings centered on a central claim: accurate visual grounding is the critical determinant of task success, so modular systems pairing a general planner (GPT-4o) with a specialized grounder (UGround or UI-TARS-1.5-7B) currently lead. The other findings concern action-space limitations, step-efficiency and early stopping, difficulty generalization, and cross-application memory.

Significance. If the dataset and harness are released as promised, MMBench-GUI would be a genuinely useful community resource: it unifies offline perception/grounding evaluation with online automation under one protocol on five OS families, it reuses validated environments (OSWorld, AndroidWorld, WebArena, WindowsAgentArena) rather than building fragile new ones, and its 70-task macOS online set fills a real gap if existing macOS coverage is as thin as the paper claims. The hierarchical design (L1 to L4) enables correlational diagnosis of where agents fail, and the EQA idea — crediting agents that finish within budget — targets a real blind spot of success-rate-only benchmarks, even though the current formulation needs repair (Major Comment 1). The baseline sweep is broad, including open and closed models, and the effort to attach difficulty bands to L1/L2 is commendable. The paper's central causal claim about grounding is not yet supported by controlled comparisons, so the benchmark's value currently exceeds the reliability of its headline findings; with matched ablations and a repaired EQA definition it would be a strong contribution.

major comments (4)
  1. EQA as defined in Eqs. (7)–(10) is not invariant to the ordering of the task list, and the paper never specifies an ordering rule. Because u_k = T_k / T_max in Eq. (8) is the cumulative normalized cost up to task k, a successful task placed late in the list contributes 1 − u_k with u_k inflated by the steps of all preceding episodes, including failed ones; reordering the same multiset of per-task step counts changes the reported score, so the same model can obtain different EQA values depending only on evaluation order. Eq. (12) is mathematically a correct reformulation of Eq. (10), but it hides this dependence, and the paper then interprets (1/|C|) Σ_{i∈C} u_i as the average steps in which a task is completed (Eqs. (13)–(14) and Table 9); that reading is only valid for per-task times, not for the cumulative u_i of Eqs. (7)–(8), so the 'average remaining steps per successful task' and 'redundant step bill' interpretations in Finding 3 are not generally correct. Cross-budget comparisons (EQA15 vs EQA50, and the ΔEQ1/ΔEQ2 columns of Table 9) are additionally confounded because T_max = N · t_max rescales u_i mechanically when the step budget changes. I recommend redefining EQA with per-task normalized cost u_i = t_i / t_max, which is order-invariant and makes the physical interpretations in Eq. (13) literal, or, if the cumulative definition is kept, fixing a canonical task ordering, reporting sensitivity to random orderings, and restricting efficiency comparisons to a single budget.
  2. Finding 2's central claim — that grounding is the primary bottleneck — rests on the 2.8× (Δ = 17.25) versus 1.15× (Δ = 3.58) asymmetry, but the manuscript never states which model pairs these numbers come from, and the comparisons do not isolate grounding accuracy. Reconstructed from Table 7 (Max Step = 50 averages), the two legs appear to be GPT-4o versus GPT-4o+UI-TARS-1.5-7B (SR 9.35 → 26.60) and native UI-TARS-1.5-7B versus GPT-4o+UI-TARS-1.5-7B (23.02 → 26.60); these systems differ in action interface, prompt structure, and end-to-end versus modular architecture, not only in localization quality, so the 'single-variable control principle' announced at the start of §5 is not realized. In addition, L2 grounding is measured offline on static screenshots drawn from a different distribution than the online L3/L4 environments (OSWorld, AndroidWorld, WebArena, WindowsAgentArena, and the new macOS tasks), so the assumed transfer from L2 accuracy to L3/L4 success is never validated, and the claimed near-linear L2-to-L3 relationship in Figure 4 (right) is presented without a correlation statistic or per-model scatter data. The authors' own Future Work states that pinpointing failure causes requires step-by-step logging they have not yet done, which is consistent with my assessment that the causal attribution outruns the evidence. Please add matched ablations that vary only the localization source (e.g., predicted boxes versus ground-truth boxes with identical planner, prompts, and action space) or reframe the finding as correlational.
  3. There are internal numerical inconsistencies that block verification of the efficiency narrative. §4.4 and Finding 1 state that GPT-4o alone achieves SR of 4.05%/6.13% (Max Step 15/50), but Table 7 lists its average SR as 7.14%/9.35%; the quoted figures are actually GPT-4o's EQA values. §4.4 also states that increasing the budget from 15 to 50 'improves SR and EQA values for all models and settings,' which is contradicted by the tables: for example, Table 7 shows GPT-4o on Windows dropping from 5.56/3.27 to 3.49/2.26, Aguvis-72B on Windows from 4.14 to 3.49 and on Web from 9.03 to 8.39, and Table 8 shows GPT-4o+UGround-V1-7B on Web falling from 4.26/0.61 to 0.00/0.00. Since all results are single runs with no error bars, episode counts, or sampling temperature, small effects that matter for the conclusions — the 3.58-point SR gap behind the 1.15× leg in Finding 2, or several ΔEQ2 changes in Table 9 — are within plausible stochastic variation. Please correct the quoted numbers and report variance or the number of repeats.
  4. The near-zero L2 scores for general VLMs (GPT-4o 2.87, Claude-3.7 4.66 overall; e.g., 1.48% on Windows Basic) are a strong outlier relative to the same models' published results on the closely related ScreenSpot benchmark cited in §2, and these numbers carry the 'virtually blind' premise in Finding 1 and form the baseline of the 2.8× leg in Finding 2. Before these numbers are used as evidence, the protocol should be validated: report the output parse-failure rate for each model, manually inspect a sample of GPT-4o failures to separate true localization misses from coordinate-format or parsing artifacts, and consider a sanity condition (e.g., a set-of-marks variant or an element-choice variant) that measures a general VLM's grounding without requiring coordinate output. If the near-zero scores are genuine, the paper should explain why they differ so strongly from prior evaluations of the same models on similar grounding instructions.
minor comments (8)
  1. [§3.2, Eqs. (2)–(3)] L1 scores are chance-adjusted via α = (m_i − 1)/m_i, so a perfect response earns at most (m_i − 1)/m_i < 1 per item and the reported percentages are not raw accuracies; the text and Table 5 caption should state this rescaling and its range explicitly, and the two Acc definitions in Eqs. (2) and (3) should be unified.
  2. [Table 8] The 'Avg' columns aggregate over different platform subsets because '-' entries (missing switch_tab support) drop Web from several rows; state the averaging rule or compute averages over a common platform subset so rows are comparable across models.
  3. [§3.4, Eqs. (6)–(7)] The notation S is used for the application set in Eq. (6) (and S_m in Eq. (11)) while S_k denotes cumulative success in Eq. (7); rename one of the two to avoid confusion.
  4. [Table 7] The row label 'GPT-4o + UI-TARS-1.5-7B (2024)' is misattributed; UI-TARS-1.5-7B is a 2025 release (see References), so the year should be corrected.
  5. [§3.5 and §4.4] The claims of being 'the first benchmark that spans all widely used operating systems' and 'the first to include online task scenarios for macOS' should be qualified, since OSWorld already provides macOS support and the Linux tasks here are drawn from OSWorld; specify what the 70 new macOS tasks add beyond existing coverage.
  6. [§4.1] The evaluation protocol omits the sampling temperature, the number of episodes per task, API access dates for closed models, and the prompt templates; for online environments with stochastic rollouts these details are needed to reproduce Tables 7–8.
  7. [Figure 5] The caption says 'interpolating both the step budget and the success rate (SR) 100 times,' which does not match Eq. (10), where 101 equally spaced points (M = 101) are used for the area approximation; align the two descriptions.
  8. [§3.2, Step 4] The manual QA pass is described only as 'manual sampling'; report the number of items checked and any inter-annotator agreement, since L1 questions were generated and verified entirely by the same model families.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's claims are empirical measurements, not derivations reduced to their own inputs.

full rationale

This paper is an empirical evaluation study, not a formal derivation, so the circularity patterns that apply to self-defined metrics or fitted-parameter predictions largely do not arise. The central claim—that accurate visual grounding is a critical determinant of GUI task success—is supported by measured success rates and grounding accuracies of third-party models on a newly constructed benchmark. I checked the candidate circularity mechanisms explicitly. First, no self-definitional step appears: L2 grounding accuracy is measured by coordinate-in-bbox matching on static screenshots, while L3/L4 success is measured by environment state checks; neither metric is defined in terms of the other, and the correlation observed between them is empirical rather than constructed. Second, no fitted input is renamed as a prediction: the paper fits no parameters to a subset of data and then reports those fits as predictions; EQA is a metric definition, not a fitted model output. Third, self-citations exist—several authors are affiliated with prior work such as OSWorld, InternVL, MMBench, OS-Atlas, and Aguvis—but these citations supply evaluated models and benchmark environments, not the paper's conclusions. The conclusion about grounding is not justified by citing those works; it is justified by Tables 6–8, which compare externally developed systems. Fourth, no uniqueness theorem or ansatz is imported from the authors' prior work to force a modeling choice; the modular-versus-native framing is presented descriptively, not derived from an author-authored theorem. Fifth, the paper does not rename a known empirical pattern as organization: the hierarchical L1–L4 structure and the EQA metric are new measurement instruments, and their design is transparently specified. Two scientific weaknesses noted in the text are real but are not circularity: the EQA metric's dependence on task ordering is an evaluation-protocol ambiguity (Equation 8 uses cumulative budget over an ordered task list without an ordering rule), and the grounding-bottleneck attribution in Finding 2 is confounded because the compared systems differ in architecture and action space, not only in grounding accuracy. These are correctness and reproducibility risks, not instances of the paper's conclusions being equivalent to its inputs by construction. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical or formal entities are introduced. EQA is a metric, not an entity.

free parameters (1)
  • EQA interpolation grid size M = 101
    Equation 10 uses M=101 uniformly spaced points for the discrete approximation of EQA; chosen by hand, not derived. The approximation is insensitive to M at large values, but the choice is ad hoc.
assumptions (3)
  • domain assumption Screenshot-only input is sufficient to evaluate GUI agent competence
    Section 4 states the benchmark supplies only screenshots and task descriptions, excluding accessibility trees and HTML. This narrows the scope to vision-only agents and may not capture the performance of agents that use structured input.
  • domain assumption The four-level hierarchy decomposes GUI competence into separable abilities
    Section 3.1 assumes L1-L4 form a progressive hierarchy of skills, but the paper does not validate that these levels are independent or that difficulty increases monotonically.
  • domain assumption Per-task maximum step budget t_max is a valid global normalization for EQA
    Equation 8 normalizes cumulative steps by N times the per-task max. This treats all tasks as equal in cost and makes EQA depend on task ordering, a design choice not justified in the paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents." pith.science (2026). https://pith.science/paper/XPQEDQ52

@misc{pith2026250719478,
  author       = {Pith},
  title        = {Pith review of: MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XPQEDQ52}},
  note         = {Machine review of arXiv:2507.19478}
}
read the original abstract

We introduce MMBench-GUI, a hierarchical benchmark for evaluating GUI automation agents across Windows, macOS, Linux, iOS, Android, and Web platforms. It comprises four levels: GUI Content Understanding, Element Grounding, Task Automation, and Task Collaboration, covering essential skills for GUI agents. In addition, we propose a novel Efficiency-Quality Area (EQA) metric to assess GUI agent execution efficiency in online automation scenarios. Through MMBench-GUI, we identify accurate visual grounding as a critical determinant of overall task success, emphasizing the substantial benefits of modular frameworks that integrate specialized grounding modules. Furthermore, to achieve reliable GUI automation, an agent requires strong task planning and cross-platform generalization abilities, with long-context memory, a broad action space, and long-term reasoning playing a critical role. More important, task efficiency remains a critically underexplored dimension, and all models suffer from substantial inefficiencies, with excessive redundant steps even when tasks are ultimately completed. The integration of precise localization, effective planning, and early stopping strategies is indispensable to enable truly efficient and scalable GUI automation. Our benchmark code, evaluation data, and running environment will be publicly available at https://github.com/open-compass/MMBench-GUI.

Figures

Figures reproduced from arXiv: 2507.19478 by the authors.

Figure 1
Figure 1. MMBench-GUI: a hierarchical benchmark spanning four levels of increasing difficulty, covering over 8,000 tasks across six commonly used platforms. From L1 to L4, task complexity increases progressively, placing growing demands on the agent’s generalization and reasoning abilities. Based on this benchmark, we visualize the performance of various models in the right figure, clearly illustrating their respective streng… view at source ↗
Figure 2
Figure 2. Examples for L1&L2. Both of them are offline tasks. We provide examples from different platforms for each level. For clarity, some less critical fields are not shown here and full examples are available for download in our public repository. data sources, and evaluation protocols. Finally, we offer a statistical analysis of the proposed MMBench-GUI in Section 3.6. 3.1 HIERARCHICAL EVALUATION Existing GUI agents typi… view at source ↗
Figure 3
Figure 3. Examples for L3&L4. Tasks of these levels are evaluated in the virtual environment with an online manner. In L4, we provide two images belonging to different applications as examples to demonstrate that collaboration is the core aspect for this level. first style encapsulates the entire interaction process within multi-turn dialogues, while the second one condenses history into natural language and injects it into t… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Left: Demonstrates the relative contribution of visual grounding versus planning in driving performance gains under current conditions. We consider two experimental conditions—fixing the planner while varying the grounder, and vice versa—and examine how different combi…
Figure 5
Figure 5. Figure 5: EQA visualization across different models under L3 for different allowed steps. As discussed in Section 3.4, EQA reflects a combination of task completion and efficiency (i.e., the number of steps used upon completion). In practice, we compute it by interpolating both …
Figure 6
Figure 6. Figure 6: Difficulty-Gradient Heatmap. Models’ scores across difficulty levels are encoded with a single-hue palette whose saturation fades from high (dark) to low (light). Colored rectangles outline comparable model groups. Within and across these groups, the color consistently…

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Grounding Computer Use Agents on Human Demonstrations

    cs.LG 2025-11 conditional novelty 7.0 of 10

    GroundCUA, a 3.56M-element human-annotated desktop grounding dataset, and GroundNext models achieve strong UI grounding with less than one-tenth the SFT data of prior work.

  2. GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs

    cs.CV 2026-08 conditional novelty 6.0 of 10

    GUI-Lens improves GUI grounding by having a VLM iteratively select crops to zoom into, guided by OCR and UI detector references, and verifies its proposed clicks before outputting a coordinate.

  3. InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Training

    cs.CL 2026-01 conditional novelty 6.0 of 10

    InfiniteWeb auto-generates complete websites with tasks and dense-reward evaluators; training UI-TARS-1.5-7B on 600 such tasks lifts OSWorld from 24.5% to 31.4%.

  4. Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

    cs.AI 2026-07 conditional novelty 5.5 of 10

    A real-device-centric foundation GUI agent with hybrid GUI+CLI batched actions, AutoResearch data flywheel, online RL, and a proactive harness reaches SOTA mobile and competitive desktop/web scores.

  5. HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents

    cs.CV 2026-07 reject novelty 5.0 of 10

    A 3B-scale GUI agent reportedly scores 82.6% on AndroidWorld and 42% on real-device tasks, but the evidence is not independently verified and may overlap with its RL training.

Reference graph

Works this paper leans on

31 extracted references · 30 linked inside Pith · cited by 5 Pith papers

  1. [3]

    Windows agent arena: Evaluating multi-modal os agents at scale

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, et al. Windows agent arena: Evaluating multi-modal os agents at scale. arXiv preprint arXiv:2409.08264,

    arXiv

  2. [6]

    doi: 10.18653/v1/2024.acl-long.505

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.505. URL https://aclanthology.org/2024.acl-long. 505/. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114, 2023a. Xiang Deng, ...

    arXiv 2024

  3. [7]

    Assistgui: Task-oriented desktop graphical user interface automation

    Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, et al. Assistgui: Task-oriented desktop graphical user interface automation. arXiv preprint arXiv:2312.13108,

    arXiv

  4. [8]

    Navigating the digital world as humans do: Universal visual grounding for gui agents

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243,

    arXiv

  5. [9]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    25 MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919,

    arXiv

  6. [10]

    Screenqa: Large-scale question-answer pairs over mobile app screenshots

    Yu-Chung Hsiao, Fedir Zubach, Gilles Baechler, Victor Carbune, Jason Lin, Maria Wang, Srinivas Sunkara, Yun Zhu, and Jindong Chen. Screenqa: Large-scale question-answer pairs over mobile app screenshots. arXiv preprint arXiv:2209.08199,

    arXiv

  7. [12]

    Screenspot-pro: Gui grounding for professional high-resolution computer use

    Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.07981,

    arXiv

  8. [13]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick

    URL https://arxiv.org/abs/2411.17465. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740–755. Springer,

    arXiv

Show all 31 references
  1. [15]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216–233. Springer, 2024c. Quanfeng Lu, Wenqi ...

    arXiv

  2. [16]

    Chartqa: A bench- mark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244,

    arXiv

  3. [17]

    Ui-vision: A desktop- centric gui benchmark for visual perception and interaction

    26 MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M Tamer Özsu, Aishwarya Agrawal, David Vazquez, et al. Ui-vision: A desktop- centric ...

    arXiv

  4. [18]

    Ui-tars: Pioneering automated gui interaction with native agents

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326,

    arXiv

  5. [19]

    Androidworld: A dynamic benchmarking environment for autonomous agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573,

    arXiv

  6. [20]

    Os-genesis: Automating gui agent trajectory construction via reverse task synthesis

    Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412.19723,

    arXiv

  7. [21]

    Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows

    Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, et al. Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows. arXiv preprint arXiv:2505.19897,

    arXiv

  8. [22]

    Kimi-vl technical report.arXiv preprint arXiv:2504.07491,

    Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang, Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report.arXiv preprint arXiv:2504.07491,

    arXiv

  9. [23]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al

    URL https://arena.xlang.ai. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191,

    arXiv

  10. [24]

    Gui-actor: Coordinate-free visual grounding for gui agents

    Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, et al. Gui-actor: Coordinate-free visual grounding for gui agents. arXiv preprint arXiv:2506.03143,

    arXiv

  11. [25]

    Os-copilot: Towards generalist computer agents with self-improvement

    Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07456, 2024a. Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun,...

    arXiv

  12. [26]

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al

    URL https://arxiv.org/abs/ 2506.03569. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Adva...

    arXiv

  13. [27]

    Scaling computer-use grounding via user interface decomposition and synthesis

    27 MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, et al. Scaling computer-use grounding via user interface decomposition a...

  14. [28]

    Androidlab: Training and systematic benchmarking of android autonomous agents

    Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. Androidlab: Training and systematic benchmarking of android autonomous agents. arXiv preprint arXiv:2410.24024, 2024a. Yiheng Xu, Zekun Wang, Junli Wang, Dunjie ...

    arXiv

  15. [29]

    Breaking the data barrier–building gui agents through task generalization

    Junlei Zhang, Zichen Ding, Chang Ma, Zijie Chen, Qiushi Sun, Zhenzhong Lan, and Junxian He. Breaking the data barrier–building gui agents through task generalization. arXiv preprint arXiv:2504.10127,

    arXiv

  16. [30]

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al

    URL https://openreview.net/forum?id=piecKJ2DlB. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854,

    arXiv

  17. [31]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479,

    arXiv

  18. [2014]

    Autoglm: Autonomous foundation agents for guis

    Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, et al. Autoglm: Autonomous foundation agents for guis. arXiv preprint arXiv:2411.00820, 2024a. Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xi...

    arXiv

  19. [2021]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024b. Ka...

    arXiv

  20. [2022]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276,

    arXiv

  21. [2023]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923,

    arXiv

  22. [2024]

    Guicourse: From general vision language models to versatile gui agents

    Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. Guicourse: From general vision language models to versatile gui agents. arXiv preprint arXiv:2406.11317, 2024a. Xingyu Chen, Zihan Zhao, Lu Chen, Danyan...

    arXiv

  23. [2025]

    com/news/claude-3-7-sonnet

    URL https://www.anthropic. com/news/claude-3-7-sonnet . Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966,

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.