Pith. sign in

REVIEW 3 major objections 7 minor 11 cited by

GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GUI-Actor replaces text-coordinate generation with an attention-based action head that grounds a dedicated <ACTOR> token to visual patches, and this coordinate-free design lets a 7B model outperform a 72B model on ScreenSpot-Pro.

desk verdict Solid, genuinely novel attention-based grounding method; the missing supervision-target ablation and lack of error bars are the main soft spots, but the core claim holds up. read the letter →

arxiv 2506.03143 v1 pith:FNFSF2DY submitted 2025-06-03 cs.CL cs.AIcs.CV

Qianhui Wu , Kanzhi Cheng , Rui Yang , Chaoyun Zhang , Jianwei Yang , Huiqiang Jiang , Jian Mu , Baolin Peng
show 10 more authors
Bo Qiao Reuben Tan Si Qin Lars Liden Qingwei Lin Huan Zhang Tong Zhang Jianbing Zhang Dongmei Zhang Jianfeng Gao
This is my paper
classification cs.CLcs.AIcs.CV
keywords GUIvisualgroundingcoordinate-freeattention-basedactionhead<ACTOR>tokenmulti-patchsupervisionverifiervision-languagemodelsagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

GUI-Actor claims that GUI agents should stop predicting actions as x,y text coordinates and instead localize target elements directly in the visual space of the backbone. The paper introduces an attention-based action head: a dedicated token, placed in the text response, is trained to attend to the screenshot's visual patch tokens, so the attention map itself is the action proposal. Because all patches inside the ground-truth bounding box are treated as positive targets, the model tolerates the natural ambiguity of where inside an element a click is valid, and it can propose several candidate regions in one forward pass. The paper reports that this coordinate-free design beats coordinate-generation baselines on ScreenSpot-Pro, a high-resolution, out-of-distribution GUI grounding benchmark, with GUI-Actor-7B scoring 40.7 (Qwen2-VL backbone) and 44.6 (Qwen2.5-VL backbone), above UI-TARS-72B at 38.1; it also reports that freezing the backbone and training only the new action head is enough for competitive grounding.

What carries the argument

The central mechanism is the attention-based action head, built around a dedicated <ACTOR> token that acts as the contextual anchor for the intended action. The head applies a self-attention layer over visual patch features, projects the <ACTOR> hidden state and the contextualized patches into a shared embedding space via two MLPs, and computes an attention distribution over all patches. This distribution is trained with a KL target that is uniform over every patch inside the ground-truth bounding box, turning a single click label into dense spatial supervision. The same attention map yields multiple candidate regions in one forward pass, and a separately trained grounding verifier—a lightweight VLM that judges whether a marked point satisfies the instruction—selects among them at inference.

What would settle it

Train GUI-Actor identically but replace the uniform-over-box KL target with a Gaussian target centered on the element's functional click point; if element accuracy on ScreenSpot-Pro does not drop, the dense uniform target is not what drives the reported gains, and the assumption of compatibility with single-point extraction is falsified.

Watch

Extended reading notes

Core claim

The paper's central discovery is that GUI action grounding can be cast as a patch-selection problem instead of a coordinate regression problem, and that the attention distribution of a special token provides a high-quality patch-selection mechanism. GUI-Actor inserts <ACTOR_START><ACTOR><ACTOR_END> into the action string, takes the last-layer hidden state of <ACTOR>, projects it and the contextualized visual patch features into a shared space, and computes attention weights $a_i$. The training loss is next-token prediction plus the KL divergence between this attention distribution and a uniform distribution over all patches covered by the ground-truth bounding box. A lightweight grounding verifier then scores candidate click points derived from the attention map and picks the most plausible one. On ScreenSpot-Pro, the 7B model reaches 40.7 with Qwen2-VL and 44.6 with Qwen2.5-VL, surpassing UI-TARS-72B at 38.1, and the paper attributes the gains to explicit spatial-semantic alignment at the backbone's native patch resolution.

Load-bearing premise

The load-bearing premise is that training the attention map to spread probability uniformly over every patch inside the ground-truth box is compatible with extracting one precise, functional click point at inference; if attention mass drifts toward the box edges, the weighted center can land outside the actually clickable part of the element.

Editorial extensions

If this is right

  • A 7B-parameter GUI-Actor can surpass the much larger UI-TARS-72B on ScreenSpot-Pro, so model scale is not the dominant factor in GUI grounding; how actions are represented in the model matters.
  • Because grounding happens at the vision backbone's patch resolution, the method transfers to unseen screen sizes and layouts without coordinate recalibration.
  • Fine-tuning only the action head (about 100M parameters of the 7B model) with the backbone frozen yields competitive grounding, so grounding can be added to a general-purpose VLM without destroying its other capabilities.
  • The same attention map can nominate several distinct valid action regions in one forward pass, which enables cheap verification-based selection and multi-target operations such as clicking one of several visually equivalent options.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extending the paper's logic, the uniform-over-box KL target deserves a direct test against a center-weighted target such as a Gaussian over the bounding box, since the paper does not analyze whether the training distribution matches the single point extracted at inference.
  • If attention maps encode actionable regions this reliably, they could serve as a cheap source of pseudo-labels for grounding in websites and apps where DOM metadata is noisy or unavailable.
  • The same <ACTOR> attention mechanism could carry over to referring-expression segmentation or to instructions that name several elements, because the model already learns to distribute probability over all relevant patches.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper proposes GUI-Actor, a GUI visual-grounding method that replaces text-based coordinate generation with an attention-based action head. A dedicated <ACTOR> token is inserted into the VLM output sequence, and a lightweight MLP head computes an attention distribution over visual patch tokens; training uses a KL loss that encourages uniform attention over all patches intersecting the ground-truth bounding box. A separately trained grounding verifier clusters and scores top-attention candidates to select a final click point. The authors report results on ScreenSpot, ScreenSpot-v2, ScreenSpot-Pro, and a 49-task OSWorld subset, including ablations against Aguvis under matched training data, a lightweight LiteTrain variant, and sample-efficiency curves.

Significance. If the reported results are reproducible, GUI-Actor is a meaningful contribution: it offers a concrete alternative to coordinate generation, produces multiple candidate regions in one forward pass, and decouples grounding from text decoding. The ablations against Aguvis under the same data recipe are a strength, as is the explicit exclusion of Wave-UI overlap with test sets. The central architectural idea is plausible and the gains on ScreenSpot-Pro are sizable. However, the empirical evidence is weakened by the absence of statistical uncertainty, per-benchmark tuning of the verifier, and especially the lack of any ablation isolating the multi-patch supervision target from the attention architecture and the verifier.

major comments (3)
  1. [Section 3, Eq. (6); Appendix B; Appendix F.2] The training objective L_Action_Attn pushes the <ACTOR> attention distribution toward a uniform distribution over all patches intersecting the ground-truth box, but the only reported evaluation metric is Element Accuracy of a single click point extracted by the clustering/verifier procedure in Appendix F.2. The manuscript does not analyze this distribution-to-point conversion and does not ablate the supervision target. Since the paper attributes robustness and sample efficiency specifically to multi-patch supervision (Section 5, 'Improved Sample Efficiency'), an ablation with a single-center-patch or Gaussian target under identical architecture and data is needed, together with a failure analysis on elongated elements or boxes with large inactive margins, to substantiate the claim.
  2. [Section 5, Tables 1-4] All benchmark numbers are reported as point estimates from a single training run, with no standard deviations, confidence intervals, or significance tests. Several margins that support the headline superiority are small (e.g., Table 4, ScreenSpot-v2: GUI-Actor-7B 92.1 vs. Jedi-7B 91.7; Table 3, ScreenSpot-v2: GUI-Actor-2B 88.6 vs. UGround-V1-7B 87.6), and Tables 2-3 show UI-TARS-7B above GUI-Actor-7B on two of the three main benchmarks. The 'consistently outperform' claim is therefore not established to the usual standard; the authors should report multiple seeds and error bars, or at least a formal comparison for the key claims.
  3. [Section 5 Implementation Details; Appendix F.2] The verifier confidence threshold gamma is set to 0.95 for ScreenSpot-Pro and 0.8 for ScreenSpot/ScreenSpot-v2, and additional verifier crop sizes are introduced for ScreenSpot-Pro in Appendix G.1. Because the verifier contributes up to 13 points for LiteTrain on ScreenSpot-Pro (Table 5), the per-benchmark choice of gamma is a free parameter that could materially affect the headline generalization result. Please provide a sensitivity analysis over gamma and l_crop, and justify the choices using a validation split rather than the test benchmarks.
minor comments (7)
  1. [Abstract and Section 5] The statement that LiteTrain reaches performance 'comparable to previous state-of-the-art models' is broader than Table 5 supports: even with the verifier, LiteTrain-7B scores 81.3 vs. 88.4 on ScreenSpot and 83.8 vs. 90.3 on ScreenSpot-v2 relative to UI-TARS-72B, with only ScreenSpot-Pro close (35.8 vs. 38.1).
  2. [Appendix A] The sentence 'we introduce a simple mitigation strategy in this work' for small interface elements is never followed by a description of that strategy; please add the description or remove the sentence.
  3. [Table 8] The column header says '# of Screenshots' but the listed counts sum to exactly the reported 730K examples; clarify whether the column reports screenshots, queries, or generated examples.
  4. [Appendix C and Section 3] The note that the attention grid may need W//2 and H//2 due to the Naive Dynamic Resolution operation in Qwen2-VL should be formalized in the main text or in Appendix F.2, because the mapping from attention-map indices to screen coordinates is essential to the claimed coordinate-free grounding.
  5. [Section 3, Eq. (5)] No weight is specified for L_Action_Attn relative to L_NTP; please state whether the two losses were balanced and how the action-head warm-up interacts with the subsequent full fine-tuning stage.
  6. [Figure 3] The x-axis is labeled 'Percentage of Training Data (%)' with values 0-100, while the text says GUI-Actor 'reaches its final accuracy ... using only ~60% of the training data'; clarify how the percentage is computed and how many optimization steps each curve covers.
  7. [Appendix A] There is a typo in 'align natually' that should read 'align naturally'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: GUI-Actor's benchmark claims are externally evaluated, and its supervision targets are defined from external bounding-box annotations.

full rationale

GUI-Actor is an empirical architecture and training paper. Its central claims are supported by held-out benchmark accuracy, not derived from the model's own definitions. The action-head training target (Eq. 6 and Appendix B, Eq. 9) is a KL divergence toward a uniform distribution over ground-truth box patches; this target is defined from external bounding-box annotations, and the inference-time point extraction described in Appendix F.2 is a separate post-processing procedure rather than a quantity forced by the loss definition. The reported gains over coordinate-generation baselines (Tables 1-4 and 6) are comparisons against independently trained models on external benchmarks, with training data disjoint from the test sets and Wave-UI overlap explicitly excluded. The grounding verifier is trained on the public OS-Atlas dataset and only used for candidate selection; it is not fitted to the benchmarks, and the per-benchmark confidence thresholds (gamma 0.8 vs 0.95) are ordinary hyperparameters, not fitted predictions renamed as results. Although several prior works by the same group (SeeClick, OS-Atlas, Magma) are cited as baselines or data sources, none is invoked as a load-bearing uniqueness theorem or as the justification for the attention-head design. The acknowledged limitation about small elements (Appendix A) is a resolution constraint, not a circular reduction. Consequently, no specific circular step can be exhibited by quoting an equation that reduces to its own input; the correct finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 3 invented entities

The method's training and evaluation rely on standard supervised-learning assumptions, on the quality of public GUI datasets and benchmark annotations, on the comparability of published baseline numbers, and on the internal assumption that dot-product attention over projected patch features yields a spatially meaningful action map. No new physical or formal entities are postulated beyond the special tokens and learned modules; the confidence threshold and crop size are hand-chosen per benchmark and are the principal free parameters.

free parameters (3)
  • Verifier confidence threshold gamma = 0.95 (ScreenSpot-Pro), 0.8 (ScreenSpot/ScreenSpot-v2)
    Set per benchmark in Section F.2; not justified by a validation set, and performance of the verifier-augmented system depends on it.
  • Verifier crop size l_crop = 1000 px default; 1200 and 1400 for self-aggregation
    Chosen in Section F.2 and G.1; affects verifier scores and final selected point.
  • Candidate pool size K = 20
    Top-scoring patches considered in Section F.2.
assumptions (4)
  • domain assumption Ground-truth bounding boxes in the training datasets correctly delimit actionable regions.
    Multi-patch supervision relies on boxes from UGround, GUI-Env, AMEX, etc. (Table 7). Label noise would propagate to the attention target distribution.
  • ad hoc to paper Attention scores over visual patches are spatially meaningful after the self-attention layer and MLP projection.
    Equation 4 assumes the dot-product attention between <ACTOR> and patch projections yields a usable spatial map; no proof is offered.
  • domain assumption Published baseline scores are measured under the same element-accuracy protocol and are directly comparable.
    Tables 1-3 mix numbers taken from original papers, the UI-TARS benchmark, and the authors' own re-implementation; comparability is assumed.
  • domain assumption The verifier's True/False judgment on a marked crop transfers to unseen benchmarks.
    Verifier is trained on OS-Atlas (Section F.1) and applied to ScreenSpot variants; no analysis of domain shift is provided.
invented entities (3)
  • <ACTOR> token and <ACTOR_START>/<ACTOR_END> delimiters
    purpose: Contextual anchor whose hidden state is projected to attend over image patches (Section 3)
    New special tokens added to the VLM vocabulary; no external falsifiable prediction, only internal grounding signal.
  • Action attention head
    purpose: Computes attention distribution from <ACTOR> to visual patches, replacing text coordinates (Section 3)
    New learned module; its outputs are evaluated on benchmarks but it makes no independently testable prediction.
  • Grounding verifier
    purpose: Scores candidate regions by marking them and predicting True/False (Section 4)
    A separate finetuned VLM; again an internal module with no external falsifiable handle beyond benchmark performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents." pith.science (2026). https://pith.science/paper/FNFSF2DY

@misc{pith2026250603143,
  author       = {Pith},
  title        = {Pith review of: GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FNFSF2DY}},
  note         = {Machine review of arXiv:2506.03143}
}
read the original abstract

One of the principal challenges in building VLM-powered GUI agents is visual grounding, i.e., localizing the appropriate screen region for action execution based on both the visual content and the textual plans. Most existing work formulates this as a text-based coordinate generation task. However, these approaches suffer from several limitations: weak spatial-semantic alignment, inability to handle ambiguous supervision targets, and a mismatch between the dense nature of screen coordinates and the coarse, patch-level granularity of visual features extracted by models like Vision Transformers. In this paper, we propose GUI-Actor, a VLM-based method for coordinate-free GUI grounding. At its core, GUI-Actor introduces an attention-based action head that learns to align a dedicated <ACTOR> token with all relevant visual patch tokens, enabling the model to propose one or more action regions in a single forward pass. In line with this, we further design a grounding verifier to evaluate and select the most plausible action region from the candidates proposed for action execution. Extensive experiments show that GUI-Actor outperforms prior state-of-the-art methods on multiple GUI action grounding benchmarks, with improved generalization to unseen screen resolutions and layouts. Notably, GUI-Actor-7B even surpasses UI-TARS-72B (38.1) on ScreenSpot-Pro, achieving scores of 40.7 with Qwen2-VL and 44.6 with Qwen2.5-VL as backbones. Furthermore, by incorporating the verifier, we find that fine-tuning only the newly introduced action head (~100M parameters for 7B model) while keeping the VLM backbone frozen is sufficient to achieve performance comparable to previous state-of-the-art models, highlighting that GUI-Actor can endow the underlying VLM with effective grounding capabilities without compromising its general-purpose strengths.

Figures

Figures reproduced from arXiv: 2506.03143 by the authors.

Figure 1
Figure 1. Left: Model performance vs. training data scale on the ScreenSpot-Pro benchmark. Higher and more left is better; larger points indicate models with more parameters. We only show GUI-Actor models built upon Qwen2-VL here for fair comparison. With Qwen2.5-VL as the backbone, GUI￾Actor-3B/7B reaches scores up to 42.2/44.6 (without Verifier). Right: Illustration of action attention. GUI-Actor grounds target elements by … view at source ↗
Figure 2
Figure 2. Overview of GUI-Actor. (a) Illustration of how the action head works with a VLM for [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Accuracy Progression Over Training Steps. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: (a) Hit@1 and Hit@3 for different methods. For Aguvis baselines, we run inference 3 times [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Example visualizations from (a) ScreenSpot and (b)(c)(d) ScreenSpot-Pro. Each image [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Illustration of positive and negative examples used to train the grounding verifier. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Comparison with AGUVIS using the verifier. AGUVIS inferences 21 times for verification. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Grounding Computer Use Agents on Human Demonstrations

    cs.LG 2025-11 conditional novelty 7.0 of 10

    GroundCUA, a 3.56M-element human-annotated desktop grounding dataset, and GroundNext models achieve strong UI grounding with less than one-tenth the SFT data of prior work.

  2. OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

    cs.AI 2026-07 accept novelty 6.0 of 10

    VLM judges of CUA trajectories are systematically lenient; OSReward measures this with human gold, and OS-Shepherd open models close most of the reliability gap cheaply.

  3. GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL

    cs.LG 2026-02 conditional novelty 6.0 of 10

    Action-weighted SFT plus KL-regularized GRPO on a curated 81K reasoning dataset lifts open-source GUI agents by 11–22 points on online task-completion benchmarks.

  4. CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A decoupled planner-executor GUI agent, trained by per-app reinforcement learning followed by specialist-to-generalist distillation, lifts ScienceBoard success from about 7.6% to 21.0% average and 40% pass@8.

  5. MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents

    cs.CV 2025-07 conditional novelty 6.0 of 10

    MMBench-GUI provides a multi-platform, four-level benchmark and an efficiency-aware metric, and its experiments indicate visual grounding is the main bottleneck for current GUI agents.

  6. Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

    cs.AI 2026-07 conditional novelty 5.5 of 10

    A real-device-centric foundation GUI agent with hybrid GUI+CLI batched actions, AutoResearch data flywheel, online RL, and a proactive harness reaches SOTA mobile and competitive desktop/web scores.

  7. PrecisionCUA: Iterative Visual Refinement for Pixel-Precise Cursor Grounding in Code Editors

    cs.CV 2026-04 unverdicted novelty 5.0 of 10

    Multi-turn visual feedback refinement outperforms single-shot coordinate prediction for pixel-precise GUI grounding in complex coding environments.

  8. Enhancing Trustworthy GUI Grounding via Self-Critiqued Reinforcement Learning

    cs.CV 2025-10 conditional novelty 5.0 of 10

    HyperClick trains GUI grounding models with GRPO to output clicks plus confidence scores, jointly rewarding correct clicks and Brier-calibrated confidence, and reports SOTA accuracy on six of seven benchmarks with bet...

  9. How Auxiliary Reasoning Unleashes GUI Grounding in VLMs

    cs.CV 2025-09 conditional novelty 5.0 of 10

    Overlaying labeled grids and axes on screenshots substantially improves zero-shot GUI grounding in most VLMs, with the best variant zooming into grid cells.

  10. Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward

    cs.MA 2026-02 unverdicted novelty 4.0 of 10

    The paper surveys agent skills for LLMs across architecture, acquisition, deployment, and security, proposing a four-tier Skill Trust and Lifecycle Governance Framework to address vulnerabilities in community skills.

  11. GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Modeling GUI elements as Gaussian distributions instead of binary targets yields 92.0% (ScreenSpot), 93.3% (ScreenSpot-v2), and 47.5% (ScreenSpot-Pro) for a 7B model, outperforming UI-TARS-72B by a relative 24.7% on t...

Reference graph

Works this paper leans on

60 extracted references · 14 canonical work pages · cited by 11 Pith papers

  1. [46]

    Attention-driven gui grounding: Leveraging pretrained multimodal large language models without fine-tuning

    Hai-Ming Xu, Qi Chen, Lei Wang, and Lingqiao Liu. Attention-driven gui grounding: Leveraging pretrained multimodal large language models without fine-tuning. InProceedings of the AAAI Conference on Artificial Intelligence, pages 8851–8859, 2025. 13

    work page 2025

  2. [1]

    Ufo: A ui-focused agent for windows os interaction.arXiv preprint arXiv:2402.07939, 2024

    Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. Ufo: A ui-focused agent for windows os interaction.arXiv preprint arXiv:2402.07939, 2024

    arXiv 2024

  3. [2]

    Ufo2: The desktop agentos.arXiv preprint arXiv:2504.14603, 2025

    Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, et al. Ufo2: The desktop agentos.arXiv preprint arXiv:2504.14603, 2025

    arXiv 2025

  4. [3]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. InICLR 2024 Workshop on Large Language Model (LLM) Agents

    work page 2024

  5. [4]

    Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614, 2024

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614, 2024

    arXiv 2024

  6. [5]

    Large language model-brained gui agents: A survey.arXiv preprint arXiv:2411.18279, 2024

    Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, et al. Large language model-brained gui agents: A survey.arXiv preprint arXiv:2411.18279, 2024

    arXiv 2024

  7. [6]

    Large action models: From inception to implementation.arXiv preprint arXiv:2412.10047, 2024

    Lu Wang, Fangkai Yang, Chaoyun Zhang, Junting Lu, Jiaxu Qian, Shilin He, Pu Zhao, Bo Qiao, Ray Huang, Si Qin, et al. Large action models: From inception to implementation.arXiv preprint arXiv:2412.10047, 2024

    arXiv 2024

  8. [7]

    Aguvis: Unified pure vision agents for autonomous gui interaction.arXiv preprint arXiv:2412.04454, 2024

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction.arXiv preprint arXiv:2412.04454, 2024

    arXiv 2024

Show all 60 references
  1. [8]

    Navigating the digital world as humans do: Universal visual grounding for gui agents.arXiv preprint arXiv:2410.05243, 2024

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents.arXiv preprint arXiv:2410.05243, 2024

    2024 arXiv

  2. [9]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  3. [10]

    Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  4. [11]

    Roi pooled correlation filters for visual tracking

    Yuxuan Sun, Chong Sun, Dong Wang, You He, and Huchuan Lu. Roi pooled correlation filters for visual tracking. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5783–5791, 2019

    2019

  5. [12]

    An empirical study of spatial attention mechanisms in deep networks

    Xizhou Zhu, Dazhi Cheng, Zheng Zhang, Stephen Lin, and Jifeng Dai. An empirical study of spatial attention mechanisms in deep networks. InProceedings of the IEEE/CVF international conference on computer vision, pages 6688–6697, 2019

    2019

  6. [13]

    Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573, 2024

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Al- ice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573, 2024

    2024 arXiv

  7. [14]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023

    2023 arXiv

  8. [15]

    Mind2web: Towards a generalist agent for the web.Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web.Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    2023

  9. [16]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Systems, 37: 52040–52094, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Sy...

    2024

  10. [17]

    Os-copilot: Towards generalist computer agents with self-improvement.arXiv preprint arXiv:2402.07456, 2024

    Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement.arXiv preprint arXiv:2402.07456, 2024

    2024 arXiv

  11. [18]

    Agentstore: Scalable integration of heterogeneous agents as specialized generalist computer assistant.arXiv preprint arXiv:2410.18603, 2024

    Chengyou Jia, Minnan Luo, Zhuohang Dang, Qiushi Sun, Fangzhi Xu, Junlin Hu, Tianbao Xie, and Zhiyong Wu. Agentstore: Scalable integration of heterogeneous agents as specialized generalist computer assistant.arXiv preprint arXiv:2410.18603, 2024

    arXiv

  12. [19]

    A survey of neural code intelligence: Paradigms, advances and beyond.arXiv preprint arXiv:2403.14734, 2024

    Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: Paradigms, advances and beyond.arXiv preprint arXiv:2403.14734, 2024

    2024 arXiv

  13. [20]

    Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows.arXiv preprint arXiv:2505.19897, 2025

    Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, et al. Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows.arXiv preprint arXiv:2505.19897, 2025

    2025 arXiv

  14. [21]

    Appagent: Multimodal agents as smartphone users

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–20, 2025

    2025

  15. [22]

    Api agents vs

    Chaoyun Zhang, Shilin He, Liqun Li, Si Qin, Yu Kang, Qingwei Lin, and Dongmei Zhang. Api agents vs. gui agents: Divergence and convergence.arXiv preprint arXiv:2503.11069, 2025

    2025 arXiv

  16. [23]

    Guicourse: From general vision language models to versatile gui agents.arXiv preprint arXiv:2406.11317, 2024

    Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, et al. Guicourse: From general vision language models to versatile gui agents.arXiv preprint arXiv:2406.11317, 2024

    2024 arXiv

  17. [24]

    Amex: Android multi-annotation expo dataset for mobile gui agents.arXiv preprint arXiv:2407.17490, 2024

    Yuxiang Chai, Siyuan Huang, Yazhe Niu, Han Xiao, Liang Liu, Dingyu Zhang, Peng Gao, Shuai Ren, and Hongsheng Li. Amex: Android multi-annotation expo dataset for mobile gui agents.arXiv preprint arXiv:2407.17490, 2024

    2024 arXiv

  18. [25]

    Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices.arXiv preprint arXiv:2406.08451, 2024

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices.arXiv preprint arXiv:2406.08451, 2024

    2024 arXiv

  19. [26]

    Ferret-ui: Grounded mobile ui understanding with multimodal llms

    Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms. InEuropean Conference on Computer Vision, pages 240–255. Springer, 2024. 12

    2024

  20. [27]

    Androidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36: 59708–59728, 2023

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Androidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36: 59708–59728, 2023

    2023

  21. [28]

    Android in the zoo: Chain-of-action-thought for gui agents.arXiv preprint arXiv:2403.02713, 2024

    Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thought for gui agents.arXiv preprint arXiv:2403.02713, 2024

    2024 arXiv

  22. [29]

    Vem: Environment-free exploration for training gui agent with value environment model.arXiv preprint arXiv:2502.18906, 2025

    Jiani Zheng, Lu Wang, Fangkai Yang, Chaoyun Zhang, Lingrui Mei, Wenjie Yin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. Vem: Environment-free exploration for training gui agent with value environment model.arXiv preprint arXiv:2502.18906, 2025

    2025 arXiv

  23. [30]

    Os-genesis: Automating gui agent trajectory construction via reverse task synthesis.arXiv preprint arXiv:2412.19723, 2024

    Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis.arXiv preprint arXiv:2412.19723, 2024

    2024 arXiv

  24. [31]

    Vision-language models can self-improve reasoning via reflection.arXiv preprint arXiv:2411.00855, 2024

    Kanzhi Cheng, Yantao Li, Fangzhi Xu, Jianbing Zhang, Hao Zhou, and Yang Liu. Vision-language models can self-improve reasoning via reflection.arXiv preprint arXiv:2411.00855, 2024

    2024 arXiv

  25. [32]

    Breaking the data barrier–building gui agents through task generalization.arXiv preprint arXiv:2504.10127, 2025

    Junlei Zhang, Zichen Ding, Chang Ma, Zijie Chen, Qiushi Sun, Zhenzhong Lan, and Junxian He. Breaking the data barrier–building gui agents through task generalization.arXiv preprint arXiv:2504.10127, 2025

    2025 arXiv

  26. [33]

    Webvoyager: Building an end-to-end web agent with large multimodal models.arXiv preprint arXiv:2401.13919, 2024

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models.arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  27. [34]

    Seeclick: Harnessing gui grounding for advanced visual gui agents.arXiv preprint arXiv:2401.10935, 2024

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents.arXiv preprint arXiv:2401.10935, 2024

    2024 arXiv

  28. [35]

    Computer-using agent

    OpenAI. Computer-using agent. Available at: https://openai.com/index/computer-using-agent, 2025

    2025

  29. [36]

    Claude computer use

    Anthropic. Claude computer use. Available at: https://www.anthropic.com/news/developing-computer-use, 2024

    2024

  30. [37]

    Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic.arXiv preprint arXiv:2306.15195, 2023

    2023 arXiv

  31. [38]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  32. [39]

    Showui: One vision-language-action model for gui visual agent.arXiv preprint arXiv:2411.17465, 2024

    Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-action model for gui visual agent.arXiv preprint arXiv:2411.17465, 2024

    2024 arXiv

  33. [40]

    Aria-ui: Visual grounding for gui instructions.arXiv preprint arXiv:2412.16256, 2024

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions.arXiv preprint arXiv:2412.16256, 2024

    2024 arXiv

  34. [41]

    Tinyclick: Single-turn agent for empowering gui automation.arXiv preprint arXiv:2410.11871, 2024

    Pawel Pawlowski, Krystian Zawistowski, Wojciech Lapacz, Marcin Skorupa, Adam Wiacek, Sebastien Postansque, and Jakub Hoscilowicz. Tinyclick: Single-turn agent for empowering gui automation.arXiv preprint arXiv:2410.11871, 2024

    arXiv

  35. [42]

    Think twice, click once: Enhancing gui grounding via fast and slow systems.arXiv preprint arXiv:2503.06470, 2025

    Fei Tang, Yongliang Shen, Hang Zhang, Siqi Chen, Guiyang Hou, Wenqi Zhang, Wenqiao Zhang, Kaitao Song, Weiming Lu, and Yueting Zhuang. Think twice, click once: Enhancing gui grounding via fast and slow systems.arXiv preprint arXiv:2503.06470, 2025

    2025 arXiv

  36. [43]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  37. [44]

    Scaling computer-use grounding via user interface decomposition and synthesis, 2025

    Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, and Caiming Xiong. Scaling computer-use grounding via user interface decomposition and synthesis, 2025

    2025

  38. [45]

    Os-atlas: A foundation action model for generalist gui agents

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024

    arXiv

  39. [47]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  40. [48]

    Screenspot-pro: Gui grounding for professional high-resolution computer use.arXiv preprint arXiv:2504.07981, 2025

    Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use.arXiv preprint arXiv:2504.07981, 2025

    2025 arXiv

  41. [49]

    Introducing gpt-4o

    OpenAI. Introducing gpt-4o. Available at: https://openai.com/index/hello-gpt-4o, 2024

    2024

  42. [50]

    The dawn of gui agent: A preliminary case study with claude 3.5 computer use, 2024

    Siyuan Hu, Mingyu Ouyang, Difei Gao, and Mike Zheng Shou. The dawn of gui agent: A preliminary case study with claude 3.5 computer use, 2024. URLhttps://arxiv.org/abs/2411.10323

    2024 arXiv

  43. [51]

    Introducing gemini 2.0

    Google. Introducing gemini 2.0. Available at: https://blog.google/technology/google-deepmind/google- gemini-ai-update-december-2024, 2024

    2024

  44. [52]

    Magma: A foundation model for multimodal ai agents.arXiv preprint arXiv:2502.13130, 2025

    Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: A foundation model for multimodal ai agents.arXiv preprint arXiv:2502.13130, 2025

    2025 arXiv

  45. [53]

    Windows agent arena: Evaluating multi-modal os agents at scale.arXiv preprint arXiv:2409.08264, 2024

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, et al. Windows agent arena: Evaluating multi-modal os agents at scale.arXiv preprint arXiv:2409.08264, 2024

    2024 arXiv

  46. [54]

    Omniparser for pure vision based gui agent.arXiv preprint arXiv:2408.00203, 2024

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent.arXiv preprint arXiv:2408.00203, 2024

    2024 arXiv

  47. [55]

    On the effects of data scale on ui control agents.Advances in Neural Information Processing Systems, 37:92130–92154, 2024

    Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on ui control agents.Advances in Neural Information Processing Systems, 37:92130–92154, 2024

    2024

  48. [56]

    The fineweb datasets: Decanting the web for the finest text data at scale.Advances in Neural Information Processing Systems, 37:30811–30849, 2024

    Guilherme Penedo, Hynek Kydlí ˇcek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro V on Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale.Advances in Neural Information Processing Systems, 37:30811–30849, 2024

    2024

  49. [57]

    Uibert: Learning generic multimodal representations for ui understanding.arXiv preprint arXiv:2107.13731, 2021

    Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, et al. Uibert: Learning generic multimodal representations for ui understanding.arXiv preprint arXiv:2107.13731, 2021

    2021 arXiv

  50. [58]

    Mapping natural language instructions to mobile ui action sequences.arXiv preprint arXiv:2005.03776, 2020

    Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences.arXiv preprint arXiv:2005.03776, 2020

    2005 arXiv

  51. [59]

    click the button to create a new project

    Yang Li, Gang Li, Luheng He, Jingjie Zheng, Hong Li, and Zhiwei Guan. Widget captioning: Generating natural language description for mobile user interface elements.arXiv preprint arXiv:2010.04295, 2020. A Limitations Our attention-based action generation is particularly well-s...

    2010 arXiv

  52. [60]

    ) We set lcrop = 1000pixels for all tasks. To reduce the computational cost, if a candidate position achieves a high confidence score (e.g., s(I, x)>0.95 ), we immediately return that position without evaluating the remaining candidates. In our experiments, we set the threshol...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.