Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read State-of-the-art computer-using agents solve only 11.4 percent of a new 416-task desktop benchmark, and nearly none of the tasks that demand adaptation or orchestration.

desk verdict OS-MAP is a genuinely useful benchmark with a likely-robust headline result, but the level-wise capability story is confounded with task length and step budget and needs rework before it can carry the paper's conclusions. read the letter →

arxiv 2507.19132 v1 pith:7GQ7QF62 submitted 2025-07-25 cs.AI cs.CLcs.CVcs.HC

Xuetian Chen , Yinghao Chen , Xinfeng Yuan , Zhuo Peng , Lu Chen , Yuekeng Li , Zhoujia Zhang , Yingqian Huang
show 7 more authors
Leyan Huang Jiaqing Liang Tianbao Xie Zhiyong Wu Qiushi Sun Biqing Qi Bowen Zhou
This is my paper · ORCID
classification cs.AIcs.CLcs.CVcs.HC
keywords computer-usingagentsGUIbenchmarkautomationlevelsgeneralizationscopeevaluationmatrixdesktopvision-languagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that current computer-using agents fail much harder than existing benchmarks suggest, and that the reason is not only difficulty but a mismatch between task structure and the capabilities agents actually have. To make this visible, it introduces OS-MAP, a benchmark of 416 realistic desktop tasks ranked on a five-level automation scale (from simple execution to full orchestration) and sorted by how much generalization each task demands. On this benchmark the strongest available agent, UI-TARS-72B, completes only 11.4% of tasks overall, and almost none of the higher-level tasks that require adapting to surprises or coordinating several applications. The paper argues that such a structured evaluation matrix, rather than a flat collection of tasks, is what the field needs to identify which capability is missing at each level and to guide progress toward reliable deployment.

What carries the argument

The central object is the OS-MAP evaluation matrix, which organizes tasks along a five-level automation taxonomy (L1 reactive execution, L2 deterministic planning, L3 adaptive behavior, L4 global orchestration, and L5 proactive companionship, modeled on the SAE driving-automation scale) crossed with three generalization scopes (S1 narrow, S2 domain-level, S3 general) derived from a demand hierarchy over six real-world usage domains. This matrix is what carries the argument: by decoupling autonomy depth from breadth, it turns a pool of 416 tasks into a diagnostic instrument that attributes failures to specific capability gaps, and it defines an upper-right high-value region that no current agent reaches.

What would settle it

Re-run the 416 tasks with the L3 and L4 instructions rewritten as explicit step-by-step click sequences while keeping the underlying operations identical; if success rates jump to L1/L2 levels, the automation ladder is measuring instruction ambiguity rather than agent capability. In parallel, have a second annotator team independently relabel all tasks and report agreement on level assignments.

Watch

Extended reading notes

Core claim

The paper's central claim is that state-of-the-art computer-using agents remain far from practical deployment, and that this gap is best understood along two independent dimensions: how much autonomy a task requires and how broadly a task generalizes across real user demand. Evaluating ten agent configurations on 416 tasks spanning 15 desktop applications, the paper finds the strongest system reaches an 11.4% overall success rate, with a stepwise decline across its automation levels: 48.0% on level-1 execution tasks, 14.0% on level-2 planning, 1.0% on level-3 adaptation, and 0.0% on level-4 orchestration, against human performance of about 71.9% overall. The failure analysis attributes the decline to distinct bottlenecks at each level, such as grounding non-textual UI elements, avoiding similar but incorrect options, recovering with fallback strategies when the obvious action fails, and decomposing long workflows with correct dependency ordering, rather than to a single missing ingredient.

Load-bearing premise

The entire hierarchy rests on the assumption that the L1-L4 labels assigned by annotators measure distinct capabilities such as perception, planning, adaptation, and orchestration, rather than just longer instructions, more steps, or vaguer wording.

Editorial extensions

If this is right

  • Flat-list benchmark scores overstate agent readiness: the 11.4% ceiling on OS-MAP, with near-zero performance on adaptive and orchestration tasks, is the number that any practical deployment claim must beat.
  • The field gets a targeted roadmap: grounding and basic planning are partially solved, while fallback behavior under deviation (L3) and task decomposition with context switching (L4) are the binding constraints.
  • Open-source GUI-trained models such as UI-TARS-72B can match or beat much larger proprietary general VLMs for end-to-end computer use, so scaling general models is not the only route forward.
  • The new CALL_USER action makes human-in-the-loop evaluation possible, allowing benchmark tasks to include realistic permission boundaries such as login walls.
  • Because the matrix is extensible, new tasks and scenarios can be added to underexplored cells, and L5 proactive behavior is staked out as the next unmeasured frontier.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the automation-level labels mainly track instruction vagueness and task length rather than distinct cognitive capabilities, the reported stepwise decline could shrink substantially when the same underlying steps are re-issued as concrete click-by-click instructions.
  • The mix of single-app versus multi-app tasks positions OS-MAP to quantify the jump from within-application competence to cross-application coordination, which may turn out to be the real threshold for everyday usefulness.
  • A natural extension is to reuse the taxonomy on other surfaces such as mobile, web, and terminal agents, giving the community a common vocabulary for automation levels across platforms.
  • The paper reports no inter-annotator agreement on level assignments, so a simple relabeling study by an independent team would determine whether the ladder measures the agent or measures the wording of the prompt.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces OS-MAP, a benchmark for computer-using agents that organizes 416 tasks across 15 Ubuntu applications along two dimensions: a five-level automation taxonomy (L1 execution through L5 proactivity, with L5 not yet instantiated) and a three-level generalization scope derived from a user-demand hierarchy. The benchmark is built on the OSWorld VM infrastructure, uses screenshot-only observations and atomic GUI actions, and evaluates a range of general-purpose VLMs, a GUI-specialized model, and planning-grounding agents. The headline empirical result is that the best agent achieves only 11.4% overall success, with strong performance only on the simplest level (48% at L1) and near-zero performance at L3 and L4, while human performance is reported at 71.9%. The paper also provides a failure analysis attributing breakdowns to instruction following, hallucination, grounding, planning, adaptation, and orchestration.

Significance. If the benchmark and its level-wise analysis are sound, OS-MAP is a valuable community resource: it provides an executable, state-based evaluation environment; a task taxonomy defined before experimentation; hand-labeled tasks; a measured human baseline rather than an assumed one; and public code, environments, baselines, and data. The aggregate finding that current agents fail most realistic desktop tasks is credible and important, and the two-dimensional evaluation matrix is a useful organizing framework. However, the paper's most distinctive claim, the stepwise capability decline across automation levels, is currently under-identified because it is confounded with task horizon and instruction detail, and several metadata inconsistencies weaken confidence in the supporting statistics. The contribution is therefore significant but conditional on additional validation and reporting.

major comments (5)
  1. [Section 3.5, Table 1, Appendix C.3] The benchmark statistics are internally inconsistent. Table 1 reports 283 single-app and 154 multi-app tasks, which sum to 437 rather than the stated total of 416, and the reported percentages (62.3% and 37.7%) do not match either 283/416 or 154/416. Section 3.5 states that 138 tasks were authored and 296 were relabeled from OSWorld, summing to 434, whereas Table 1 and Appendix C.3 give 161 authored and 255 OSWorld-derived tasks, summing to 416. These discrepancies must be reconciled, because the per-level counts and source split are used to support the paper's descriptive claims about the benchmark's composition.
  2. [Section 4.2, Table 3, Appendix C.1, Appendix D.1] The stepwise decline across automation levels, which is the paper's central empirical discovery, is confounded with two design choices. Appendix C.1 states that higher-level tasks omit instruction details and include misleading cues, and Appendix D.1 assigns different maximum interaction steps of 15, 15, 30, and 50 for L1 through L4. Because success is a single binary outcome after a complete trajectory, a constant per-step reliability q already yields expected end-to-end success of approximately q^N, so longer and more underspecified L3 and L4 tasks are expected to show near-zero success without any level-specific capability deficit. The paper should report per-level task horizons, distributions of trajectory lengths, success conditioned on reaching intermediate milestones, or a regression controlling for instruction length and step budget before attributing the decline to distinct capabilities such as adaptation and orchestration.
  3. [Section 3.4, Appendix C] No inter-annotator agreement is reported for the L1-L4 and S1-S3 labels, even though the curation process involved nine student annotators. If the automation-level labels mainly track instruction detail or trajectory length rather than the intended capabilities, the monotonic decline in Table 3 would be an artifact of the labeling scheme. The authors should report agreement statistics such as Cohen's kappa on a sample of tasks, and ideally provide an external validation showing that the levels correspond to independently measured difficulty or capability requirements.
  4. [Table 3 and Section 4.2] The human performance baseline of 71.9% is a load-bearing comparison for the paper's conclusion that agents remain far from practical deployment, but no protocol is provided for how it was obtained. The authors should report the number of human participants, the instructions they received, whether they used the same screenshot-only VM interface, and how edge cases or disagreements were handled. Without this information, the human baseline is not reproducible and the size of the human-agent gap cannot be assessed.
  5. [Table 3, Section 4.2] The per-level sample sizes are small, and the headline cells are correspondingly noisy. With L1 containing 25 tasks and L4 containing 42 tasks, the UI-TARS-72B results of 48% at L1 and 0% at L4 correspond to 12 successes out of 25 and 0 out of 42. Confidence intervals or bootstrap estimates should be reported so that the fine-grained ordering among agents is not over-interpreted, particularly because several near-zero percentages differ by only one or two successful tasks across the evaluated systems.
minor comments (5)
  1. [Appendix E.1, Table 4] The system prompt examples use the action type MOUSE_MOVE, while Table 4 defines the action type MOVE_TO. If these are aliases, that should be stated; otherwise the prompt and the action-space definition need to be aligned.
  2. [Abstract and Section 3.1] The paper calls the taxonomy a five-level automation taxonomy, but OS-MAP only instantiates L1 through L4 and defers L5 to future work. The main text should state this limitation more prominently when the taxonomy is introduced, not only in the L5 paragraph.
  3. [Section 3.5, Table 1] The average instruction length is reported as 34.3 words without any dispersion measure or per-level breakdown. Reporting standard deviations and per-level averages would help readers evaluate the confound between instruction detail and automation level.
  4. [Appendix B.5] The CALL_USER action is described as delegating control to a simulated human supervisor, but the appendix then says that during Google account login a script autofills the credentials. Clarify whether CALL_USER invokes a human, a script, or both, since this affects the interpretation of human-in-the-loop evaluation.
  5. [Throughout] The benchmark name is rendered inconsistently as OS-MAP, OS-M AP, and OS-MAp across the title, abstract, tables, and references. A single consistent spelling should be used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's headline results are measured outcomes in an executable environment, not derived from fitted inputs or load-bearing self-citations.

full rationale

The paper's central claims are empirical: it constructs a task benchmark, defines automation levels and generalization scopes before experimentation, runs multiple agents in a VM-based executable environment, and reports measured success rates with a measured human baseline (Table 3). There is no fitted parameter that is subsequently renamed as a prediction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The stepwise decline across L1-L4 is an observed outcome, not an identity forced by the level definitions: the L-level definitions describe intended capability demands, while task success is determined by state-based evaluation against reference states. Self-citations to OSWorld and OS-Copilot are infrastructure provenance and benchmark-construction references; they do not carry the central empirical claim, and the OSWorld-derived subset is explicitly filtered and relabeled rather than assumed equivalent. The skeptical concern that L3/L4 tasks are confounded with longer horizons, omitted instruction details, and misleading cues (Appendix C.1, D.1) is a validity threat to the level-specific bottleneck interpretation, but it is not circularity: the measured success rates are not equal to the task labels by construction, and no derivation step reduces to its own input. Therefore the analysis finds no significant circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 2 invented entities

The central claims rest on the benchmark's construct validity: the OSWorld environment represents real desktop use, the L1-L4 labels capture distinct agent capabilities, the demand hierarchy matches actual user needs, and the human baseline was measured properly. No free parameters are fitted to force the headline results; the only hand-chosen settings are per-level step limits and default decoding hyperparameters. The two evaluation axes are paper-defined constructs without external falsifiable handles.

free parameters (2)
  • Max interaction steps per automation level = 15 (L1), 15 (L2), 30 (L3), 50 (L4)
    Chosen by hand and described as 'sufficient to complete most tasks' (Appendix D.1). Different limits would change success rates, especially for long-horizon L4 tasks.
  • Decoding hyperparameters = temperature=1.0, top_p=0.9, max_tokens=1500
    Default sampling settings (Appendix D.1); the high temperature increases variance and inflates the need for repeated runs, which the paper does not provide.
assumptions (5)
  • domain assumption OSWorld provides a valid executable proxy for real desktop computer use, and its state-based evaluation is reliable.
    OS-MAP inherits the OSWorld VM infrastructure and evaluation tools (Section 2, Section 3.4). If the VM's fixed Ubuntu desktop diverges from real user environments, the benchmark results may not transfer.
  • domain assumption The five automation levels (L1-L5) correspond to distinct, meaningful capability requirements.
    The taxonomy is inspired by SAE driving automation (Section 3.1), but no independent evidence shows the levels separate agent capabilities beyond task length or instruction detail.
  • domain assumption The demand hierarchy derived from mobile usage statistics and OECD data, adapted to desktop, reflects real desktop user demands.
    Section 3.2 adapts State of Mobile 2025 data to desktop by excluding mobile-only categories and adding office work; this adaptation is not validated against desktop usage logs.
  • domain assumption Excluding tasks that require real accounts or network-side effects does not materially reduce representativeness.
    Appendix C.3 removes tasks tied to user accounts and external effects for reproducibility; this may make the benchmark more synthetic and less aligned with the paper's real-world claims.
  • domain assumption The reported human performance (71.9% overall) was measured on the same tasks under comparable conditions.
    Table 3 lists human success rates, but the protocol (participants, instructions, action space, number of tasks) is not described in the main text or the included appendices.
invented entities (2)
  • Automation-level taxonomy (L1-L5)
    purpose: Categorize benchmark tasks by the degree of agent autonomy, from reactive execution to proactive assistance.
    A paper-defined construct with no falsifiable handle outside the paper; the levels inform the task labels and the interpretation of results, but are not independently measurable.
  • Generalization scope (S1-S3)
    purpose: Categorize tasks by breadth across user-demand domains: narrow, domain-level, and general.
    Another paper-defined construct derived from an adapted mobile-usage hierarchy; no external validation shows the scopes correspond to real capability transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?." pith.science (2026). https://pith.science/paper/7GQ7QF62

@misc{pith2026250719132,
  author       = {Pith},
  title        = {Pith review of: OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7GQ7QF62}},
  note         = {Machine review of arXiv:2507.19132}
}
read the original abstract

Computer-using agents have shown strong potential to boost human productivity and enable new application forms across platforms. While recent advances have led to usable applications, existing benchmarks fail to account for the internal task heterogeneity and the corresponding agent capabilities, as well as their alignment with actual user demands-hindering both targeted capability development and the reliable transition of research progress into practical deployment. To bridge the gap, we present OS-MAP, a benchmark for daily computer-using automation that organizes its 416 realistic tasks across 15 applications along two key dimensions: a five-level taxonomy of automation and a generalization scope derived from a real-world user demand hierarchy. To enable fine-grained analysis of required capabilities and alignment with real-world scenarios, OS-MAP evaluates agents along two dimensions: automation level across a five-level taxonomy, and generalization scope across a demand hierarchy. This design captures varying levels of required agent autonomy and generalization, forming a performance-generalization evaluation matrix for structured and comprehensive assessment. Experiments show that even State-of-the-Art agents with VLM backbones struggle with higher-level tasks involving perception, reasoning, and coordination-highlighting the need for a deeper understanding of current strengths and limitations to drive the future progress in computer-using agents research and deployment. All code, environments, baselines, and data are publicly available at https://github.com/OS-Copilot/OS-Map.

Figures

Figures reproduced from arXiv: 2507.19132 by the authors.

Figure 1
Figure 1. OS-MAP qualitative evaluation matrix, summarizing how different types of agents perform across two dimensions. General-purpose models show strong generalization, while scenario experts excel at specific tasks. Mainstream computer-using agents aim to balance both, yet still face major challenges. Agent positioning is based on reported performance, as detailed in Appendix A. emergence of new methods, the open-ended se… view at source ↗
Figure 2
Figure 2. OS-MAP is built on an executable desktop environment designed for daily computer tasks, integrating a suite of applications and tools. It provides the infrastructure for reliable evaluation by handling task initialization and success verification. Agents interact autonomously via GUI operations, guided by instructions and screenshot perception. fine-grained control, consistent reproducibility, flexible extensibility… view at source ↗
Figure 3
Figure 3. Automation levels demonstration on a specific task: rotating wallpapers daily. From the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Task distribution on the demand hier￾archy in OS-MAP benchmark [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Agent prefers enter￾ing a URL instead of navigat￾ing websites [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 8
Figure 8. Figure 8: Agent is deleting all history, not just those from YouTube [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 12
Figure 12. Figure 12: Detailed Specification of the task goal in ToDo (not informed in the task instruction). [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: illustrates a severe hallucination, where the agent mistakenly identifies the current webpage as a Chrome browser interface and treats the top search bar as a search engine input. Figures 14 and 15 show two L3 tasks in which the agent fails to adapt when the straightf…
Figure 15
Figure 15. Figure 15: (L3) Task instruction: Enter theater mode and resize the scale to 48. However, the resize button is hidden in theater mode, and the agent does not know it should swap the execution order [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: (L3) Task instruction: Locate the MOST geographically central station in Paris on this map and jump to its location on Google Maps. The agent simply ignores the current page (weather station map) and searches for subway stations in the center of Paris on Google Maps. …
Figure 17
Figure 17. Figure 17: (L4) Task instruction: I am writing my course paper and I need to cite a reference. I remember it in Zotero, but I can’t remember which one it is. Please help me find this article and imitate the two IEEE formats above to com￾plete the citation. Note that the font for…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    OSWorld 2.0 is a benchmark of 108 realistic long-horizon computer-use tasks where current agents achieve only 20.6% binary completion, struggling with state inference and constraint tracking.

  2. OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

    cs.AI 2026-07 accept novelty 6.0 of 10

    VLM judges of CUA trajectories are systematically lenient; OSReward measures this with human gold, and OS-Shepherd open models close most of the reliability gap cheaply.

  3. OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows

    cs.AI 2025-10 conditional novelty 6.0 of 10

    A hybrid rule-plus-VLM detector for mobile GUI agents, tested on a new 204-trajectory Android benchmark, reports 10-30% gains over baselines.

Reference graph

Works this paper leans on

120 extracted references · 16 canonical work pages · cited by 3 Pith papers

  1. [1]

    Os agents: A survey on mllm-based agents for general computing devices use, 2024

    Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, et al. Os agents: A survey on mllm-based agents for general computing devices use, 2024. 10

    2024

  2. [2]

    The dawn of gui agent: A preliminary case study with claude 3.5 computer use

    Siyuan Hu, Mingyu Ouyang, Difei Gao, and Mike Zheng Shou. The dawn of gui agent: A preliminary case study with claude 3.5 computer use. arXiv preprint arXiv:2411.10323, 2024

    arXiv 2024

  3. [3]

    Introducing operator

    OpenAI. Introducing operator. https://openai.com/index/introducing-operator/,

  4. [4]

    Claude 3.5 sonnet

    Anthropic. Claude 3.5 sonnet. https://www.anthropic.com/news/ 3-5-models-and-computer-use , 2024. Accessed: 2025-04-25

    2024

  5. [5]

    Ufo2: The desktop agentos

    Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, et al. Ufo2: The desktop agentos. arXiv preprint arXiv:2504.14603, 2025

    arXiv 2025

  6. [6]

    Ui-tars: Pioneering automated gui interaction with native agents

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025

    arXiv 2025

  7. [7]

    Os-copilot: Towards generalist computer agents with self-improvement

    Zhiyong Wu, Chengcheng Han, Zichen Ding, Zhenmin Weng, Zhoumianze Liu, Shunyu Yao, Tao Yu, and Lingpeng Kong. Os-copilot: Towards generalist computer agents with self-improvement. arXiv preprint arXiv:2402.07456, 2024

    arXiv 2024

  8. [8]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

Show all 120 references
  1. [9]

    Autoglm: Autonomous foundation agents for guis

    Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, et al. Autoglm: Autonomous foundation agents for guis. arXiv preprint arXiv:2411.00820, 2024

    2024 arXiv

  2. [10]

    Aguvis: Unified pure vision agents for autonomous gui interaction

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454, 2024

    2024 arXiv

  3. [11]

    Os-atlas: A foundation action model for generalist gui agents

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024

    arXiv

  4. [12]

    Ap- pagentx: Evolving gui agents as proficient smartphone users

    Wenjia Jiang, Yangyang Zhuang, Chenxi Song, Xu Yang, Joey Tianyi Zhou, and Chi Zhang. Ap- pagentx: Evolving gui agents as proficient smartphone users. arXiv preprint arXiv:2503.02268, 2025

    2025 arXiv

  5. [13]

    Agent s: An open agentic framework that uses computers like a human.arXiv preprint arXiv:2410.08164, 2024

    Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like a human.arXiv preprint arXiv:2410.08164, 2024

    2024 arXiv

  6. [14]

    Agent s2: A compositional generalist-specialist framework for computer use agents

    Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s2: A compositional generalist-specialist framework for computer use agents. arXiv preprint arXiv:2504.00906, 2025

    2025 arXiv

  7. [15]

    Agentstore: Scalable integration of heterogeneous agents as specialized generalist computer assistant

    Chengyou Jia, Minnan Luo, Zhuohang Dang, Qiushi Sun, Fangzhi Xu, Junlin Hu, Tianbao Xie, and Zhiyong Wu. Agentstore: Scalable integration of heterogeneous agents as specialized generalist computer assistant. arXiv preprint arXiv:2410.18603, 2024

    arXiv

  8. [16]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114, 2023

    2023

  9. [17]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023

    2023 arXiv

  10. [18]

    Spider2-v: How far are multi- modal agents from automating data science and engineering workflows? Advances in Neural Information Processing Systems, 37:107703–107744, 2024

    Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Wenjing Hu, Yuchen Mao, et al. Spider2-v: How far are multi- modal agents from automating data science and engineering workflows? Advances in Neural Information Proces...

    2024

  11. [19]

    Androidworld: A dynamic benchmarking environment for autonomous agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573, 2024

    2024 arXiv

  12. [20]

    Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.07718, 2024

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.07718, 2024

    2024 arXiv

  13. [21]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing S...

    2024

  14. [22]

    Windows agent arena: Evaluating multi-modal os agents at scale

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, et al. Windows agent arena: Evaluating multi-modal os agents at scale. arXiv preprint arXiv:2409.08264, 2024

    2024 arXiv

  15. [23]

    On the effects of data scale on ui control agents

    Wei Li, William E Bishop, Alice Li, Christopher Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on ui control agents. Advances in Neural Information Processing Systems, 37:92130–92154, 2024

    2024

  16. [24]

    Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles, April 2021

    On-Road Automated Driving (ORAD) Committee. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles, April 2021. URL https: //doi.org/10.4271/J3016_202104

    2021 doi

  17. [25]

    Seeclick: Harnessing gui grounding for advanced visual gui agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024

    2024 arXiv

  18. [26]

    Navigating the digital world as humans do: Universal visual grounding for gui agents

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024

    2024 arXiv

  19. [27]

    Proactive agent: Shifting llm agents from reactive responses to active assistance

    Yaxi Lu, Shenzhi Yang, Cheng Qian, Guirong Chen, Qinyu Luo, Yesai Wu, Huadong Wang, Xin Cong, Zhong Zhang, Yankai Lin, et al. Proactive agent: Shifting llm agents from reactive responses to active assistance. arXiv preprint arXiv:2410.12361, 2024

    arXiv

  20. [28]

    Need help? designing proactive ai assistants for programming

    Valerie Chen, Alan Zhu, Sebastian Zhao, Hussein Mozannar, David Sontag, and Ameet Tal- walkar. Need help? designing proactive ai assistants for programming. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–18, 2025

    2025

  21. [29]

    How should my chatbot interact? a survey on social characteristics in human–chatbot interaction design

    Ana Paula Chaves and Marco Aurelio Gerosa. How should my chatbot interact? a survey on social characteristics in human–chatbot interaction design. International Journal of Human– Computer Interaction, 37(8):729–758, 2021

    2021

  22. [30]

    Proactive conversational agents in the post-chatgpt world

    Lizi Liao, Grace Hui Yang, and Chirag Shah. Proactive conversational agents in the post-chatgpt world. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3452–3455, 2023

    2023

  23. [31]

    A3: Android agent arena for mobile gui agents

    Yuxiang Chai, Hanhao Li, Jiayu Zhang, Liang Liu, Guangyi Liu, Guozhi Wang, Shuai Ren, Siyuan Huang, and Hongsheng Li. A3: Android agent arena for mobile gui agents. arXiv preprint arXiv:2501.01149, 2025

    2025

  24. [32]

    Reinforcement learning on web interfaces using workflow-guided exploration.arXiv preprint arXiv:1802.08802, 2018

    Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration.arXiv preprint arXiv:1802.08802, 2018

    2018 arXiv

  25. [33]

    State of mobile 2025: The industry’s leading report

    Sensor Tower. State of mobile 2025: The industry’s leading report. https://sensortower. com/state-of-mobile-2025 , 2025. Accessed: 2025-04-25

    2025

  26. [34]

    Ict access and usage database

    OECD. Ict access and usage database. https://oe.cd/dx/ict-access-usage , 2025. Accessed: 2025-04-25. 12

    2025

  27. [35]

    Position: Levels of agi for operationalizing progress on the path to agi

    Meredith Ringel Morris, Jascha Sohl-Dickstein, Noah Fiedel, Tris Warkentin, Allan Dafoe, Aleksandra Faust, Clement Farabet, and Shane Legg. Position: Levels of agi for operationalizing progress on the path to agi. In Forty-first International Conference on Machine Learning, 2024

    2024

  28. [36]

    Towards building specialized generalist ai with system 1 and system 2 fusion

    Kaiyan Zhang, Biqing Qi, and Bowen Zhou. Towards building specialized generalist ai with system 1 and system 2 fusion. arXiv preprint arXiv:2407.08642, 2024

    2024 arXiv

  29. [37]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023

    2023

  30. [38]

    Webvoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024

    2024 arXiv

  31. [39]

    Mapping natural language instructions to mobile ui action sequences

    Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776, 2020

    2005 arXiv

  32. [40]

    Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web

    Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem AlShikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web. In European Conference on Computer Vision, pages 161...

    2024

  33. [41]

    Webshop: Towards scalable real-world web interaction with grounded language agents

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757, 2022

    2022

  34. [42]

    Understanding the weakness of large language model agents within a complex android environment

    Mingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao. Understanding the weakness of large language model agents within a complex android environment. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6061–6072, 2024

    2024

  35. [43]

    Theagentcompany: benchmarking llm agents on consequential real world tasks

    Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking llm agents on consequential real world tasks. arXiv preprint arXiv:2412.14161, 2024

    2024 arXiv

  36. [44]

    Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows

    Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, et al. Scienceboard: Evaluating multimodal autonomous agents in realistic scientific workflows. arXiv preprint arXiv:2505.19897, 2025

    2025 arXiv

  37. [45]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  38. [46]

    Claude 3.7 sonnet

    Anthropic. Claude 3.7 sonnet. https://www.anthropic.com/news/claude-3-7-sonnet ,

  39. [47]

    Introducing gemini 2.0: our new ai model for the agentic era

    Gemini Team. Introducing gemini 2.0: our new ai model for the agentic era. https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/ , 2025. Accessed: 2025-04-25

    2025

  40. [48]

    Accessed: 2025-04-25

    2025

  41. [49]

    Gui-actor: Coordinate-free visual grounding for gui agents, 2025

    Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, Si Qin, Lars Liden, Qingwei Lin, Huan Zhang, Tong Zhang, Jianbing Zhang, Dongmei Zhang, and Jianfeng Gao. Gui-actor: Coordinate-free visual grounding for...

    2025 arXiv

  42. [50]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  43. [51]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649, 2024

    2024 arXiv

  44. [52]

    A survey of neural code intelligence: Paradigms, advances and beyond

    Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint arXiv:2403.14734, 2024. 13

    2024 arXiv

  45. [53]

    An- droidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36:59708–59728, 2023

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36:59708–59728, 2023

    2023

  46. [54]

    Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05955, 2024

    Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05955, 2024

    2024 arXiv

  47. [55]

    Gui-world: A dataset for gui-oriented multimodal llm-based agents

    Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, et al. Gui-world: A dataset for gui-oriented multimodal llm-based agents. arXiv e-prints, pages arXiv–2406, 2024

    2024

  48. [56]

    Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451, 2024

    2024 arXiv

  49. [57]

    Screenspot-pro: Gui grounding for professional high-resolution computer use

    Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.07981, 2025

    2025 arXiv

  50. [58]

    Vision- language models can self-improve reasoning via reflection

    Kanzhi Cheng, Li YanTao, Fangzhi Xu, Jianbing Zhang, Hao Zhou, and Yang Liu. Vision- language models can self-improve reasoning via reflection. In Proceedings of the 2025 Confer- ence of the Nations of the Americas Chapter of the Association for Computational Linguistics: Huma...

    2025 doi

  51. [59]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14281–1...

    2024

  52. [60]

    Ui- vision: A desktop-centric gui benchmark for visual perception and interaction

    Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M Tamer Özsu, Aishwarya Agrawal, David Vazquez, et al. Ui- vision: A desktop-centric gui benchmark for visual perception and interaction. arXiv preprint arXiv:2503.1...

    2025 arXiv

  53. [61]

    Aria-ui: Visual grounding for gui instructions

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256, 2024

    2024 arXiv

  54. [62]

    Ferret-ui: Grounded mobile ui understanding with multimodal llms

    Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms. In European Conference on Computer Vision, pages 240–255. Springer, 2024

    2024

  55. [63]

    Iris: Breaking gui complexity with adaptive focus and self-refining

    Zhiqi Ge, Juncheng Li, Xinglei Pang, Minghe Gao, Kaihang Pan, Wang Lin, Hao Fei, Wenqiao Zhang, Siliang Tang, and Yueting Zhuang. Iris: Breaking gui complexity with adaptive focus and self-refining. arXiv preprint arXiv:2412.10342, 2024

    2024 arXiv

  56. [64]

    Ferret-ui 2: Mastering universal user interface understanding across platforms

    Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui 2: Mastering universal user interface understanding across platforms. arXiv preprint arXiv:2410.18967, 2024

    arXiv

  57. [65]

    Showui: One vision-language-action model for generalist gui agent

    Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-action model for generalist gui agent. In NeurIPS 2024 Workshop on Open-World Agents, 2024

    2024

  58. [66]

    Ui-hawk: Unleashing the screen stream understanding for gui agents

    Jiwen Zhang, Yaqi Yu, Minghui Liao, Wentao Li, Jihao Wu, and Zhongyu Wei. Ui-hawk: Unleashing the screen stream understanding for gui agents. Preprints, manuscript/202408.2137, 2024. 14

    2024

  59. [67]

    Mobilevlm: A vision-language model for better intra-and inter-ui understanding

    Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, and Shuo Shang. Mobilevlm: A vision-language model for better intra-and inter-ui understanding. arXiv preprint arXiv:2409.14818, 2024

    2024 arXiv

  60. [68]

    Edge: Enhanced grounded gui understanding with enriched multi-granularity synthetic data

    Xuetian Chen, Hangcheng Li, Jiaqing Liang, Sihang Jiang, and Deqing Yang. Edge: Enhanced grounded gui understanding with enriched multi-granularity synthetic data. arXiv preprint arXiv:2410.19461, 2024

    2024 arXiv

  61. [69]

    Mobileviews: A large-scale mobile gui dataset

    Longxi Gao, Li Zhang, Shihe Wang, Shangguang Wang, Yuanchun Li, and Mengwei Xu. Mobileviews: A large-scale mobile gui dataset. arXiv preprint arXiv:2409.14337, 2024

    2024

  62. [70]

    Omniparser for pure vision based gui agent

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.00203, 2024

    2024 arXiv

  63. [71]

    Android in the zoo: Chain-of-action-thought for gui agents

    Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thought for gui agents. arXiv preprint arXiv:2403.02713, 2024

    2024 arXiv

  64. [72]

    Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale

    Tianyue Ou, Frank F Xu, Aman Madaan, Jiarui Liu, Robert Lo, Abishek Sridhar, Sudipta Sengupta, Dan Roth, Graham Neubig, and Shuyan Zhou. Synatra: Turning indirect knowledge into direct demonstrations for digital agents at scale. arXiv preprint arXiv:2409.15637, 2024

    2024 arXiv

  65. [73]

    Learn-by- interact: A data-centric framework for self-adaptive agents in realistic environments

    Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö Arık. Learn-by- interact: A data-centric framework for self-adaptive agents in realistic environments. arXiv preprint arXiv:2501.10893, 2025

    2025 arXiv

  66. [74]

    Os-genesis: Automating gui agent trajectory construction via reverse task synthesis

    Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, et al. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412.19723, 2024

    2024 arXiv

  67. [75]

    Agile: A novel reinforcement learning framework of llm agents

    Peiyuan Feng, Yichen He, Guanhua Huang, Yuan Lin, Hanchong Zhang, Yuchen Zhang, and Hang Li. Agile: A novel reinforcement learning framework of llm agents. arXiv preprint arXiv:2405.14751, 2024

    2024 arXiv

  68. [76]

    Gui-bee: Align gui action grounding to novel environments via autonomous exploration

    Yue Fan, Handong Zhao, Ruiyi Zhang, Yu Shen, Xin Eric Wang, and Gang Wu. Gui-bee: Align gui action grounding to novel environments via autonomous exploration. arXiv preprint arXiv:2501.13896, 2025

    2025 arXiv

  69. [77]

    Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning

    Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. arXiv preprint arXiv:2411.02337, 2024

    2024 arXiv

  70. [78]

    Autowebglm: A large language model- based web navigating agent

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: A large language model- based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowl- edge Discovery and D...

    2024

  71. [79]

    Gui-r1: A generalist r1-style vision-language action model for gui agents

    Xiaobo Xia and Run Luo. Gui-r1: A generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458, 2025

    2025 arXiv

  72. [80]

    Ui-r1: Enhancing action prediction of gui agents by reinforcement learning

    Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Guanjing Xiong, and Hongsheng Li. Ui-r1: Enhancing action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.21620, 2025

    2025 arXiv

  73. [81]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. 15

    2023

  74. [82]

    Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners

    Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners. arXiv preprint arXiv:2504.14239, 2025

    2025 arXiv

  75. [83]

    Is your llm secretly a world model of the internet? model-based planning for web agents

    Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, et al. Is your llm secretly a world model of the internet? model-based planning for web agents. arXiv preprint arXiv:2411.06559, 2024

    2024 arXiv

  76. [84]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    2023

  77. [85]

    Gpt-4v (ision) is a generalist web agent, if grounded

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024

    2024 arXiv

  78. [86]

    Agent-e: From autonomous web navigation to foundational design principles in agentic systems

    Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. arXiv preprint arXiv:2407.13032, 2024

    2024 arXiv

  79. [87]

    Corex: Pushing the boundaries of complex reasoning through multi-model collaboration.arXiv preprint arXiv:2310.00280, 2023

    Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, and Lingpeng Kong. Corex: Pushing the boundaries of complex reasoning through multi-model collaboration.arXiv preprint arXiv:2310.00280, 2023

    2023 arXiv

  80. [88]

    Web agents with world models: Learning and leveraging environment dynamics in web navigation

    Hyungjoo Chae, Namyoung Kim, Kai Tzu-iunn Ong, Minju Gwak, Gwanwoo Song, Jihoon Kim, Sunghwan Kim, Dongha Lee, and Jinyoung Yeo. Web agents with world models: Learning and leveraging environment dynamics in web navigation. arXiv preprint arXiv:2410.13232, 2024

    arXiv

  81. [89]

    Mobile-agent-e: Self-evolving mobile assistant for complex tasks

    Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. Mobile-agent-e: Self-evolving mobile assistant for complex tasks. arXiv preprint arXiv:2501.11733, 2025

    2025 arXiv

  82. [90]

    Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration

    Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv preprint arXiv:2406.01014, 2024

    2024 arXiv

  83. [91]

    Human-automation interaction

    Thomas B Sheridan and Raja Parasuraman. Human-automation interaction. Reviews of human factors and ergonomics, 1(1):89–129, 2005

    2005

  84. [92]

    Learnact: Few-shot mobile gui agent with a unified demonstration benchmark

    Guangyi Liu, Pengxiang Zhao, Liang Liu, Zhiming Chen, Yuxiang Chai, Shuai Ren, Hao Wang, Shibo He, and Wenchao Meng. Learnact: Few-shot mobile gui agent with a unified demonstration benchmark. arXiv preprint arXiv:2504.13805, 2025

    2025 arXiv

  85. [93]

    Artificial general intelligence: concept, state of the art, and future prospects

    Ben Goertzel. Artificial general intelligence: concept, state of the art, and future prospects. Journal of Artificial General Intelligence, 5(1):1, 2014

    2014

  86. [94]

    A model for types and levels of human interaction with automation

    Raja Parasuraman, Thomas B Sheridan, and Christopher D Wickens. A model for types and levels of human interaction with automation. IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans, 30(3):286–297, 2000

    2000

  87. [95]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023. 16 A Qualitative Evaluation Matrix This section explains the qualitative criteria used t...

    2023 arXiv

  88. [96]

    Personal llm agents: Insights and survey about the capability, efficiency and security

    Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459, 2024

    2024 arXiv

  89. [98]

    To ensure reproducibility, each task begins by loading a designated snapshot

    Initialization. To ensure reproducibility, each task begins by loading a designated snapshot. Afterward, a predefined initialization script is executed. Snapshots and initialization scripts are designed to work in tandem, offering both high flexibility and low initialization overhead

  90. [99]

    Once initialized, the system enters the execution loop

    Task execution. Once initialized, the system enters the execution loop. At each step, the VMC captures the current observation and passes it to the agent. Based on the current state and interaction history, the agent outputs a textual action. This action is parsed and executed...

  91. [100]

    For certain tasks, the final state after agent execution is not directly extractable

    Post-execution configuration (optional). For certain tasks, the final state after agent execution is not directly extractable. In such cases, additional actions are required to bring the system into a verifiable state. For example, after adding an item to the cart on the Decat...

  92. [101]

    The VMC includes a set of state extraction functions designed to retrieve relevant information from the VM

    State extraction. The VMC includes a set of state extraction functions designed to retrieve relevant information from the VM. These serve as input for the next evaluation step

  93. [102]

    Evaluation functions are task-specific and compare the extracted state against expected conditions

    Evaluation. Evaluation functions are task-specific and compare the extracted state against expected conditions. Depending on the nature of the state, corresponding comparison logic is applied—such as string matching, file equivalence, or key–value comparison. B.2 Initializatio...

    1920

  94. [103]

    We begin by identifying underrepresented scenes within the demand hierarchy

    Task selection. We begin by identifying underrepresented scenes within the demand hierarchy. For each selected scene, we determine a representative application and outline a task concept aligned with that context

  95. [104]

    Annotators study the target app or website using official documen- tation, demos, and hands-on interaction

    Exploration & specification. Annotators study the target app or website using official documen- tation, demos, and hands-on interaction. They then define a concrete task objective, assign an appropriate difficulty level, and manually execute the task flow to verify feasibility...

  96. [105]

    Annotators craft clear and concise task instructions and executable initialization configurations

    Instruction & configuration. Annotators craft clear and concise task instructions and executable initialization configurations. Together, they control task difficulty—higher-level tasks omit de- tails or include (human-recognizable) misleading cues, requiring agents to activel...

  97. [106]

    The annotators manually complete the task to record a standard success state for the following evaluation process

    Reference state preparation. The annotators manually complete the task to record a standard success state for the following evaluation process

  98. [107]

    Evaluation involves comparing VM file or system states against predefined targets

    Evaluation setup. Evaluation involves comparing VM file or system states against predefined targets. Some tasks also require post-execution scripts or logic (postconfig) to expose the key status for assessment

  99. [108]

    Cross-validation. Each task undergoes a rigorous review by two other annotators across several dimensions before inclusion: (1) task authenticity and representativeness, (2) clarity and unam- biguity of instructions, (3) reproducibility, (4) correctness and (5) robustness of e...

  100. [109]

    Task selection. Upon reviewing the current task set, we found a gap in L4-level tasks within the office productivity domain—particularly tasks involving tool use, to-do management, and email communication. We thus defined a task prototype: write a to-do item that instructs the...

  101. [110]

    We selected the How’s Life reports from the official OECD website as the document source

    Exploration & specification. We selected the How’s Life reports from the official OECD website as the document source. A to-do entry was added in a ToDo application, with a detailed task description specifying file names, save locations, and expected actions (see Figure 12)

  102. [111]

    Through reverse engineering of the ToDo application, we identified the configuration file’s location and edit protocol

    Instruction & configuration. Through reverse engineering of the ToDo application, we identified the configuration file’s location and edit protocol. Based on this, we created a config file and imported it during task initialization, so the to-do item loads automatically. Simil...

  103. [112]

    We manually completed the task to obtain a reference success state—defined as the appearance of a new email in the recipient’s local mail server directory

    Reference state preparation. We manually completed the task to obtain a reference success state—defined as the appearance of a new email in the recipient’s local mail server directory. During testing, we observed long download times and limits on translation input and attachme...

  104. [113]

    The evaluation checks for textual equality between the expected and actual email file and is provided as part of the task package

    Evaluation setup. The evaluation checks for textual equality between the expected and actual email file and is provided as part of the task package

  105. [114]

    Could you please convert a PowerPoint presentation to video and play it with VLC?

    Cross-validation. The task was tested by two additional annotators to validate both procedure correctness and robustness—i.e., whether the task would still pass evaluation despite minor execution variations or small errors. 20 C.3 Filtering of Tasks from OSWorld We reused and ...

    2024

  106. [115]

    ,→ ,→ ,→

    To reiterate, regardless of whether you include reasoning, your output MUST contain an action in the SPECIFIED FORMAT (a dictionary enclosed in triple backticks as shown in the examples below), and it must include a valid `action_type` and parameters as defined above. ,→ ,→ ,→

  107. [116]

    action_type

    For `MOUSE_MOVE`, you must specify the exact target `x` and `y` coordinates. The screen bounds are `X_MAX = 1920`, `Y_MAX = 1080`. The coordinates must fall within [0, 1920] and [0, 1080]. Example: ,→ ,→ ``` { "action_type": "MOUSE_MOVE", "x": 1319, "y": 65 } ```

    1920

  108. [117]

    action_type

    For `[CLICK, RIGHT_CLICK, DOUBLE_CLICK, DRAG_TO]`, specifying `x` and `y` is optional. If omitted, the action defaults to the current cursor position (often used after `MOUSE_MOVE`). However, it is RECOMMENDED to specify the coordinates explicitly. Same format as `MOUSE_MOVE`:...

  109. [118]

    Use them when you think it's necessary to wait, when the task has failed, or when it has succeeded

    Other special actions are `[WAIT, FAIL, DONE]`. Use them when you think it's necessary to wait, when the task has failed, or when it has succeeded. Each `WAIT` pauses for ~2 seconds. Do not declare `FAIL` lightly without attempting reasonable actions and explorations, but if y...

  110. [119]

    If there is no clear description of the file name, save location, etc., use the default

    If the task is file editing, make sure it is saved successfully. If there is no clear description of the file name, save location, etc., use the default. ,→ ,→

  111. [120]

    Otherwise, you can do whatever you want as long as the task is completed.,→

    If there are clear step-level instructions, please follow them strictly. Otherwise, you can do whatever you want as long as the task is completed.,→

  112. [121]

    password

    My computer password is `"password"`. You may use it freely whenever `sudo` access is required.,→ Please think step by step. Carefully observe the current screenshot and then output your reasoning (optional), your plan, the current action and expected results, and most importa...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.