Pith. sign in

REVIEW 4 major objections 4 minor 6 cited by

MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MCPWorld proposes that computer-use agents be evaluated by hooking internal application behavior, and reports that a hybrid GUI-plus-API agent reaches 75.12% task success across 201 desktop tasks.

desk verdict MCPWorld is a genuinely useful white-box benchmark for GUI/API/hybrid agents; the headline modality comparison is real but confounded by BASH availability, so read the empirical claims with caution. read the letter →

arxiv 2506.07672 v1 pith:U7WVJYRV submitted 2025-06-09 cs.AI

Yunhe Yan , Shihe Wang , Jiajun Du , Yexuan Yang , Yuxuan Shan , Qichen Qiu , Xianqing Jia , Xinge Wang
show 7 more authors
Xin Yuan Xu Han Mao Qin Yinxiao Chen Chen Peng Shangguang Wang Mengwei Xu
This is my paper · ORCID
classification cs.AI
keywords computeruseagentsbenchmarkingtestbedModelContextProtocol(MCP)white-boxevaluationGUIautomationAPIhybriddesktopapplications
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MCPWorld sets out to make computer-use agents comparable across every way they can interact with a computer: through a graphical interface, through application APIs, or through both. Its central proposal is to evaluate agents with 'white-box apps'—open-source applications that can be recompiled and instrumented so the evaluator verifies task completion by hooking internal functions and events instead of matching screenshots, UI trees, or output files. The paper argues this decouples evaluation from the agent's implementation and from UI state, allowing one testbed to grade GUI-only, MCP-only, and hybrid agents on the same 201 tasks. A representative LLM-powered agent reaches 75.12% task success in hybrid mode, above GUI-only (70.65%) and MCP-only (53.23%), and the authors read this as initial evidence that combining MCP APIs with GUI fallbacks helps agents on complex desktop tasks. If the paradigm is sound, agent benchmarks can grade what the application actually did rather than what the screen looked like.

What carries the argument

The load-bearing mechanism is the white-box application and its verification hooks. Each of the ten applications is open source, so MCPWorld can recompile it, attach MCP servers, and place evaluation logic at semantically meaningful internal events—function calls, event emissions, or state transitions—using dynamic instrumentation, targeted code injection, or API-driven state queries. When the agent acts, the hooks fire at the exact moment the action takes effect inside the app, letting the evaluator verify both intermediate key steps and final completion without querying the screen, matching a trajectory, or waiting for a file to be written. Around this, a unified tool space presents the agent with GUI tools (screenshots, mouse and keyboard) and MCP tools (application-specific functions) so the same task can be attempted by GUI, API, or hybrid policies and judged by the same internal signals.

What would settle it

Take any MCPWorld task, complete it by directly editing the underlying file or database that the task is meant to change, and see whether the hook-based evaluator marks it successful; if it does not, the hooks have missed a legitimate completion path. A second check would have human judges score a random sample of recorded runs and compare their verdicts with the hook-based scores, looking for systematic disagreement on tasks without a unique hooked path.

Watch

Extended reading notes

Core claim

The paper's central claim is that computer-use agents are best evaluated from inside the application, not from outside it. It proposes that task completion be verified by white-box hooks that observe internal function invocations, events, and state transitions, which it argues are more accurate, more robust to UI changes, and more fair across interaction modes than screenshot matching, UI-tree matching, or output-file matching. To demonstrate, it builds MCPWorld, a containerized desktop testbed with 201 tasks across 10 open-source applications, where each task has annotated key steps and hook-based success criteria. Running a representative LLM-powered agent on all 201 tasks, the paper reports 75.12% task success for a hybrid agent with both GUI and MCP tools, 70.65% for GUI-only, and 53.23% for MCP-only, with the MCP-only shortfall attributed mainly to limited MCP coverage and tool descriptions rather than to planning ability; on hard tasks, the hybrid's smaller drop in success is taken as evidence that API access adds robustness.

Load-bearing premise

The whole benchmark assumes that the hooks placed in the application cover every legitimate way to finish a task, yet the paper's own appendix shows agents can bypass annotated key steps through shell commands while still succeeding.

Editorial extensions

If this is right

  • Evaluator scores become robust to cosmetic UI changes, because success is determined by internal state transitions rather than pixels, accessibility trees, or output files.
  • The same app instance can fairly compare GUI-only, API-only, and hybrid agents, and the paper's three-way success rates demonstrate that comparison on a shared testbed.
  • MCP access helps most on complex tasks: hybrid success falls 39.3 points from easy to hard, versus 54.9 for GUI-only, so API tools provide a fallback that preserves partial progress.
  • Tasks whose completion exists only in memory, such as a breakpoint hit with its call stack, become verifiable even when nothing is ever written to disk.
  • The config-driven design means the suite can grow: adding a new app or task requires registering verification logic and state snapshots rather than building a new evaluator from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same hook-based verification idea could move from benchmarks into production settings, where an application's internal events could serve as ground truth for whether an agent's action actually took effect.
  • Editorial inference: because the MCP-only score depends on the maturity of community-written MCP servers, the 53.23% figure measures today's MCP ecosystem as much as agent capability; a better server for one application could shift the comparison.
  • Editorial inference: the paper's observation that hybrid success dips on medium tasks suggests tool-call prompt length and choice overhead can offset API benefits, a variable future benchmark designers could control explicitly by compressing tool descriptions.
  • Editorial inference: hook placement is per-task annotation work, so scaling to more apps will depend on whether LLM-assisted hook discovery, which the paper reports using, can be automated without losing verification fidelity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. MCPWorld proposes a benchmark testbed for computer-use agents (CUAs) that interact with desktop applications through GUI, API (MCP), or hybrid interfaces. The framework uses ten open-source desktop applications, 201 curated tasks, and a white-box evaluation paradigm that verifies task completion by instrumenting application internals rather than by matching screenshots or output files. The paper reports preliminary experiments with a Claude 3.7 Sonnet-based agent: a hybrid configuration reaches 75.12% task success, compared with 70.65% for GUI-only and 53.23% for MCP-only, and an appendix ablates the effect of removing the BashTool. The central empirical claim is that adding MCP tools enhances CUA capability, with the hybrid configuration outperforming both unimodal baselines.

Significance. If the benchmark is taken as a resource rather than as a clean modality comparison, it is a potentially valuable contribution: it is fully containerized, uses open-source applications, integrates existing MCP servers, and its dynamic instrumentation / code-injection / API-query evaluation is a genuinely different verification approach from UI matching or output-file comparison. Task curation appears careful, with two cross-validations per task, DuckTrack human traces, and detailed per-task verification logic. However, the paper's headline empirical comparison is confounded by the availability of the BashTool, and the reported differences lack uncertainty quantification. The benchmark infrastructure is therefore useful and worth publishing after revision, but the current empirical evidence for the claim that MCP enhances CUA capability is not convincing as reported.

major comments (4)
  1. [Section 4.1 and Appendices C/D] The three-way modality comparison is confounded by BashTool availability. Section 4.1 says that in the MCP-Only setup 'the agent could only use these MCP tools, forcing reliance on the programmatic interface,' but Appendix C states that 'BashTool and the EditTool might still be available depending on the specific experimental variant,' and Appendix D reports that the main-table numbers are the Bash-enabled results. Since GUI-Only also included BashTool, the headline comparison measures 'Bash plus modality' rather than isolated GUI, API, or hybrid interaction. Appendix D shows that when BashTool is disabled, the success rates become GUI-only 67.00%, MCP-only 43.00%, and Hybrid 65.50%: the hybrid advantage over GUI-only reverses, and MCP-only drops substantially. Consequently, the claims that 'enabling MCP enhances the capabilities of CUA' and the failure attributions in Table 5 are not supported by the reported experiments. The authors should either make the no-BASH comparison primary or fully cross BashTool availability with each modality, and should temper the conclusions accordingly.
  2. [Section 4.2 and Table 4] All performance comparisons are reported as point estimates over three attempts per task with no confidence intervals, standard errors, or significance tests. The GUI-only versus Hybrid gap in Task Success Rate is 4.47 percentage points, which is small relative to the likely per-task variance across 201 tasks, and even the larger MCP-only gap is asserted without uncertainty quantification. The paper should report per-task variance, bootstrap confidence intervals, or paired significance tests before drawing conclusions about the relative ordering of the three configurations.
  3. [Section 3.2 and Appendix D] The validity of the benchmark's evaluation depends on author-written verification hooks capturing all legitimate completion paths and firing only on semantically correct events. Appendix D explicitly acknowledges that agents can use BashTool to bypass annotated key steps while still achieving final success, which shows that the key-step layer is not a complete model of task progress. The paper should validate hook coverage (for example, against human traces and against multiple solution paths), report hook precision and recall if possible, and discuss how missed or spurious hook firings could affect both Task Success Rate and Key Step Completion Rate.
  4. [Section 4.3 and Table 5] The failure-reason attribution in Table 5 is load-bearing for the discussion of MCP coverage, but the manuscript does not state how the labels were obtained: whether by LLM judgment, log inspection, human annotation, or an automatic classifier, and whether each failure was labeled once per run or per task. This method must be specified, and ideally validated with inter-annotator agreement or a manual audit, before the table can be used to support the claim that MCP-only underperformance is due mainly to 'Insufficient MCP coverage.'
minor comments (4)
  1. [Section 4.3] There is a typo in the sentence 'a more flexible action space that provices CUAs more freedom,' which should read 'provides.'
  2. [Table 2] The QGIS LOC entry appears as '1.3×10 6' with a malformed superscript; it should read '1.3×10^6' or equivalent.
  3. [Section 2.2] The text lists three input modes (GUI-only, CLI-only, and multimodal), but the experiments only compare GUI-only, MCP-only, and Hybrid; the relationship between 'CLI-only' and 'MCP-only' should be clarified.
  4. [Table 3] The difficulty thresholds '0-5', '5-10', and '10+' have overlapping boundaries at 5 and 10 steps; the paper should specify whether the boundaries are inclusive or exclusive.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: MCPWorld's success rates are measured outcomes from instrumentation hooks, not quantities derived from fitted inputs, and the only self-citation (LlamaTouch) is descriptive and not load-bearing.

full rationale

MCPWorld is a benchmark-construction paper: it contains no fitted parameters, no equations, and no derivation chain whose output could reduce to its own inputs. Task Success Rate and Key Step Completion Rate are recorded measurements produced by author-written verification hooks, and the paper's own Appendix D shows the two metrics diverge under BASH disabling (e.g., MCP-only SR 53.23% vs KSCR 59.78%, and Hybrid SR dropping from 75.12% to 65.50% when BASH is removed), which would be impossible if success were definitionally identical to the verification signal. The only overlapping-author citation is LlamaTouch [25] (authors Wang, Jia, Yan, Xu also appear here), used in Table 1 and Sections 2.3-2.4 to characterize prior external-state-matching evaluation methods ("matching agent actions against pre-defined UI trajectories [21, 25, 5, 13], inspecting UI accessibility trees [17, 25, 5, 9, ...]"). That citation supports background landscape claims only; the white-box verification paradigm is implemented directly with Frida and targeted code injection, not imported from [25], so it is not load-bearing. The 'first MCP-enabled testbed' claim is an empirical novelty claim checked against the surveyed prior work in Table 1, not a uniqueness theorem invoked to forbid alternatives. The BASH confound (Appendix C says BashTool/EditTool 'might still be available' in MCP-only setups; Figure 4 shows the headline modality ordering reverses when BASH is disabled) is a validity/correctness concern about what the modality comparison isolates, not circularity: the numbers remain measurements rather than quantities forced by construction. The Appendix D admission that agents can bypass annotated key steps via BASH is an honest limitation statement and does not create definitional coupling between the key-step metric and final success. No step equates a prediction to an input by definition, and no load-bearing premise rests on a self-citation. Score 1 reflects one minor, non-load-bearing self-citation only.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No physical entities are postulated. The load-bearing inputs are hand-authored tasks, hook placements, and experimental design constants; these are internal to the paper and are not independently verified by external artifacts beyond the public repository.

free parameters (3)
  • Difficulty thresholds = Easy: 0-5, Medium: 5-10, Hard: 10+ human GUI steps
    Hand-chosen step-count brackets define the difficulty distribution and drive the difficulty analysis in Table 6.
  • Per-task attempt count = 3
    Each task was attempted 3 times per configuration, and this hand-chosen sample size sets the resolution of all reported success-rate estimates.
  • Task timeout = 300 seconds
    The 300-second cutoff is a hand-chosen limit that directly affects reported success rates and the failure attribution in Table 5.
assumptions (5)
  • domain assumption Author-placed internal hooks in application source code correctly identify task completion and key-step completion.
    Section 3.2 describes annotators selecting the most critical moments in source code, but no independent validation or inter-annotator agreement on hook correctness is reported.
  • domain assumption Human GUI interaction step count is a valid proxy for task difficulty.
    Section 3.1 maps difficulty levels to step counts from human traces; this proxy is reasonable but is not validated against perceived difficulty or API-level complexity.
  • domain assumption Claude 3.7 Sonnet on the computer-use-demo agent is a representative agent for comparing interaction modalities.
    Section 4.1 uses a single agent framework, and the limitations section acknowledges that this restricts the generality of the empirical conclusions.
  • domain assumption Three repeated attempts per task are sufficient to estimate a stable task success rate.
    Section 4.1 sets three runs per configuration without reporting variance or confidence intervals, so the stability of the point estimates is unknown.
  • domain assumption Open-source applications with existing MCP servers represent the target population of CUA benchmarks.
    Section 3.1 selects apps partly based on MCP server availability, and the limitations section acknowledges the white-box constraint; this limits generalization to closed-source software.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents." pith.science (2026). https://pith.science/paper/U7WVJYRV

@misc{pith2026250607672,
  author       = {Pith},
  title        = {Pith review of: MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U7WVJYRV}},
  note         = {Machine review of arXiv:2506.07672}
}
read the original abstract

(M)LLM-powered computer use agents (CUA) are emerging as a transformative technique to automate human-computer interaction. However, existing CUA benchmarks predominantly target GUI agents, whose evaluation methods are susceptible to UI changes and ignore function interactions exposed by application APIs, e.g., Model Context Protocol (MCP). To this end, we propose MCPWorld, the first automatic CUA testbed for API, GUI, and API-GUI hybrid agents. A key principle of MCPWorld is the use of "white-box apps", i.e., those with source code availability and can be revised/re-compiled as needed (e.g., adding MCP support), with two notable advantages: (1) It greatly broadens the design space of CUA, such as what and how the app features to be exposed/extracted as CUA-callable APIs. (2) It allows MCPWorld to programmatically verify task completion by directly monitoring application behavior through techniques like dynamic code instrumentation, offering robust, accurate CUA evaluation decoupled from specific agent implementations or UI states. Currently, MCPWorld includes 201 well curated and annotated user tasks, covering diversified use cases and difficulty levels. MCPWorld is also fully containerized with GPU acceleration support for flexible adoption on different OS/hardware environments. Our preliminary experiments, using a representative LLM-powered CUA framework, achieve 75.12% task completion accuracy, simultaneously providing initial evidence on the practical effectiveness of agent automation leveraging MCP. Overall, we anticipate MCPWorld to facilitate and standardize the benchmarking of next-generation computer use agents that can leverage rich external tools. Our code and dataset are publicly available at https://github.com/SAAgent/MCPWorld.

Figures

Figures reproduced from arXiv: 2506.07672 by the authors.

Figure 1
Figure 1. The [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Comparison of evaluation paradigms for a dynamic task: debugging in an IDE. (a) [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Distribution of tasks in MCPWorld spanning ten open source applications. 3 The MCPWorld Benchmark Suite 3.1 Applications and Tasks The current MCPWorld benchmark suite includes 10 applications across diverse categories, as shown in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GABench: A Comprehensive Benchmark for Evaluating LLM Agents on Graph Analysis Tasks

    cs.AI 2026-08 conditional novelty 6.0 of 10

    GABench provides the first agentic graph-analysis benchmark with 10,400 executable tasks, and finds existing LLM agents succeed on under 40% of complex graph tasks.

  2. StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents

    cs.SE 2026-07 conditional novelty 6.0 of 10

    StateAct, a code-first multi-agent harness that acts on program state instead of pixels, improves Claude Opus 4.8 on OSWorld 2.0 from 20.6% to 26.9% binary success at ~9x lower cost.

  3. Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions

    cs.SE 2026-02 conditional novelty 6.0 of 10

    Most MCP tool descriptions (97.1%) contain quality smells, and augmenting them improves agent success by a median of 5.85 percentage points at a 67.46% increase in execution steps.

  4. Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning

    cs.LG 2025-12 conditional novelty 6.0 of 10

    CrossAgent learns step-level action-interface selection via a three-stage SFT + single-turn GRPO + multi-turn GRPO pipeline, reporting 54.6% mean success on 800+ Minecraft tasks after RL on only 30 tasks.

  5. MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models

    cs.AI 2025-07 conditional novelty 6.0 of 10

    MCPEval is an automated MCP-based framework that generates, verifies, and scores LLM agent tool-use tasks; its experiments reveal a consistent gap between how well agents execute tool calls and how well they synthesiz...

  6. A Lightweight Incentive-Based Privacy-Preserving Smart Metering Protocol for Value-Added Services

    cs.CR 2025-08 unverdicted novelty 4.0 of 10

    A layered protocol of local differential privacy, blind signatures, pseudonyms, temporal aggregation, and anonymous routing is claimed to keep smart-meter readings private while still enabling reward token redemption.

Reference graph

Works this paper leans on

33 extracted references · 20 canonical work pages · cited by 6 Pith papers

  1. [1]

    Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku

    Anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku. https: //www.anthropic.com/news/3-5-models-and-computer-use

    work page

  2. [2]

    Introduction: Get started with the model context protocol (mcp)

    Anthropic. Introduction: Get started with the model context protocol (mcp). https:// modelcontextprotocol.io/introduction

    work page

  3. [3]

    Meet claude on your desktop.https://claude.ai/download

    Anthropic. Meet claude on your desktop.https://claude.ai/download

    work page

  4. [4]

    Windows agent arena: Evaluating multi-modal os agents at scale, 2024

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale, 2024

    2024

  5. [5]

    A3: Android agent arena for mobile gui agents, 2025

    Yuxiang Chai, Hanhao Li, Jiayu Zhang, Liang Liu, Guangyi Liu, Guozhi Wang, Shuai Ren, Siyuan Huang, and Hongsheng Li. A3: Android agent arena for mobile gui agents, 2025

    work page 2025

  6. [6]

    Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks?, 2024

    2024

  7. [7]

    Assistgui: Task-oriented desktop graphical user interface automation, 2024

    Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, Hengxu Wang, Luowei Zhou, and Mike Zheng Shou. Assistgui: Task-oriented desktop graphical user interface automation, 2024. 10

    work page 2024

  8. [8]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024

    2024

Show all 33 references
  1. [9]

    Benchmarking mobile device control agents across diverse configurations, 2024

    Juyong Lee, Taywon Min, Minyong An, Dongyoon Hahm, Haeone Lee, Changyeon Kim, and Kimin Lee. Benchmarking mobile device control agents across diverse configurations, 2024

    2024

  2. [10]

    Api-bank: A comprehensive benchmark for tool-augmented llms, 2023

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for tool-augmented llms, 2023

    2023

  3. [11]

    Taskmatrix.ai: Completing tasks by connecting foundation models with millions of apis, 2023

    Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, Yun Wang, Linjun Shou, Ming Gong, and Nan Duan. Taskmatrix.ai: Completing tasks by connecting foundation models with millions of apis, 2023

    2023

  4. [12]

    Agentbench: Evaluating llms as agents, 2023

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench:...

    2023

  5. [13]

    Visualagentbench: Towards large multimodal models as visual foundation agents, 2024

    Xiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong, Yifan Xu, Xixuan Song, Shudan Zhang, Hanyu Lai, Xinyi Liu, Hanlin Zhao, Jiadai Sun, Xinyue Yang, Yu Yang, Zehan Qi, Shuntian Yao, Xueqiao Sun, Siyi Cheng, Qinkai Zheng, Hao Yu, Hanchen Zhang, Wenyi Hong, Ming Ding, Lihang Pan, Xia...

    2024

  6. [14]

    Gaia: a benchmark for general ai assistants, 2023

    Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants, 2023

    2023

  7. [15]

    frida: Dynamic instrumentation toolkit for developers, reverse-engineers, and security researchers.https://frida.re/

    oleavr. frida: Dynamic instrumentation toolkit for developers, reverse-engineers, and security researchers.https://frida.re/

  8. [16]

    Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to m...

    2023

  9. [17]

    Androidworld: A dynamic benchmarking environment for autonomous agents, 2025

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchma...

    2025

  10. [18]

    Toolalpaca: Generalized tool learning for language models with 3000 simulated cases, 2023

    Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases, 2023

    2023

  11. [19]

    Ducktrack: Multimodal computer agent data collection program

    TheDuckAI. Ducktrack: Multimodal computer agent data collection program. https:// github.com/TheDuckAI/DuckTrack

  12. [20]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for op...

    2024

  13. [21]

    Understanding the weakness of large language model agents within a complex android environment, 2024

    Mingzhe Xing, Rongkai Zhang, Hui Xue, Qi Chen, Fan Yang, and Zhen Xiao. Understanding the weakness of large language model agents within a complex android environment, 2024

    2024

  14. [22]

    Androidlab: Training and systematic benchmarking of android autonomous agents, 2024

    Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. Androidlab: Training and systematic benchmarking of android autonomous agents, 2024

    2024

  15. [23]

    Webshop: Towards scalable real-world web interaction with grounded language agents, 2023

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023. 11

    2023

  16. [24]

    React: Synergizing reasoning and acting in language models, 2023

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023

    2023

  17. [25]

    Llamatouch: A faithful and scalable testbed for mobile ui task automation, 2024

    Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. Llamatouch: A faithful and scalable testbed for mobile ui task automation, 2024

    2024

  18. [26]

    Agentstudio: A toolkit for building general virtual agents, 2025

    Longtao Zheng, Zhiyuan Huang, Zhenghai Xue, Xinrun Wang, Bo An, and Shuicheng Yan. Agentstudio: A toolkit for building general virtual agents, 2025

    2025

  19. [27]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024

    2024

  20. [28]

    source=<path_to_executable_on_host>,target=<path_in_container>,type=bind

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. Toolqa: A dataset for llm question answering with external tools, 2023. 12 Appendix A Details of theMCPWorldEnvironment A.1 Overall Infrastructure The MCPWorld execution environment is built upon an Ubuntu 22.04 co...

    2023

  21. [29]

    DISPLAY")} and use a subshell. For example

    Default System Prompt:Used in standard configurations where GUI interaction, BASH tool, and potentially MCP tools are available. Listing 1: Default System Prompt 1SYSTEM_PROMPT = f"""<SYSTEM_CAPABILITY> 2* You are utilising an Ubuntu virtual machine using {platform.machine()} ...

  22. [30]

    API-Only System Prompt:Used when the agent is configured for MCP-only interaction, restricting GUI tools but potentially allowing BASH. Listing 2: API-Only System Prompt 1SYSTEM_PROMPT_API_ONLY = f"""<SYSTEM_CAPABILITY> 2* You are utilising an Ubuntu virtual machine using {pla...

  23. [31]

    skip this step

    No-BASH System Prompt:Used in configurations where the BASH tool is disabled, but GUI interaction is still allowed. Listing 3: No-BASH System Prompt 1SYSTEM_PROMPT_NO_BASH = f"""<SYSTEM_CAPABILITY> 2* You are utilising an Ubuntu virtual machine using {platform.machine()} archi...

  24. [32]

    Translate the selected text in this document to French and save it

    No-BASH API-Only System Prompt:Used in the most restrictive setting where both BASH and direct GUI manipulation tools are disabled, forcing reliance on MCP tools only. Listing 4: No-BASH API-Only System Prompt 1SYSTEM_PROMPT_NO_BASH_API_ONLY = f"""<SYSTEM_CAPABILITY> 2* You ar...

  25. [33]

    6 Hard 36 Table 10: List of all tasks inMCPWorld

    to (-2, -2, 0). 6 Hard 36 Table 10: List of all tasks inMCPWorld. ID App Name Template Key Steps Difficulty Level 199 Blender In Blender, add a {type} type {meshtype}, apply a {modifiertype} Modifier. 4 Medium 200 Blender In Blender, add a {type} type {meshtype}, move it down ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.