Pith. sign in

REVIEW 4 major objections 5 minor 74 references

Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Existing mobile agents master atomic steps but fail when instructions compose several dependent steps; this paper shows the gap is large and that a lightweight scheduler, AGENT-NEXUS, recovers 76-88% of it.

desk verdict A genuinely useful benchmark and a plausible scheduling system whose headline attribution to the scheduler itself outruns the current experiments. read the letter →

arxiv 2506.08972 v1 pith:XDPC4DDS submitted 2025-06-10 cs.CL

Yuan Guo , Tingjia Miao , Zheng Wu , Pengzhou Cheng , Ming Zhou , Zhuosheng Zhang This is my paper
classification cs.CL
keywords mobileagentsatomic-to-compositionalgeneralizationcompositionaltasksUI-NEXUSbenchmarkhierarchicalschedulingprocessmemorymultimodalLLMGUI
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most mobile agents handle single-screen, self-contained operations well but stumble when one instruction chains several dependent steps—reading an invitation, checking a calendar, replying conditionally. This paper introduces UI-NEXUS, a 100-template bilingual benchmark whose tasks follow three dependency structures—Simple Concatenation, Context Transition, and Deep Dive—and documents a systematic atomic-to-compositional generalization gap: every tested agent succeeds far more often when the same task is handed to it as manually decomposed atomic subtasks. To close that gap, the paper proposes AGENT-NEXUS, a hierarchical scheduler that decomposes the long-horizon instruction into self-contained subtasks, stores intermediate results in process memory, and re-plans after each subtask. On the 50-task local anchor subset, AGENT-NEXUS raises M3A from 50% to 74% and UI-TARS-7B-SFT from 6% to 46%, recovering 88% and 76% of the measured gap respectively with only modest added latency and token cost. The paper's message is that composition management, not atomic skill, is the current bottleneck, and that a decoupled scheduling layer can supply most of it.

What carries the argument

The load-bearing mechanism is hierarchical adaptive orchestration: AGENT-NEXUS's Scheduling Module decomposes each compositional goal into typed subtasks (act, think, tool); the Execution Module carries them out through an Analyst for general reasoning, a Navigator for device interaction, and a Tool for shortcuts like returning home; and Process Memory stores each subtask's return signal so that Context Transition and Deep Dive dependencies are instantiated with real intermediate data rather than fabricated values. Re-planning after every subtask, as $\pi_{t+1} = M(s_{t+1}, g, P_t + 1)$, adapts the schedule to execution outcomes. The companion measurement device is the PGR metric, defined as $\frac{\text{atomic-to-compositional performance} - \text{weak performance}}{\text{strong ceiling performance} - \text{weak performance}}$, which quantifies how much of the gap any method closes.

What would settle it

Run the same five baselines and AGENT-NEXUS on a held-out set of compositional instructions written by independent annotators from the same compositional grammar but never passed through the paper's GPT-based brainstorming and human-refinement pipeline; if the documented gap shrinks or AGENT-NEXUS's PGR falls well below 76-88%, the central claim weakens. A second check: strip AGENT-NEXUS's scheduling layer while keeping process memory on the navigator; if most of the success gain survives, decomposition itself is not the active ingredient.

Watch

Extended reading notes

Core claim

The central claim is that instruction-level atomic-to-compositional generalization is a distinct, measurable capability that current mobile agents largely lack, and that most of that missing capability can be supplied externally. The authors formalize an atomic subtask as a self-contained unit $\langle c, p, e\rangle$ and a compositional task as a set of subtasks with a dependency structure $\langle A_{\mathrm{sub}}, D\rangle$, categorized into Simple Concatenation (independent steps), Context Transition (the output of one step feeds the next), and Deep Dive (reasoning between steps). Measuring the gap as the difference between performance on the full compositional instruction and performance on manually decomposed atomic instructions, they find large gaps: UI-TARS-7B-SFT drops from 60% to 11% and M3A from 80% to 57%. AGENT-NEXUS—a scheduling module with a GPT-4o orchestrator, an execution module with an analyst and a navigator, and a process memory that carries return signals—recovers 88% (with M3A) and 76% (with UI-TARS) of that gap, quantified by their Performance Gap Recovered (PGR) metric. The conclusion is that dynamically decomposing tasks and managing intermediate information is sufficient, without significantly sacrificing inference overhead, to extrapolate existing agents' abilities to compositional mobile tasks.

Load-bearing premise

The whole argument rests on the assumption that UI-NEXUS's 100 task templates, the human-refined manual atomic decompositions used as the strong ceiling, and the success verification procedure genuinely measure atomic-to-compositional generalization rather than a quirk of how the tasks are worded, decomposed, or checked.

Editorial extensions

If this is right

  • A small fine-tuned navigator like UI-TARS-7B-SFT can be lifted from 6% to 46% on the local anchor subset, and from 4-8% to 28-32% on online apps, purely by adding the scheduling layer—so capability gains need not come from a larger model.
  • AGENT-NEXUS with M3A reaches 74% on UI-NEXUS-ANCHOR with an 88% PGR, implying that even a strong workflow agent's residual failures on compositional tasks are mostly scheduling failures rather than grounding failures.
  • On Context Transition tasks, the system shows that passing real intermediate information through process memory can substitute for the limited memory span of agent-as-a-model systems.
  • Efficiency is preserved: with the UI-TARS navigator, the system costs $0.0063 per step and 6.53 seconds per step versus $0.0025 and 4.35 seconds for the plain navigator, while multiplying the anchor-subset success rate by more than seven.
  • The benchmark plus PGR gives the field a standardized way to measure instruction-level composition generalization, so future methods can be compared on how much of the gap they recover rather than raw success alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If composition management is the bottleneck, the same decoupled-scheduling recipe should transfer to web and desktop GUI agents, where long-horizon instructions with context transitions are equally common; UI-NEXUS's dependency taxonomy gives a template for building equivalent benchmarks there.
  • A testable consequence: if the scheduler's subtask trajectories are used as training data for the navigator, the navigator may internalize the composition skill, eventually allowing the scheduler to be simplified or removed; the paper's PGR numbers define the upper bound such distillation could achieve.
  • The online results hint that environment instability (changing UIs, unavailable accessibility trees) interacts with the composition gap—M3A's Chinese-app success falls to 4% with a 40% collapse rate—so real-world scheduling gains may require pairing with more robust perception.
  • Because the strong ceiling comes from the authors' own manual atomic decompositions, PGR could overstate both the gap and the recovery if that decomposition over-specifies the optimal path; re-annotating with independent raters is a cheap validation step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces UI-NEXUS, an interactive benchmark of 100 Android task templates spanning 50 apps and three compositional types (Simple Concatenation, Context Transition, Deep Dive), together with an extensible evaluation infrastructure and a 20-app controllable anchor subset. It evaluates five mobile agents and reports that all perform poorly on compositional instructions while performing much better on manually decomposed atomic subtasks, demonstrating an atomic-to-compositional generalization gap. To close this gap, the paper proposes AGENT-NEXUS, a scheduling system that uses a GPT-4o orchestrator and analyst to decompose long-horizon tasks into subtasks and to manage a process memory, while a separate navigator (M3A or UI-TARS-7B-SFT) executes device operations. The headline result is 24–40 percentage-point improvements in success rate and a Performance Gap Recovered (PGR) of 76–88%, with the claim that this is achieved without significantly sacrificing inference overhead.

Significance. If the causal claims were established, UI-NEXUS would be a useful community resource: it provides a systematic compositional-task taxonomy, a controllable offline anchor subset, public code and data, and a transparently defined PGR metric with no fitted parameters. The reported gains are directionally consistent across Tables 2 and 5, and the observation that small specialized navigators benefit from a separate planning/reasoning layer is scientifically interesting. However, the central attribution of the gains to the scheduling mechanism rather than to the added GPT-4o reasoning compute is not yet supported, and the efficiency claim is measured only per step. The benchmark and the direction of the effect are valuable, but the paper needs targeted ablations and uncertainty quantification before the headline claims can be accepted.

major comments (4)
  1. [Section 4, Tables 2–3, Appendix C] The causal role of the scheduling mechanism is not isolated from the added GPT-4o reasoning compute. AGENT-NEXUS's Orchestrator and Analyst are implemented with GPT-4o (Section 4 and Appendix C), and in the UI-TARS condition the system is effectively a GPT-4o planner/analyst plus a 7B navigator, while the baseline is UI-TARS alone. Consequently, the 40-point improvement on the anchor subset (Table 2) and the PGR of 76% (Table 5) could be produced by upgrading the reasoning model rather than by dynamic decomposition and re-planning. Please add ablations that (i) run UI-TARS with a GPT-4o analyst in a non-decomposed, step-level reasoning loop, (ii) replace the GPT-4o scheduler/analyst with a model of comparable size to the navigator, and (iii) disable the dynamic re-planning after each subtask (i.e., use a static decomposition). At least one of these controls is needed to attribute the improvement to the scheduling mechanism.
  2. [Abstract, Table 3, Section 5.2] The claim that improvements come 'without significantly sacrificing inference overhead' is not supported by the reported per-step metrics. Table 3 reports latency and cost per step only, but the baselines and AGENT-NEXUS produce very different trajectories: UI-TARS alone has 84% Budget Exceeded on the anchor subset (Table 2), so per-step costs cannot be converted to per-task or per-success costs without step counts. For UI-TARS, per-step cost rises from $0.0025 to $0.0063 and latency from 4.35s to 6.53s; the total-overhead comparison is therefore essential. Please report total inference cost and total latency per task (and per successful task) for every condition, together with the number of steps taken.
  3. [Tables 2, 4, and 5] The success rates are point estimates over small samples with no measure of uncertainty. The number of tasks underlying each online row in Table 4 is not stated, and no confidence intervals, bootstrap intervals, or repeated-run variances are reported anywhere. For instance, the 8% to 32% improvement on Chinese online apps corresponds to a small number of successes, and on the anchor subset a 24–40 point improvement over 50 tasks may still be compatible with substantial sampling noise. Please state the number of tasks per row and per composition category, report standard errors or confidence intervals (e.g., Wilson or bootstrap), and specify whether runs are single or repeated. This is load-bearing for the headline improvement claim.
  4. [Section 3.3, Section 3.4, Appendix H, Table 5] The PGR metric and the benchmark's ground truth depend on the authors' manual atomic decompositions and on the verification protocol, but neither is shown to be reliable. The strong ceiling in Eq. (1) is computed from manually decomposed atomic instructions; these decompositions are not released and no inter-annotator agreement is reported. Likewise, the online verification uses MLLM-as-a-judge plus human verification (Section 3.4), but no count of human-verified trajectories or agreement statistics are given, and Appendix H provides only the prompt text. Please release the manual decompositions, report agreement between annotators on a sample, and quantify how many online episodes were human-verified. Without these, the measured generalization gap and the PGR values could be an artifact of annotation choices rather than a property of the agents.
minor comments (5)
  1. [Section 5.2] The sentence 'Detailed experimental results ... are listed in Table 2, Table and Table 4' contains a missing table number; it should read 'Table 2, Table 3, and Table 4'.
  2. [Section 5.3.1] There is a typo in 'restrains their portential for compositional tasks'; it should be 'potential'.
  3. [Appendix C] The phrase 'every steo input' should be 'every step input'.
  4. [Appendix H.2] The heading 'Task Brainstorming Prompt' appears above the success-verification instructions; this heading should be renamed to reflect the actual content (e.g., 'Task Success Verification Prompt').
  5. [Table 5] The parenthetical values in the Overall-Atom column (e.g., '↑87%', '↑452%') are relative improvements of the strong ceiling over the weak baseline, not PGR values; the final column is populated only for AGENT-NEXUS. Please add an explicit note distinguishing these two quantities, since the column layout invites confusion with the PGR definition in Eq. (1).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: PGR is a measurement-normalization definition over independent runs; no fitted parameter or self-citation chain forces the headline results.

full rationale

The paper's derivation chain is empirical, not analytic. The central quantities—UI-NEXUS success rates, termination reasons, per-step latency/cost, and the PGR values in Table 5—are measured outcomes, not outputs of a derivation that assumes them. PGR (Eq. 1) is a definitional normalization: PGR = (atomic-to-compositional performance − weak performance)/(strong ceiling performance − weak performance). Each quantity on the right is obtained from separate evaluation runs: weak performance is the agent given the full compositional instruction; strong ceiling is the same agent given manually decomposed atomic subtasks; atomic-to-compositional performance for AGENT-NEXUS is its own success rate when it performs dynamic decomposition. AGENT-NEXUS does not receive the manual atomic decompositions at test time, so its PGR is not forced by construction. There are no fitted parameters, no training on the benchmark, and no self-citation that supplies a load-bearing premise: the citations to the authors' prior work ([17], [65], [69]) are used for general context or failure-mode analogies, not to justify the gap being measured or to rule out alternatives. The closest concerns—that the benchmark is new and self-referential, that strong-ceiling annotations are author-provided, and that the gains of AGENT-NEXUS are not ablated from the added GPT-4o scheduler/analyst compute—are validity and attribution risks, not circularity; no equation or construction reduces to its own input. Therefore no circular step is exhibited.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No physical entities are invented and no models are fit to data. The ledger records two hand-chosen evaluation hyperparameters and three construct-validity assumptions on which the measured generalization gap rests.

free parameters (2)
  • Step budget multiplier (max steps per task) = approximately 2x human-annotated optimal steps
    Termination reasons and success rates depend directly on this budget. Agents that exceed it are marked as Step Budget Exceeded rather than successful. The multiplier is chosen by hand in Appendix E, not derived from a principle.
  • UI-TARS screenshot history length N = 5
    The number of previous screenshots given to UI-TARS affects its context and success rate. It is set to the official repository default and not swept or justified against the benchmark, making it a hand-chosen evaluation hyperparameter that influences the UI-TARS baseline.
assumptions (3)
  • domain assumption Success on the 100 UI-NEXUS templates measures atomic-to-compositional generalization in mobile agents.
    The benchmark defines the phenomenon it claims to measure. No external benchmark or independent construct validation is provided, so the central gap could be partly an artifact of task phrasing or environment setup.
  • ad hoc to paper Manual atomic decompositions by the authors are a valid, near-optimal strong ceiling for the PGR metric.
    The strong ceiling in Eq. 1 and Table 5 is constructed by the authors manually decomposing selected tasks. No inter-annotator agreement or comparison with other decomposition methods is reported, so the ceiling could be optimistic or idiosyncratic.
  • domain assumption MLLM-as-a-judge plus human verification correctly labels success on online service app tasks.
    Appendix H provides verification prompts but no agreement statistics or audit of judge reliability. Online environments are also changing and manually configured, so success labels in Tables 4, 6, and 7 carry unmeasured verification error.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System." pith.science (2026). https://pith.science/paper/XDPC4DDS

@misc{pith2026250608972,
  author       = {Pith},
  title        = {Pith review of: Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XDPC4DDS}},
  note         = {Machine review of arXiv:2506.08972}
}
read the original abstract

Autonomous agents powered by multimodal large language models have been developed to facilitate task execution on mobile devices. However, prior work has predominantly focused on atomic tasks -- such as shot-chain execution tasks and single-screen grounding tasks -- while overlooking the generalization to compositional tasks, which are indispensable for real-world applications. This work introduces UI-NEXUS, a comprehensive benchmark designed to evaluate mobile agents on three categories of compositional operations: Simple Concatenation, Context Transition, and Deep Dive. UI-NEXUS supports interactive evaluation in 20 fully controllable local utility app environments, as well as 30 online Chinese and English service apps. It comprises 100 interactive task templates with an average optimal step count of 14.05. Experimental results across a range of mobile agents with agentic workflow or agent-as-a-model show that UI-NEXUS presents significant challenges. Specifically, existing agents generally struggle to balance performance and efficiency, exhibiting representative failure modes such as under-execution, over-execution, and attention drift, causing visible atomic-to-compositional generalization gap. Inspired by these findings, we propose AGENT-NEXUS, a lightweight and efficient scheduling system to tackle compositional mobile tasks. AGENT-NEXUS extrapolates the abilities of existing mobile agents by dynamically decomposing long-horizon tasks to a series of self-contained atomic subtasks. AGENT-NEXUS achieves 24% to 40% task success rate improvement for existing mobile agents on compositional operation tasks within the UI-NEXUS benchmark without significantly sacrificing inference overhead. The demo video, dataset, and code are available on the project page at https://ui-nexus.github.io.

Figures

Figures reproduced from arXiv: 2506.08972 by the authors.

Figure 1
Figure 1. Visualization of the atomic-to-compositional generalization issue of mobile agents. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the composition type taxonomy in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of UI-NEXUS. It features comprehensive coverage of applications and task scenarios, systematic analysis of subtask dependencies, diverse evaluation metrics, and supports both fully reproducible offline evaluations and real-world online tests. UI-NEXUS is built upon our unified plug-and-play framework that seamlessly integrates heterogeneous agents and devices. 3.2 Task Formulation In this section, we formal… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Data statistics of UI-NEXUS instruction templates. There are 100 challenging templates in total, covering systematic subtask dependency structures, application types and using scenarios. 3.5 Benchmark Statistics [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Overview of AGENT-NEXUS framework. AGENT-NEXUS follows a hierarchical adaptive orchestration design, with three interconnected modules Scheduling Layer, Execution Layer an Pro￾cess Memory. We demonstrate a representative example where the different components seamlessl…
Figure 6
Figure 6. Figure 6: Representative examples of attention-drift and over-execution failure mode. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Representative examples of under-execution and context confusion failure mode. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Representative examples of inner logic error failure mode. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

74 extracted references · 12 canonical work pages

  1. [1]

    Large language model-brained gui agents: A survey

    Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Qingwei Lin, Saravan Rajmohan, et al. Large language model-brained gui agents: A survey. arXiv preprint arXiv:2411.18279, 2024. 10

    arXiv 2024

  2. [2]

    Llm-powered gui agents in phone automation: Surveying progress and prospects

    Guangyi Liu, Pengxiang Zhao, Liang Liu, Yaxuan Guo, Han Xiao, Weifeng Lin, Yuxiang Chai, Yue Han, Shuai Ren, Hao Wang, et al. Llm-powered gui agents in phone automation: Surveying progress and prospects. arXiv preprint arXiv:2504.19838, 2025

    arXiv 2025

  3. [3]

    Gui agents: A survey

    Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, et al. Gui agents: A survey. arXiv preprint arXiv:2412.13501, 2024

    arXiv 2024

  4. [4]

    Os agents: A survey on mllm-based agents for general computing devices use, 2024

    Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, et al. Os agents: A survey on mllm-based agents for general computing devices use, 2024

    work page 2024

  5. [5]

    Gui agents with foundation models: A comprehensive survey

    Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, et al. Gui agents with foundation models: A comprehensive survey. arXiv preprint arXiv:2411.04890, 2024

    arXiv 2024

  6. [6]

    Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning

    Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. arXiv preprint arXiv:2406.11896, 2024

    arXiv 2024

  7. [7]

    Distrl: An asynchronous distributed reinforcement learning framework for on-device control agents

    Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Distrl: An asynchronous distributed reinforcement learning framework for on-device control agents. arXiv preprint arXiv:2410.14803, 2024

    arXiv 2024

  8. [8]

    Autoglm: Autonomous foundation agents for guis

    Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, et al. Autoglm: Autonomous foundation agents for guis. arXiv preprint arXiv:2411.00820, 2024

    arXiv 2024

Show all 74 references
  1. [9]

    Ui-tars: Pioneering automated gui interaction with native agents

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  2. [10]

    Appagentx: Evolving gui agents as proficient smartphone users

    Wenjia Jiang, Yangyang Zhuang, Chenxi Song, Xu Yang, and Chi Zhang. Appagentx: Evolving gui agents as proficient smartphone users. arXiv preprint arXiv:2503.02268, 2025

    2025 arXiv

  3. [11]

    Os-atlas: A foundation action model for generalist gui agents

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024

    arXiv

  4. [12]

    Navigating the digital world as humans do: Universal visual grounding for gui agents

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024

    2024 arXiv

  5. [13]

    A3: Android agent arena for mobile gui agents

    Yuxiang Chai, Hanhao Li, Jiayu Zhang, Liang Liu, Guangyi Liu, Guozhi Wang, Shuai Ren, Siyuan Huang, and Hongsheng Li. A3: Android agent arena for mobile gui agents. arXiv preprint arXiv:2501.01149, 2025

    2025

  6. [14]

    Androidworld: A dynamic benchmarking environment for autonomous agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573, 2024

    2024 arXiv

  7. [15]

    Androidlab: Training and systematic benchmarking of android autonomous agents

    Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. Androidlab: Training and systematic benchmarking of android autonomous agents. arXiv preprint arXiv:2410.24024, 2024

    arXiv

  8. [16]

    An- droidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36, 2024

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36, 2024

    2024

  9. [17]

    You only look at screens: Multimodal chain-of-action agents

    Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. arXiv preprint arXiv:2309.11436, 2023. 11

    2023 arXiv

  10. [18]

    Ferret-ui: Grounded mobile ui understanding with multimodal llms

    Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms. In European Conference on Computer Vision, pages 240–255. Springer, 2024

    2024

  11. [19]

    On the effects of data scale on computer control agents

    Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. On the effects of data scale on computer control agents. arXiv e-prints, pages arXiv–2406, 2024

    2024

  12. [20]

    Seeclick: Harnessing gui grounding for advanced visual gui agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024

    2024 arXiv

  13. [21]

    Mobile-bench: An evaluation benchmark for llm-based mobile agents

    Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, et al. Mobile-bench: An evaluation benchmark for llm-based mobile agents. arXiv preprint arXiv:2407.00993, 2024

    2024 arXiv

  14. [22]

    Appagent: Multimodal agents as smartphone users

    Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint arXiv:2312.13771, 2023

    2023 arXiv

  15. [23]

    Autodroid: Llm-powered task automation in android

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pag...

    2024

  16. [24]

    Spa-bench: A comprehensive benchmark for smartphone agent evaluation

    Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, et al. Spa-bench: A comprehensive benchmark for smartphone agent evaluation. In NeurIPS 2024 Workshop on Open-World Agents, 2024

    2024

  17. [25]

    Appagent v2: Advanced agent for flexible mobile interactions.arXiv preprint arXiv:2408.11824, 2024

    Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. Appagent v2: Advanced agent for flexible mobile interactions.arXiv preprint arXiv:2408.11824, 2024

    2024

  18. [26]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158, 2024

    2024 arXiv

  19. [27]

    Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration

    Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv preprint arXiv:2406.01014, 2024

    2024 arXiv

  20. [28]

    Mobile-agent-e: Self-evolving mobile assistant for complex tasks

    Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. Mobile-agent-e: Self-evolving mobile assistant for complex tasks. arXiv preprint arXiv:2501.11733, 2025

    2025 arXiv

  21. [29]

    Ufo: A ui-focused agent for windows os interaction

    Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, et al. Ufo: A ui-focused agent for windows os interaction. arXiv preprint arXiv:2402.07939, 2024

    2024 arXiv

  22. [30]

    Gpt-4v (ision) is a generalist web agent, if grounded

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024

    2024 arXiv

  23. [31]

    Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection

    Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, and Fei Wu. Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection. arXiv preprint arXiv:2501.04575, 2025

    2025 arXiv

  24. [32]

    Agent s: An open agentic framework that uses computers like a human.arXiv preprint arXiv:2410.08164, 2024

    Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like a human.arXiv preprint arXiv:2410.08164, 2024

    2024 arXiv

  25. [33]

    Agent workflow memory

    Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint arXiv:2409.07429, 2024

    2024 arXiv

  26. [34]

    Omniparser for pure vision based gui agent

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.00203, 2024. 12

    2024 arXiv

  27. [35]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  28. [36]

    Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation

    An Yan, Zhengyuan Yang, Wanrong Zhu, Kevin Lin, Linjie Li, Jianfeng Wang, Jianwei Yang, Yiwu Zhong, Julian McAuley, Jianfeng Gao, et al. Gpt-4v in wonderland: Large multimodal models for zero-shot smartphone gui navigation. arXiv preprint arXiv:2311.07562, 2023

    2023 arXiv

  29. [37]

    Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451, 2024

    2024 arXiv

  30. [38]

    Showui: One vision-language-action model for gui visual agent

    Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-action model for gui visual agent. arXiv preprint arXiv:2411.17465, 2024

    2024 arXiv

  31. [39]

    Spiritsight agent: Advanced gui agent with one look

    Zhiyuan Huang, Ziming Cheng, Junting Pan, Zhaohui Hou, and Mingjie Zhan. Spiritsight agent: Advanced gui agent with one look. arXiv preprint arXiv:2503.03196, 2025

    2025 arXiv

  32. [40]

    Android in the zoo: Chain-of-action-thought for gui agents

    Jiwen Zhang, Jihao Wu, Yihua Teng, Minghui Liao, Nuo Xu, Xiao Xiao, Zhongyu Wei, and Duyu Tang. Android in the zoo: Chain-of-action-thought for gui agents. arXiv preprint arXiv:2403.02713, 2024

    2024 arXiv

  33. [41]

    Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning

    Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, et al. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. arXiv preprint arXiv:2411.02337, 2024

    2024 arXiv

  34. [42]

    Agent q: Advanced reasoning and learning for autonomous ai agents

    Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199, 2024

    2024 arXiv

  35. [43]

    Aguvis: Unified pure vision agents for autonomous gui interaction

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454, 2024

    2024 arXiv

  36. [44]

    Screenspot-pro: Gui grounding for professional high-resolution computer use, 2025

    Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use, 2025

    2025

  37. [45]

    Gui-bee: Align gui action grounding to novel environments via autonomous exploration

    Yue Fan, Handong Zhao, Ruiyi Zhang, Yu Shen, Xin Eric Wang, and Gang Wu. Gui-bee: Align gui action grounding to novel environments via autonomous exploration. arXiv preprint arXiv:2501.13896, 2025

    2025 arXiv

  38. [46]

    Screenqa: Large-scale question-answer pairs over mobile app screenshots

    Yu-Chung Hsiao, Fedir Zubach, Gilles Baechler, Victor Carbune, Jason Lin, Maria Wang, Srinivas Sunkara, Yun Zhu, and Jindong Chen. Screenqa: Large-scale question-answer pairs over mobile app screenshots. arXiv preprint arXiv:2209.08199, 2022

    2022 arXiv

  39. [47]

    Websrc: a dataset for web-based structural reading comprehension

    Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. Websrc: a dataset for web-based structural reading comprehension. arXiv preprint arXiv:2101.09465, 2021

    2021 arXiv

  40. [48]

    Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05955, 2024

    Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? arXiv preprint arXiv:2404.05955, 2024

    2024 arXiv

  41. [49]

    Weblinx: Real-world website navigation with multi-turn dialogue

    Xing Han Lù, Zdenˇek Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue. arXiv preprint arXiv:2402.05930, 2024

    2024

  42. [50]

    Meta-gui: Towards multi-modal conversational agents on mobile gui

    Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. Meta-gui: Towards multi-modal conversational agents on mobile gui. arXiv preprint arXiv:2205.11029, 2022

    2022 arXiv

  43. [51]

    Mind2web: Towards a generalist agent for the web

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114, 2023. 13

    2023

  44. [52]

    Mobileagentbench: An efficient and user-friendly benchmark for mobile llm agents

    Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. Mobileagentbench: An efficient and user-friendly benchmark for mobile llm agents. arXiv preprint arXiv:2406.08184, 2024

    2024 arXiv

  45. [53]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processin...

    2025

  46. [54]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023

    2023 arXiv

  47. [55]

    Webcanvas: Benchmarking web agents in online environments

    Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. Webcanvas: Benchmarking web agents in online environments. arXiv preprint arXiv:2406.12373, 2024

    2024 arXiv

  48. [56]

    Windows agent arena: Evaluating multi-modal os agents at scale

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, et al. Windows agent arena: Evaluating multi-modal os agents at scale. arXiv preprint arXiv:2409.08264, 2024

    2024 arXiv

  49. [57]

    Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.07718, 2024

    Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.07718, 2024

    2024 arXiv

  50. [58]

    Pc-agent: A hierarchical multi-agent collaboration framework for complex task automation on pc

    Haowei Liu, Xi Zhang, Haiyang Xu, Yuyang Wanyan, Junyang Wang, Ming Yan, Ji Zhang, Chunfeng Yuan, Changsheng Xu, Weiming Hu, et al. Pc-agent: A hierarchical multi-agent collaboration framework for complex task automation on pc. arXiv preprint arXiv:2502.14282, 2025

    2025 arXiv

  51. [59]

    Vsc-rl: Advancing autonomous vision-language agents with variational subgoal-conditioned reinforcement learning

    Qingyuan Wu, Jianheng Liu, Jianye Hao, Jun Wang, and Kun Shao. Vsc-rl: Advancing autonomous vision-language agents with variational subgoal-conditioned reinforcement learning. arXiv preprint arXiv:2502.07949, 2025

    2025 arXiv

  52. [60]

    Markov decision processes

    Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331–434, 1990

    1990

  53. [61]

    Atom of thoughts for markov llm test-time scaling

    Fengwei Teng, Zhaoyang Yu, Quan Shi, Jiayi Zhang, Chenglin Wu, and Yuyu Luo. Atom of thoughts for markov llm test-time scaling. arXiv preprint arXiv:2502.12018, 2025

    2025

  54. [62]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  55. [63]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36:8634–8652, 2023

    2023

  56. [64]

    Weak-to-strong gener- alization: Eliciting strong capabilities with weak supervision

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschen- brenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong gener- alization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023

    2023 arXiv

  57. [65]

    Caution for the environment: Multimodal agents are susceptible to environmental distractions

    Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, and Hai Zhao. Caution for the environment: Multimodal agents are susceptible to environmental distractions. arXiv preprint arXiv:2408.02544, 2024

    2024 arXiv

  58. [66]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024. Accessed: 2025-02-26

    2024

  59. [67]

    Os-atlas: A foundation action model for generalist gui agents

    OS-Copilot. Os-atlas: A foundation action model for generalist gui agents. https: //huggingface.co/OS-Copilot/OS-Atlas-Pro-7B , 2025. Accessed: 2025-02-28. 14

    2025

  60. [68]

    Ui-tars: Pioneering automated gui interaction with native agents

    ByteDance. Ui-tars: Pioneering automated gui interaction with native agents. https:// github.com/bytedance/UI-TARS, 2025. Accessed: 2025-02-28

    2025

  61. [69]

    Thoughts are all over the place: On the underthinking of o1-like llms

    Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. Thoughts are all over the place: On the underthinking of o1-like llms. arXiv preprint arXiv:2501.18585, 2025

    2025 arXiv

  62. [70]

    Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. Advances in Neural Information Processing Systems, 36:74952–74965, 2023

    2023

  63. [71]

    Measuring faithfulness in chain-of-thought reasoning

    Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023

    2023 arXiv

  64. [72]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, p...

    2023

  65. [73]

    Generalization of compositional tasks with logical specification via implicit planning

    Duo Xu and Faramarz Fekri. Generalization of compositional tasks with logical specification via implicit planning. arXiv preprint arXiv:2410.09686, 2024

    2024 arXiv

  66. [74]

    ping-pong

    Hiroki Furuta, Yutaka Matsuo, Aleksandra Faust, and Izzeddin Gur. Exposing limitations of language model agents in sequential-task compositions on the web. arXiv preprint arXiv:2311.18751, 2023. A Limitations We acknowledge two primary limitations in our study. First, due to t...

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.