Pith. sign in

REVIEW 2 major objections 6 minor 2 cited by

Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Skill hierarchy lifts GUI agent success by up to 79%.

desk verdict A coherent and useful GUI-agent system whose headline gains are undercut by a missing overlap check between its skill memory and its evaluation tasks, especially on the authors' own AndroidLH benchmark. read the letter →

arxiv 2506.10387 v1 pith:X7552FTI submitted 2025-06-12 cs.AI

Yuquan Xie , Zaijing Li , Rui Shao , Gongwei Chen , Kaiwen Zhou , Yinchuan Li , Dongmei Jiang , Liqiang Nie This is my paper
classification cs.AI
keywords GUIagentshierarchicalskillsskillmemoryMonteCarlotreesearchlong-horizontasksmultimodallargelanguagemodelsAndroidbenchmarkdomaingap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a graphical-user-interface (GUI) agent can handle long, multi-step tasks in live mobile and web environments by keeping a hierarchical memory of skills and using that memory to guide Monte Carlo tree search. It introduces Mirage-1, an agent whose Hierarchical Multimodal Skills (HMS) module abstracts previous trajectories into execution skills, core skills, and meta-skills, and whose Skill-Augmented MCTS (SA-MCTS) samples sub-goals from those skills during online search. The paper reports that Mirage-1 outperforms prior agents by 32% on AndroidWorld, 19% on MobileMiniWob++, 15% on Mind2Web-Live, and 79% on the new AndroidLH long-horizon benchmark, across four different grounding models. If correct, this indicates that explicit skill abstraction plus skill-guided search can substantially close the offline-to-online gap that limits current GUI agents.

What carries the argument

The central machinery is the Hierarchical Multimodal Skills (HMS) module, a three-level memory in which execution skills summarize individual trajectories, core skills generalize similar execution skills into functions (e.g., add_contact(name, number)), and meta-skills group related core skills into categories such as ContactManagement. HMS is paired with Skill-Augmented Monte Carlo Tree Search (SA-MCTS), which uses HMS to sample candidate sub-goals during tree expansion, estimates their values, rolls out the chosen sub-goal, backpropagates the reward, and then refines the skill module with successful online trajectories. This combined mechanism carries the argument because it converts raw experience into reusable planning knowledge offline and then lets that knowledge focus online exploration, shrinking the action search space.

What would settle it

Compute the task-string and app-level overlap between the 200 GPT-4o-generated exploration tasks (and the offline AITW/Multimodal-Mind2Web trajectories) and the test tasks in AndroidWorld, MobileMiniWob++, Mind2Web-Live, and AndroidLH. If the overlap is substantially above random, or if removing all matching skills from HMS collapses Mirage-1's success rate back to the baseline, the reported 32%/19%/15%/79% gains would be attributable to benchmark contamination rather than to skill generalization.

Watch

Extended reading notes

Core claim

Mirage-1's central claim is that organizing prior GUI trajectories into a three-level skill hierarchy, and then letting Monte Carlo tree search propose sub-goals by drawing on that hierarchy, improves online long-horizon task success beyond what an offline-trained multimodal planner plus a visual grounding model can achieve alone. Concretely, the paper reports task-success-rate gains of 32.3%, 19.1%, and 15.8% on AndroidWorld, MobileMiniWob++, and Mind2Web-Live over the grounding-model baselines, and a 79.6% gain on AndroidLH, a new 30-task multi-app benchmark assembled from AndroidWorld's verification mechanism. The ablation study attributes the largest contributions to the meta-skill layer and to the offline-initialized skills, supporting the claim that both abstraction level and offline knowledge are load-bearing for the reported gains.

Load-bearing premise

The load-bearing premise is that the skill library built from offline trajectories (AITW for mobile, Multimodal-Mind2Web for web) and from 200 GPT-4o-generated exploration tasks generalizes to the evaluation benchmarks without sharing their task distribution; if overlap exists, the reported gains could reflect memorization of similar tasks rather than transfer.

Editorial extensions

If this is right

  • Plug-and-play improvement: wrapping an existing multimodal planner plus a visual grounding model with HMS and SA-MCTS improves its online success rate without retraining the grounding model.
  • Skill-guided search is more sample-efficient: SA-MCTS acquires more skills than direct exploration or plain MCTS in the paper's setting, which matters when online interaction is expensive.
  • Abstraction level matters: removing meta-skills hurts more than removing execution skills, implying that a flat memory of episodes is a weaker guide than a hierarchical one.
  • Long-horizon multi-app tasks become feasible: on AndroidLH, a new 30-task benchmark, Mirage-1 reaches about 80% success, well above the closest baseline, suggesting that skill memory plus search is a workable recipe for multi-app workflows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct leakage check would compare the 200 GPT-4o-generated task strings and the offline trajectory apps against the four benchmarks' task templates; if match rates are high or if removing matched skills erases the gains, the contribution is closer to few-shot memorization than to open-ended generalization.
  • Because skill creation and sub-goal sampling both call GPT-4o, the approach's cost and bias could be reduced by distilling the abstraction steps into a smaller specialist model; the paper's reported per-step inference time (23.4 s on AndroidLH) suggests the hierarchical memory is what buys the speedup, not the planner.
  • The paper itself flags that the grounding model's interpretation of planner outputs is a critical bottleneck; controlling the planner while swapping grounding models (already partially done) would isolate how much of Mirage-1's gain comes from skills versus from the MLLM planner's sub-goal decomposition.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper introduces Mirage-1, a GUI agent that combines a Hierarchical Multimodal Skills (HMS) module with a Skill-Augmented Monte Carlo Tree Search (SA-MCTS) algorithm. HMS progressively abstracts offline trajectories into execution, core, and meta skills to support long-horizon planning. SA-MCTS uses these skills during online tree exploration, and successful exploration trajectories are used to update the skill library. The agent is evaluated on AndroidWorld, MobileMiniWob++, Mind2Web-Live, and a new long-horizon benchmark AndroidLH with four different grounding models. The paper reports large relative gains over baseline agents, supported by ablations that decompose the contribution of each skill level and of offline vs. online skill sources.

Significance. If the reported results are reliable, the paper makes a useful contribution to GUI agents: the hierarchical skill representation with three levels of abstraction is a plausible mechanism for improving long-horizon planning, and the SA-MCTS approach for online adaptation is well motivated. The cross-platform evaluation with several grounding models and the introduction of the AndroidLH benchmark are additional strengths. The ablations give insight into which components matter. However, the significance is substantially contingent on resolving the train/evaluation overlap concern described below; without evidence that the skill library is not contaminated with the evaluation tasks, the generalization claims are not yet established.

major comments (2)
  1. [§4.3, Appendix F.3, Appendix D] The 200 GPT-4o-generated "unseen" tasks used to expand HMS are never defined with respect to a reference set, and no overlap analysis is provided between these tasks (or the 1000 offline trajectories used for initialization) and the evaluation tasks on AndroidWorld, MobileMiniWob++, Mind2Web-Live, and AndroidLH. This omission is load-bearing for the central claim of generalization. AndroidLH was constructed by the authors "leveraging AndroidWorld's task template and evaluation mechanisms" (Appendix D), and the case study in Figure 4 presents a task that is structurally identical to the AddContactAndSms template in Table 5. Under these conditions, the reported 79% relative improvement on AndroidLH could be attributable to the agent having acquired the exact skills needed for the evaluation tasks during SA-MCTS exploration, rather than to generalizable knowledge. The authors should release the list of exploration tasks, quantify their similarity (e.g., via task-template or embedding similarity) to each evaluation benchmark, and ideally repeat the evaluation after removing any overlapping tasks from the exploration pool.
  2. [§1, §4.4, Table 1] The headline improvement numbers in the abstract and introduction (32%, 19%, 15%, 79%) are the maximum relative gains across the four grounding-model variants, not the results of a single system. For example, on AndroidWorld the largest gain is 32.3% for the OS-Atlas variant, while the UI-TARS variant shows 19.1%. The paper should state this explicitly in the abstract or at least in Section 1, and should present per-variant confidence intervals. Additionally, no multiple-seed or statistical significance testing is reported, which makes it hard to judge whether the differences are robust given the stochasticity of GPT-4o and MCTS. Since the central claim is that Mirage-1 "outperforms previous agents by" these amounts, the choice of the best variant for each metric should be transparent.
minor comments (6)
  1. [§1] The sentence "Mirage-1 achieves a success rate exceeding 79% on AndroidLH" is factually incorrect: Table 1 reports a success rate of 30.0 for the best Mirage-1 variant on AndroidLH. The intended meaning is presumably "outperforms previous agents by 79%"; please correct this misstatement.
  2. [§3.4, Algorithm 1, Appendix E] Concrete values for the hyperparameters c_exp, N (MCTS iterations), T (tree depth), S (number of sampled sub-goals), and the reflection threshold (q<5 in Section 3.2) are not provided anywhere in the main text or appendices. These settings are necessary for reproducing the experiments and for understanding the computational cost.
  3. [Table 1] The table headers ΔSR and ΔCR do not specify whether they denote absolute or relative changes. The main text describes these as relative gains (e.g., "32.3%"), but the table could be misread; please clarify the definitions in the caption or footnotes.
  4. [Figure 3] The caption does not explain the axes. The text says SA-MCTS "acquires more skills," but the axes appear to be "Skill Num" and "AndroidLH SR". Clarify what is plotted and how the number of skills is obtained, and add error bars if possible.
  5. [Appendix D] The full set of AndroidLH task instances (the actual names, numbers, and texts substituted into the templates in Table 5) is not provided. Releasing the complete task list would enable independent evaluation and help assess potential overlap with the exploration tasks.
  6. [Figure 5] There are typos in the caption: "Hierachrical" should be "Hierarchical," and "Mirage" should be "Mirage-1" in at least one place.

Circularity Check

1 steps flagged · score 6.0 of 10

AndroidLH success is measured after SA-MCTS expands HMS on the same 30 GPT-4o-generated tasks; the 79% headline lacks a disjointness check.

  1. fitted input called prediction [Section 4.6, Figure 3 caption; Section 4.5; Algorithm 1]
    "We compared three online exploration methods: direct exploration, MCTS, and SA-MCTS on 30 GPT-4o generated tasks. Results show that SA-MCTS acquires more skills than the other methods... AndroidLH includes 30 diverse tasks across multiple applications, generated via GPT-4o to mirror real-world application scenarios. Algorithm 1: 'Human evaluates whether success or not and adds successful trajectories to the replay buffer B' and 'Ki ← RefineSkillModule(Ki−1, B)'."

    The Figure 3 ablation uses exactly 30 GPT-4o-generated tasks and reports 'AndroidLH SR' on the same axis, tying the exploration pool to the AndroidLH test set. Algorithm 1 inserts successful exploration trajectories into HMS's replay buffer and refines the skill module from them, so the AndroidLH success rate reported for SA-MCTS is measured after the skill library has ingested successful solutions to those very tasks. The result is a retrieval or memorization score, not an independent forward prediction.

full rationale

Circularity is concentrated in the AndroidLH evaluation and the SA-MCTS ablation, not in the architecture as a whole. The HMS construction from AITW and Multimodal-Mind2Web is explicit, and AndroidWorld and MobileMiniWob++ are external benchmarks with no evidence of direct training on their test instances, so those results retain independent content. The circular step is Figure 3 and Section 4.6: the ablation measures AndroidLH success after using SA-MCTS to explore exactly 30 GPT-4o-generated tasks, the same size and generation procedure as AndroidLH, and Algorithm 1 adds successful exploration trajectories to the replay buffer that refines HMS. This makes the reported AndroidLH SR after SA-MCTS reduce to retrieving skills acquired from the same task set. The headline 79% also lacks an overlap check: the 200 exploration tasks are generated by GPT-4o from the same kind of app information used to construct AndroidLH from AndroidWorld templates, so the largest claimed gain may be partially driven by same-distribution skill acquisition rather than generalization. I do not count the external-benchmark gains as circular, and self-citations (e.g., Optimus-1) are not load-bearing, so the score is 6 rather than higher.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central claim rests on a skill library assembled by a black-box LLM, on an unverified distribution match between offline/exploration tasks and test tasks, on human-labeled replay success, and on several unreported hyperparameters. These are the costs the reader must accept.

free parameters (5)
  • c_exp (UCB1 exploration constant)
    Used in Eq. (7) of SA-MCTS; no value or tuning procedure is reported in the paper.
  • Reflection quality threshold q0 = 5
    Section 3.2: Decision Reflector regenerates an action when q is below 5; threshold is fixed by hand without sensitivity analysis.
  • Number of sampled sub-goals S in SA-MCTS
    Algorithm 1 samples S sub-goals per node; the value is not reported.
  • MCTS iterations N and tree depth T
    Algorithm 1 loops N times to depth T; these values are never specified.
  • Number of GPT-4o-generated exploration tasks = 200
    Section 4.3: 'We prompt GPT-4o to generate 200 unseen tasks'; the count is chosen ad hoc.
assumptions (5)
  • domain assumption GPT-4o reliably abstracts trajectories into execution, core, and meta skills with accurate generalizations.
    Used throughout Section 3.3 to build HMS; no validation of skill quality separate from downstream success.
  • domain assumption The offline trajectories (AITW for mobile, Multimodal-Mind2Web for web) are a representative source for skills that transfer to the online evaluation benchmarks.
    Section 4.3 uses these datasets to initialize HMS; if they are unrepresentative or overlap with test tasks, the reported gains change.
  • ad hoc to paper The 200 GPT-4o-generated exploration tasks used to expand HMS do not substantially overlap with the evaluation tasks on AndroidWorld, MobileMiniWob++, Mind2Web-Live, and AndroidLH.
    Section 4.3 and Appendix F.3 describe generating 200 unseen tasks; the paper never verifies disjointness from the test benchmarks, so gains may reflect memorization of the same skill distribution.
  • domain assumption The grounding models (OS-Atlas, UGround, Aria-UI, UI-TARS) produce accurate element coordinates from screenshots.
    Operator uses these models for all interactive actions (Section 3.2, Eq. 5); grounding errors directly propagate to task failure as acknowledged in Appendix A.
  • domain assumption Human evaluation of SA-MCTS trajectory success in the replay buffer is correct and consistent.
    Algorithm 1 and Section 3.4 rely on human labels to decide which trajectories refine HMS; no protocol or inter-annotator agreement reported.
invented entities (2)
  • Hierarchical Multimodal Skills (HMS) module
    purpose: Stores three-level skill abstractions (Execution, Core, Meta) to guide sub-goal planning and reflection.
    Its value is only measured through the bundled system (Mirage-1); no standalone evaluation or falsifiable handle outside the paper's benchmarks.
  • Skill-Augmented MCTS (SA-MCTS)
    purpose: MCTS variant that uses HMS skills to generate and evaluate sub-goals during online exploration.
    Same bundled validation; no separate analysis of the search algorithm aside from Figure 3 skill-count comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills." pith.science (2026). https://pith.science/paper/X7552FTI

@misc{pith2026250610387,
  author       = {Pith},
  title        = {Pith review of: Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X7552FTI}},
  note         = {Machine review of arXiv:2506.10387}
}
read the original abstract

Recent efforts to leverage the Multi-modal Large Language Model (MLLM) as GUI agents have yielded promising outcomes. However, these agents still struggle with long-horizon tasks in online environments, primarily due to insufficient knowledge and the inherent gap between offline and online domains. In this paper, inspired by how humans generalize knowledge in open-ended environments, we propose a Hierarchical Multimodal Skills (HMS) module to tackle the issue of insufficient knowledge. It progressively abstracts trajectories into execution skills, core skills, and ultimately meta-skills, providing a hierarchical knowledge structure for long-horizon task planning. To bridge the domain gap, we propose the Skill-Augmented Monte Carlo Tree Search (SA-MCTS) algorithm, which efficiently leverages skills acquired in offline environments to reduce the action search space during online tree exploration. Building on HMS, we propose Mirage-1, a multimodal, cross-platform, plug-and-play GUI agent. To validate the performance of Mirage-1 in real-world long-horizon scenarios, we constructed a new benchmark, AndroidLH. Experimental results show that Mirage-1 outperforms previous agents by 32\%, 19\%, 15\%, and 79\% on AndroidWorld, MobileMiniWob++, Mind2Web-Live, and AndroidLH, respectively. Project page: https://cybertronagent.github.io/Mirage-1.github.io/

Figures

Figures reproduced from arXiv: 2506.10387 by the authors.

Figure 1
Figure 1. Compared with existing GUI agents, Mirage￾1 improves performance by: (1) Hierarchical Multi￾modal Skills (HMS) module that enhances the agent’s capability in long-horizon task planning, and (2) Skill￾Augmented MCTS algorithm that leverages the HMS to search feasible trajectories in online environments. The GUI agent that can automate operations on graphical user interfaces, such as mobile devices, represents one of … view at source ↗
Figure 2
Figure 2. The Mirage-1 framework comprises a Hierarchical Planner, an Operator, a Decision [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. We compared three online exploration meth￾ods: direct exploration, MCTS, and SA-MCTS on 30 GPT-4o generated tasks. Results show that SA-MCTS acquires more skills than the other methods, demon￾strating its superior effectiveness in online exploration. Sources of HMS. As shown in Table 3a, our ab￾lation experiments demonstrate the relative con￾tributions of different source skills. Removing of￾fline skills led to the … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Case study example where Mirage-1 completes a long-horizon task. First, Mirage-1 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Case study example where Mirage completes a long-horizon task. First, Mirage retrieval [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents

    cs.AI 2026-07 conditional novelty 5.5 of 10

    A real-device-centric foundation GUI agent with hybrid GUI+CLI batched actions, AutoResearch data flywheel, online RL, and a proactive harness reaches SOTA mobile and competitive desktop/web scores.

  2. Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinement

    cs.AI 2025-08 conditional novelty 4.0 of 10

    A GUI agent that trims UI input with a recommendation module and asks users when decisions are ambiguous reports state-of-the-art success rates, though the interaction module is not benchmarked.

Reference graph

Works this paper leans on

66 extracted references · 25 canonical work pages · cited by 2 Pith papers

  1. [1]

    Agent s: An open agentic framework that uses computers like a human.arXiv preprint arXiv:2410.08164, 2024

    Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like a human.arXiv preprint arXiv:2410.08164, 2024

    arXiv 2024

  2. [2]

    Lion: Empowering multimodal large language model with dual-level visual knowledge

    Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. Lion: Empowering multimodal large language model with dual-level visual knowledge. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26540–26550, 2024

    2024

  3. [3]

    Seeclick: Harnessing gui grounding for advanced visual gui agents.arXiv preprint arXiv:2401.10935, 2024

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents.arXiv preprint arXiv:2401.10935, 2024

    arXiv 2024

  4. [4]

    Mind2web: Towards a generalist agent for the web.Advances in Neural Information Processing Systems, 36, 2024

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web.Advances in Neural Information Processing Systems, 36, 2024

    2024

  5. [5]

    Iris: Breaking gui complexity with adaptive focus and self-refining.arXiv preprint arXiv:2412.10342, 2024

    Zhiqi Ge, Juncheng Li, Xinglei Pang, Minghe Gao, Kaihang Pan, Wang Lin, Hao Fei, Wenqiao Zhang, Siliang Tang, and Yueting Zhuang. Iris: Breaking gui complexity with adaptive focus and self-refining.arXiv preprint arXiv:2412.10342, 2024

    arXiv 2024

  6. [6]

    Navigating the digital world as humans do: Universal visual grounding for gui agents.arXiv preprint arXiv:2410.05243, 2024

    Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents.arXiv preprint arXiv:2410.05243, 2024

    arXiv 2024

  7. [7]

    Studying large language model generalization with influence functions.arXiv preprint arXiv:2308.03296, 2023

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions.arXiv preprint arXiv:2308.03296, 2023

    arXiv 2023

  8. [8]

    A real-world webagent with planning, long context understanding, and program synthesis.arXiv preprint arXiv:2307.12856, 2023

    Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis.arXiv preprint arXiv:2307.12856, 2023

    arXiv 2023

Show all 66 references
  1. [9]

    Reasoning with language model is planning with world model

    Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. Reasoning with language model is planning with world model. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Proce...

    2023 doi

  2. [10]

    WebVoyager: Building an end-to-end web agent with large multimodal models

    Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebVoyager: Building an end-to-end web agent with large multimodal models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors,Proceedings of the 62nd Annual Meeting of ...

    2024 doi

  3. [11]

    The dawn of gui agent: A preliminary case study with claude 3.5 computer use.arXiv preprint arXiv:2411.10323, 2024

    Siyuan Hu, Mingyu Ouyang, Difei Gao, and Mike Zheng Shou. The dawn of gui agent: A preliminary case study with claude 3.5 computer use.arXiv preprint arXiv:2411.10323, 2024

    2024 arXiv

  4. [12]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  5. [13]

    Autowebglm: A large language model- based web navigating agent

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: A large language model- based web navigating agent. InProceedings of the 30th ACM SIGKDD Conference on Knowl- edge Discovery and Da...

    2024

  6. [14]

    Star: Learning diverse robot skill abstractions through rotation-augmented vector quantization.arXiv preprint arXiv:2506.03863, 2025

    Hao Li, Qi Lv, Rui Shao, Xiang Deng, Yinchuan Li, Jianye Hao, and Liqiang Nie. Star: Learning diverse robot skill abstractions through rotation-augmented vector quantization.arXiv preprint arXiv:2506.03863, 2025. 10

    2025 arXiv

  7. [15]

    Lion-fs: Fast & slow video- language thinker as online video assistant

    Wei Li, Bing Hu, Rui Shao, Leyang Shen, and Liqiang Nie. Lion-fs: Fast & slow video- language thinker as online video assistant. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 3240–3251, 2025

    2025

  8. [16]

    Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks.arXiv preprint arXiv:2408.03615, 2024

    Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-1: Hybrid multimodal memory empowered agents excel in long-horizon tasks.arXiv preprint arXiv:2408.03615, 2024

    2024 arXiv

  9. [17]

    Optimus-2: Multimodal minecraft agent with goal-observation-action conditioned policy

    Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-2: Multimodal minecraft agent with goal-observation-action conditioned policy. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 9039–9049, 2025

    2025

  10. [18]

    Showui: One vision-language-action model for generalist gui agent

    Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-action model for generalist gui agent. InNeurIPS 2024 Workshop on Open-World Agents, 2024

    2024

  11. [19]

    Reinforcement learning on web interfaces using workflow-guided exploration

    Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. InInternational Conference on Learning Representations (ICLR), 2018. URLhttps://arxiv.org/abs/1802.08802

    2018 arXiv

  12. [20]

    Omniparser for pure vision based gui agent.arXiv preprint arXiv:2408.00203, 2024

    Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent.arXiv preprint arXiv:2408.00203, 2024

    2024 arXiv

  13. [21]

    Webcanvas: Benchmarking web agents in online environments.arXiv preprint arXiv:2406.12373, 2024

    Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, et al. Webcanvas: Benchmarking web agents in online environments.arXiv preprint arXiv:2406.12373, 2024

    2024 arXiv

  14. [22]

    Agent q: Advanced reasoning and learning for autonomous ai agents.arXiv preprint arXiv:2408.07199, 2024

    Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents.arXiv preprint arXiv:2408.07199, 2024

    2024 arXiv

  15. [23]

    Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  16. [24]

    Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573, 2024

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents.arXiv preprint arXiv:2405.14573, 2024

    2024 arXiv

  17. [25]

    An- droidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36, 2024

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. An- droidinthewild: A large-scale dataset for android device control.Advances in Neural Information Processing Systems, 36, 2024

    2024

  18. [26]

    Mome: Mixture of multimodal experts for generalist multimodal large language models.arXiv preprint arXiv:2407.12709, 2024

    Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multimodal experts for generalist multimodal large language models.arXiv preprint arXiv:2407.12709, 2024

    2024 arXiv

  19. [27]

    Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning.Advances in Neural Information Processing Systems, 36, 2024

    2024

  20. [28]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  21. [29]

    Androidenv: A reinforcement learning platform for android.arXiv preprint arXiv:2105.13231, 2021

    Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android.arXiv preprint arXiv:2105.13231, 2021

    2021 arXiv

  22. [30]

    Neural representation of abstract task structure during generalization.eLife, 10:e63226, mar 2021

    Avinash R Vaidya, Henry M Jones, Johanny Castillo, and David Badre. Neural representation of abstract task structure during generalization.eLife, 10:e63226, mar 2021. ISSN 2050-084X. doi: 10.7554/eLife.63226. URLhttps://doi.org/10.7554/eLife.63226. 11

    2021 doi

  23. [31]

    Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration.arXiv preprint arXiv:2406.01014, 2024

    Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration.arXiv preprint arXiv:2406.01014, 2024

    2024 arXiv

  24. [32]

    Mobile-agent: Autonomous multi-modal mobile device agent with visual perception

    Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158, 2024

    2024 arXiv

  25. [33]

    Mobile-agent-e: Self-evolving mobile assistant for complex tasks.arXiv preprint arXiv:2501.11733, 2025

    Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. Mobile-agent-e: Self-evolving mobile assistant for complex tasks.arXiv preprint arXiv:2501.11733, 2025

    2025 arXiv

  26. [34]

    Agent workflow memory

    Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint arXiv:2409.07429, 2024

    2024 arXiv

  27. [35]

    Os-atlas: A foundation action model for generalist gui agents.arXiv preprint arXiv:2410.23218, 2024

    Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents.arXiv preprint arXiv:2410.23218, 2024

    arXiv

  28. [36]

    Gui-explorer: Autonomous exploration and mining of transition-aware knowledge for gui agent.arXiv preprint arXiv:2505.16827, 2025

    Bin Xie, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Jie Liu, Min Zhang, and Liqiang Nie. Gui-explorer: Autonomous exploration and mining of transition-aware knowledge for gui agent.arXiv preprint arXiv:2505.16827, 2025

    2025 arXiv

  29. [37]

    Aguvis: Unified pure vision agents for autonomous gui interaction

    Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454, 2024

    2024 arXiv

  30. [38]

    Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.arXiv preprint arXiv:2310.11441, 2023

    2023 arXiv

  31. [39]

    Aria-ui: Visual grounding for gui instructions.arXiv preprint arXiv:2412.16256, 2024

    Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions.arXiv preprint arXiv:2412.16256, 2024

    2024 arXiv

  32. [40]

    Appagent: Multimodal agents as smartphone users.arXiv preprint arXiv:2312.13771, 2023

    Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users.arXiv preprint arXiv:2312.13771, 2023

    2023 arXiv

  33. [41]

    React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  34. [42]

    Large language model-brained gui agents: A survey

    Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Qingwei Lin, Saravan Rajmohan, et al. Large language model-brained gui agents: A survey. arXiv preprint arXiv:2411.18279, 2024

    2024 arXiv

  35. [43]

    On the out-of-distribution generalization of multimodal large language models.arXiv preprint arXiv:2402.06599, 2024

    Xingxuan Zhang, Jiansheng Li, Wenjing Chu, Junjia Hai, Renzhe Xu, Yuqing Yang, Shikai Guan, Jiazheng Xu, and Peng Cui. On the out-of-distribution generalization of multimodal large language models.arXiv preprint arXiv:2402.06599, 2024

    2024 arXiv

  36. [44]

    Attacking vision-language computer agents via pop-ups

    Yanzhe Zhang, Tao Yu, and Diyi Yang. Attacking vision-language computer agents via pop-ups. arXiv preprint arXiv:2411.02391, 2024

    2024 arXiv

  37. [45]

    A survey on the memory mechanism of large language model based agents

    Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404.13501, 2024

    2024 arXiv

  38. [46]

    Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614, 2024

    Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded.arXiv preprint arXiv:2401.01614, 2024

    2024 arXiv

  39. [47]

    Synapse: Trajectory-as-exemplar prompting with memory for computer control

    Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control. InThe Twelfth International Conference on Learning Representations, 2023. 12 A Limitations Multimodal large language models (MLLMs) possess inherent...

    2023

  40. [48]

    Input basic information, Name is Noa Fernandez, phone number is +13174092855 Tap Name: ContactManagement Description: The ability to add, edit, and manage contacts within a contact

    Open Contact App 2. Input basic information, Name is Noa Fernandez, phone number is +13174092855 Tap Name: ContactManagement Description: The ability to add, edit, and manage contacts within a contact. Name: SMSMessageManagement Description: The ability to send a new text mess...

  41. [49]

    Create Note named 3583_sure_guitar.md

  42. [50]

    The squeaky wheel gets the grease

    input: The squeaky wheel gets the grease. 5. send message to +13174092855 with “The squeaky wheel gets the grease” Core Skill: add_contact Core Skill: send_text_message Figure 5: Case study example where Mirage completes a long-horizon task. First, Mirage retrieval Meta Skills...

  43. [51]

    Analyze the provided task and the sequence of actions to understand the goal

  44. [52]

    Reference existing skill functions as templates for creating new ones

  45. [53]

    Generate a skill function, ensuring its actions and arguments match the provided task and actions

  46. [54]

    # Output Format - Provide the new skill function in Python format

    Ensure the skill function's arguments are reusable for similar tasks. # Output Format - Provide the new skill function in Python format. - Ensure the skill function includes a docstring describing its parameters and purpose. # Examples {example1} {example2} {example3} Your Tur...

  47. [55]

    Analyze the Task: Start by considering the main actions or objectives involved in the given task

  48. [56]

    Review Existing Skills: Go through each skill description to find potential matches or overlaps with the task requirements

  49. [57]

    Decide on a Category: Determine whether the task can be fully categorized under an existing skill, needs the combination of several skills, or demands the creation of a new skill

  50. [58]

    New Skill

    Define and Document: Clearly write out the reasoning and final categorization, including new skill descriptions if necessary. ### Output Format Reason: Provide the thought process and analysis that led to the classification. Category: Specify the appropriate existing skill or ...

  51. [59]

    abcs.txt

    NoteManager.create_note_in_markor("abcs.txt", "")

  52. [60]

    Perform paste operation in the note

  53. [61]

    Think carefully about the task and skill functions to generate the most accurate and feasible plans

    Check the note and save the note. Think carefully about the task and skill functions to generate the most accurate and feasible plans. Input: Task: {task} {context} Output must be in the following format: Reason: <reason> Plans:

  54. [62]

    You are an instructor in the Android Phone scenario, guiding the Agent to better complete users' tasks

    plan2 Listing 4: Prompt for Planning with HMS. You are an instructor in the Android Phone scenario, guiding the Agent to better complete users' tasks. Given the user's task, the current screenshot of the Android Phone, and an action that the Agent intends to take, assess wheth...

  55. [63]

    Goal Alignment: Does the action help achieve the user's goal?

  56. [64]

    How well the action brings the user closer to their goal

  57. [65]

    Likelihood of Success: How likely is it that the action will succeed based on the current context and UI state

  58. [66]

    caption": ...,

    Efficiency: How efficient is the action in terms of user effort and time required. just output the score as a number between 0-10. Output format: {{"caption": ..., "reason": ..., "state_change": ..., "score": ...}} Listing 5: Prompt for Reflection with Execution Skill. 19 Name...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.