Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that GUI agents systematically fail on real-world GUI anomalies, and that a new dataset with 200 abnormal tasks is the first to expose this gap.

desk verdict Useful benchmark and a real pipeline, but the headline claim that all five agents degrade substantially on anomalies is not supported by Table 4—only the two GUI-specialized agents show meaningful drops. read the letter →

arxiv 2506.14477 v1 pith:QARLB3NX submitted 2025-06-17 cs.AI

Jingqi Yang , Zhilong Song , Jiawei Chen , Mingli Song , Sheng Zhou , linjun sun , Xiaogang Ouyang , Chun Chen
show 1 more author
Can Wang
This is my paper · ORCID
classification cs.AI
keywords GUIagentdatasetrobustnessanomalymultimodallargelanguagemodelbenchmarkwebautomationRevAct
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that GUI agents are evaluated almost entirely in idealized conditions and therefore overstate real-world readiness. To close the gap, it introduces GUI-Robust, a dataset of 5,318 annotated tasks collected from 392 sites and desktop applications, of which 200 tasks contain one of seven common anomalies such as ad pop-ups, captcha screens, and network failures. The authors' central finding is that every GUI agent they test—two GUI-specific agents and three general multimodal models—degrades sharply on these abnormal tasks, with the best agent's task success rate falling from 21.61% to 4.73%. The paper also proposes RevAct, a reversed collection pipeline that records natural user traces and generates step and task descriptions with a multimodal LLM, cutting annotation time by a factor of over 19. If correct, this means robustness to everyday GUI disruptions is a distinct capability that current benchmarks do not measure.

What carries the argument

The load-bearing mechanism is the paired dataset and evaluation protocol: 200 abnormal tasks drawn from seven empirically observed anomaly categories, each annotated as a canonical action sequence with element coordinates, screenshots, and step descriptions. The collection pipeline RevAct carries the scalability claim: RPA records volunteer behavior, a YOLOv8 detector and OCR infer element type, location, and action content, and Qwen2.5-VL generates step and task descriptions that human reviewers only correct. The evaluation compares action accuracy, element location accuracy, and full-task success rate under normal versus abnormal conditions.

What would settle it

Independently re-annotate the 200 abnormal tasks with, say, three human experts per task, and compare their agreed action types, contents, and element coordinates against the released ground truth. If agreement is markedly low on actions such as wait and human, or if models' failure rates change substantially after label corrections, the reported degradation would reflect annotation noise rather than agent fragility.

Watch

Extended reading notes

Core claim

The central claim is that GUI-Robust is the first dataset built specifically to test GUI agents against real-world anomalies, and that using it reveals a systematic robustness gap: all five evaluated models, including dedicated GUI agents such as UI-TARS and CogAgent and general models such as GPT-4o, perform substantially worse on abnormal tasks than on standard ones. General models tend to notice anomalies but cannot locate the relevant UI elements, while GUI-specific agents localize well under normal conditions but often fail to detect or adapt to unexpected interruptions, sometimes repeating actions into infinite loops. The dataset's design choices—two anomaly-specific actions, wait and human, plus per-step ground-truth action and element annotations—are what make this degradation measurable.

Load-bearing premise

The entire robustness finding rests on the 200 abnormal tasks and their ground-truth action labels being correct, but the paper does not describe a protocol for identifying abnormal tasks and derives labels from a detection model plus OCR with only limited human review.

Editorial extensions

If this is right

  • Robustness to everyday GUI disruptions is a separate capability from element grounding and task completion, and it is not captured by existing idealized benchmarks.
  • General multimodal models can detect an anomaly but cannot reliably locate or interact with the right UI element, so improving their grounding is as important as improving their anomaly awareness.
  • GUI-specific agents that localize well in normal conditions often fail to recognize unexpected states, so they need explicit mechanisms to detect and recover from out-of-distribution screens.
  • The wait and human action types give agents a safe fallback for unresolvable anomalies, which could reduce hazardous behaviors such as repeated credential entry or accidental navigation.
  • The reported 19-fold reduction in annotation effort means large-scale robustness datasets can be built from natural user traces rather than manual task design and demonstration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the ground-truth labels for the abnormal subset are noisy, the measured degradation could be partly an artifact of label mismatch; a human re-annotation study would settle how much of the drop is genuine fragility.
  • The seven anomaly categories are static, but real deployments combine them, so extending the taxonomy to compound failures (e.g., login page plus network loss) is a natural next step the paper leaves open.
  • The wait and human actions encode a design assumption that agents should pause or escalate rather than continue; a dynamic evaluation environment could test whether agents choose those actions at the right moments.
  • Because RevAct depends on the detection model's accuracy, improving YOLOv8 or replacing it with a stronger GUI parser would likely raise label quality more than any other part of the pipeline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces GUI-Robust, a dataset of 5,318 annotated GUI task trajectories collected from 392 websites and desktop applications, including 200 abnormal tasks spanning seven anomaly types. It also proposes RevAct, a semi-automated data-collection pipeline that records natural user actions via an RPA tool, detects elements with YOLOv8, extracts text with OCR, and generates step and task descriptions with Qwen2.5-VL, followed by human revision. The authors report a roughly 19x reduction in annotation time and evaluate three general-purpose MLLMs and two GUI-specific agents on element grounding and full-task completion, claiming that all models degrade substantially under abnormal scenarios.

Significance. If the claims are supported, GUI-Robust would be a useful community resource: it is among the first benchmarks explicitly targeting real-world GUI anomalies, offers broad task diversity with cross-scenario and bilingual coverage, and ships public code and data. The RevAct pipeline is also a plausible route to cheaper dataset construction. However, the headline empirical claim of substantial degradation across all models is not yet adequately supported by the reported evidence: three of the five models are near floor in both conditions, and the paper provides no statistical tests or matched subsets. These issues are addressable, and the dataset and pipeline contributions remain valuable, so the manuscript merits a major revision rather than rejection.

major comments (4)
  1. [Section 5.1, Table 4] The abstract and Section 5.2 state that all evaluated models experience substantial performance degradation in abnormal scenarios, but Table 4 shows floor effects: Gemini 2.5-Flash-Preview achieves 0% Task SR in both conditions, GPT-4o falls from 0.33% (1/302) to 0% (0/197), and Qwen2.5-VL-72B falls from 1.32% to 0%. With these counts, the observed differences for the three general-purpose models are not statistically meaningful, and the paper reports no error bars or significance tests; NeurIPS checklist item 7 explicitly concedes that no variance analysis is provided. The claim should either be restricted to the two GUI-specific agents (UI-TARS 21.61% to 4.73%, CogAgent 14.38% to 2.14%) or be supported by bootstrap confidence intervals and a comparison on matched normal and abnormal subsets, which currently differ in size (302 vs 197) and are not matched for trajectory length, domain, or task difficulty.
  2. [Section 3.3, Section 4.2] The paper states that the seven anomaly categories emerged from analyzing 5,318 sessions, but it does not describe the protocol used to identify, classify, or sample abnormal tasks, nor does it report inter-annotator agreement or any validation of the anomaly labels. Action labels are produced by a YOLOv8 mapping plus OCR, with human review limited to step and task descriptions, so the abnormal-task labels and the ground-truth actions used to compute the degradation results are not independently verified. Additionally, the dataset is described as containing 200 abnormal tasks (Table 2) while Section 5.1 evaluates 197; the discrepancy and the selection procedure need to be explained.
  3. [Section 5.1, Section 6] Task Success Rate is defined as exact match of all actions and coordinates against a single canonical trajectory, and the paper later acknowledges that the dataset adopts a static evaluation paradigm with one canonical execution path. The text does not clarify whether a model that makes an early wrong prediction is shown the ground-truth next screenshot (teacher forcing) or its own erroneous state; this distinction is essential for interpreting the metric. Under exact-match, teacher-forced evaluation, SR is an extremely harsh criterion, and near-zero values for three models do not discriminate between anomaly-handling ability and general task-completion failure. The authors should report additional metrics such as anomaly detection accuracy, correctness of the expected wait/human action, and partial trajectory success, and should specify the interaction protocol precisely.
  4. [Section 4.3] The claimed 'over 19x' annotation-time reduction compares the full traditional pipeline (150 minutes for 100 tasks by 10 annotators) against only the RevAct review-and-revise stage (7.8 minutes), excluding the time volunteers spend recording interactions, RPA capture time, YOLOv8/OCR inference, and Qwen2.5-VL generation time. To support the efficiency claim, the end-to-end cost of RevAct, including the omitted stages, must be reported; otherwise the comparison is not fair and the factor-of-19 figure is misleading.
minor comments (6)
  1. [Throughout] There are several typos and formatting issues: 'Ralated Work' in the Section 2 heading, 'Gemeini2.5-Flash-Preview' in Table 3, 'pinepline' and 'mannual' in Section 4.2, 'Compeletion' in the Table 4 title, and 'a the website or application' in Section 3.2. These should be corrected.
  2. [Section 5.1] The text says '302 Standard Tasks (randomly selected from the 4,925-task subset)', but the dataset totals in Table 2 imply 4,414 + 503 + 201 = 5,118 normal tasks; the number 4,925 is unexplained and should be reconciled with the table.
  3. [Section 4.3 and Introduction] The introduction reports 'over 71% accuracy in automatic task generation', while Section 4.3 reports multiple figures (78.63% step and 72.57% task accuracy for Chinese; 71.71% step and 86.67% task accuracy for English). Please clarify which of these corresponds to the 71% figure and how it is aggregated across languages and task types.
  4. [Section 7] The conclusion says 'we demonstrate that existing GUI-specific agents struggle in failure-prone environments,' which is more consistent with the data than the abstract's 'all models' claim; the wording in the abstract, introduction, and conclusion should be aligned.
  5. [NeurIPS Checklist item 14] The checklist answers 'No' to the question about crowdsourcing, but the dataset construction involved volunteers and human annotators in the RevAct pipeline and the manual baseline. The paper should provide at least a brief description of instructions, consent, and compensation (or state why such details are not applicable) for those participants.
  6. [Appendix D and Table 5] The action space defines wait as having no content field, but the exact-match Task Success metric requires matching all actions and coordinates; please specify how an empty content field is matched in the evaluation code and how the wait action is compared when a model omits content.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the robustness evaluation is an independent measurement against externally captured user trajectories, and no fitted parameter is renamed as a prediction.

full rationale

The paper's central chain is empirical dataset construction, not a derivation. Ground-truth trajectories are captured from real user interactions via an RPA tool (Section 4.2, Step 1), action labels come from a YOLOv8-based element mapping and OCR (Step 2), and step/task descriptions are generated by Qwen2.5-VL and then human-reviewed (Steps 3 and 4). No parameter is fitted to the evaluated models' outputs, and no reported quantity is defined in terms of the quantity it claims to measure. The robustness claim is an external measurement: five models are run against the released trajectories and scored by exact-match success. The paper's own checklist item 7 concedes that no error bars or significance tests are provided, and Table 4 shows floor effects for three models, but these are statistical-support and validity concerns about a benchmark claim, not circular reasoning. The only author-affiliated asset is the Indeed-Intelligence RPA recorder, which is a data-capture tool rather than a load-bearing cited theorem, and the pipeline's outputs are independently assessed by human annotators. Therefore no circular step can be exhibited from the paper's text, and the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 2 invented entities

The ledger is mostly design choices rather than fitted numbers. There are no mathematical free parameters; the choices of 200 abnormal tasks and 7 categories shape the benchmark and are not justified by a sampling protocol. The key assumptions are about data validity, evaluation validity, and representativeness of anomalies.

free parameters (2)
  • Abnormal task count = 200
    The paper does not state how 200 abnormal tasks were sampled from the 5,318 sessions; the count is a design choice that shapes the benchmark difficulty.
  • Anomaly category set = 7 categories
    The seven anomaly types were identified from collected sessions without a documented protocol; the set defines the scope of the robustness claim.
assumptions (4)
  • domain assumption Ground-truth action sequences and element coordinates recorded by RPA are accurate.
    Section 4.2: RPA captures screenshots and coordinates; YOLOv8 maps element types to actions. Human review is stated to cover step/task descriptions only, so action labels are taken on faith.
  • domain assumption Exact action and coordinate match against a single canonical trajectory is a valid measure of task success.
    Section 5.1 defines Task SR as all actions and coordinates matching ground truth; this assumes one correct way to complete each task and ignores valid alternative trajectories.
  • domain assumption The seven identified anomaly types are representative of real-world GUI interruptions.
    Section 3.3 states these emerged organically from sessions, but no sampling or labeling protocol is given, so representativeness is assumed.
  • domain assumption Task descriptions generated by Qwen2.5-VL and revised by humans faithfully capture user intent.
    Section 4.2: MLLM-generated descriptions are reviewed by annotators; the agreement with original user intent is not directly measured.
invented entities (2)
  • wait action type
    purpose: Lets an agent signal that the GUI is loading and it is pausing before proceeding.
    Introduced in the GUI-Robust action space (Table 5); no external validation that this action improves real-world performance.
  • human action type
    purpose: Lets an agent report an unresolvable anomaly (login, captcha, network loss) for manual intervention.
    Introduced as part of the benchmark action space; its use is scored against ground-truth labels rather than validated in live deployments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies." pith.science (2026). https://pith.science/paper/QARLB3NX

@misc{pith2026250614477,
  author       = {Pith},
  title        = {Pith review of: GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QARLB3NX}},
  note         = {Machine review of arXiv:2506.14477}
}
read the original abstract

The development of high-quality datasets is crucial for benchmarking and advancing research in Graphical User Interface (GUI) agents. Despite their importance, existing datasets are often constructed under idealized conditions, overlooking the diverse anomalies frequently encountered in real-world deployments. To address this limitation, we introduce GUI-Robust, a novel dataset designed for comprehensive GUI agent evaluation, explicitly incorporating seven common types of anomalies observed in everyday GUI interactions. Furthermore, we propose a semi-automated dataset construction paradigm that collects user action sequences from natural interactions via RPA tools and then generate corresponding step and task descriptions for these actions with the assistance of MLLMs. This paradigm significantly reduces annotation time cost by a factor of over 19 times. Finally, we assess state-of-the-art GUI agents using the GUI-Robust dataset, revealing their substantial performance degradation in abnormal scenarios. We anticipate that our work will highlight the importance of robustness in GUI agents and inspires more future research in this direction. The dataset and code are available at https://github.com/chessbean1/GUI-Robust..

Figures

Figures reproduced from arXiv: 2506.14477 by the authors.

Figure 1
Figure 1. Comparison between (a) traditional manual data collection and (b) our proposed RevAct [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the RevAct pipeline for semi-automated data collection. Human interaction [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. An example task from GUI-Robust: Search for the best-selling comic book on Amazon. [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Examples of abnormal GUI scenarios in the GUI-Robust dataset. (a) cookie pop-up, (b) [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. GUI-Perturbed: Domain Randomization Reveals Systematic Brittleness in GUI Grounding Models

    cs.LG 2026-04 conditional novelty 7.0 of 10

    GUI-Perturbed shows that GUI grounding models suffer systematic accuracy collapse under relational instructions and visual changes such as 70% zoom, with even augmented fine-tuning worsening results.

Reference graph

Works this paper leans on

32 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

  2. [2]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    arXiv 2025

  3. [3]

    Windows agent arena: Evaluating multi-modal os agents at scale

    Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale. September 2024

    2024

  4. [4]

    Webvln: Vision-and-language navigation on websites

    Qi Chen, Dileepa Pitawela, Chongyang Zhao, Gengze Zhou, Hsiang-Ting Chen, and Qi Wu. Webvln: Vision-and-language navigation on websites. InProceedings of the AAAI Conference on Artificial Intelligence, 2024

    work page 2024

  5. [5]

    SeeClick: Harnessing GUI grounding for advanced visual GUI agents

    Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9313–9332, Bangkok, Thailand, August 2024. Association for Computational Li...

    work page 2024

  6. [6]

    Human-centered llm-agent user interface: A position paper, 2024

    Daniel Chin, Yuxuan Wang, and Gus Xia. Human-centered llm-agent user interface: A position paper, 2024. URLhttps://arxiv.org/abs/2405.13050

    arXiv 2024

  7. [7]

    Mind2web: Towards a generalist agent for the web, 2023

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023

    2023

  8. [8]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, ...

    2024

Show all 32 references
  1. [9]

    Rule-based exploratory testing of graphical user interfaces

    Theodore Hellmann and Frank Maurer. Rule-based exploratory testing of graphical user interfaces. pages 107 – 116, 09 2011. doi: 10.1109/AGILE.2011.23

    2011 doi

  2. [10]

    Cogagent: A visual language model for gui agents, 2023

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents, 2023

    2023

  3. [11]

    Mapping natural language instructions to mobile ui action sequences

    Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. InAnnual Conference of the Association for Computational Linguistics (ACL 2020), 2020. URLhttps://www.aclweb.org/anthology/ 2020.acl-main.729.pdf

    2020

  4. [12]

    Showui: One vision-language-action model for gui visual agent, 2024

    Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. Showui: One vision-language-action model for gui visual agent, 2024. URLhttps://arxiv.org/abs/2411.17465. 10

    2024 arXiv

  5. [13]

    Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices.arXiv preprint arXiv:2406.08451, 2024

    Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices.arXiv preprint arXiv:2406.08451, 2024

    2024 arXiv

  6. [14]

    Weblinx: Real-world website navigation with multi-turn dialogue, 2024

    Xing Han Lù, Zdenˇek Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue, 2024

    2024

  7. [15]

    Ahmed, Puneet Mathur, Seunghyun Yoon, Lina Yao, Branislav Kveton, Thien Huu Nguyen, Trung Bui, Tianyi Zhou, Ryan A

    Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, Xintong Li, Jing Shi, Hongjie Chen, Viet Dac Lai, Zhouhang Xie, Sungchul Kim, Ruiyi Zhang, Tong Yu, Mehrab Tanjim, Nesreen K. Ahmed, Puneet Mathur, Seunghyun Yoon, ...

    2024

  8. [16]

    OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex ...

    2024 arXiv

  9. [17]

    (181 additional authors not shown)

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, J...

    2024 arXiv

  10. [18]

    Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents.arXiv preprint arXiv:2501.12326, 2025

    2025 arXiv

  11. [19]

    Android in the wild: A large-scale dataset for android device control, 2023

    Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control, 2023. URL https://arxiv. org/abs/2307.10088. 11

    2023 arXiv

  12. [20]

    jrapture: A capture/replay tool for observation-based testing

    John Steven, Pravir Chandra, Bob Fleck, and Andy Podgurski. jrapture: A capture/replay tool for observation-based testing. InProceedings of the 2000 ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA ’00, page 158–167, New York, NY , USA,

    2000

  13. [21]

    Meta-gui: Towards multi-modal conversational agents on mobile gui.arXiv preprint arXiv:2205.11029, 2022

    Liangtai Sun, Xingyu Chen, Lu Chen, Tianle Dai, Zichen Zhu, and Kai Yu. Meta-gui: Towards multi-modal conversational agents on mobile gui.arXiv preprint arXiv:2205.11029, 2022

    2022 arXiv

  14. [22]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lil...

    2024 arXiv

  15. [23]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model’s p...

    2024 arXiv

  16. [24]

    Webwalker: Benchmarking llms in web traversal, 2025

    Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Deyu Zhou, Pengjun Xie, and Fei Huang. Webwalker: Benchmarking llms in web traversal, 2025. URL https://arxiv.org/abs/2501.07572

    2025 arXiv

  17. [25]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for op...

    2024

  18. [26]

    Search” or “Next

    Henry Hengyuan Zhao, Difei Gao, and Mike Zheng Shou. Worldgui: Dynamic testing for comprehensive desktop gui automation, 2025. URL https://arxiv.org/abs/2502.08047. A Dataset Format Each datapoint is stored as a single JSON file accompanied by a set of PNG screenshots, one for...

    2025 arXiv

  19. [28]

    human: Detected an ad pop-up blocking the process

    click/input (repeat previous action) Login Page human: Detected the login page, which needs to be completed manually Captcha Page human: Detected the captcha page, which needs to be completed manually Ad Pop-up 1. human: Detected an ad pop-up blocking the process

  20. [29]

    human: Detected a cookie pop-up blocking the process

    click (close the pop-up) Cookie Pop-up 1. human: Detected a cookie pop-up blocking the process

  21. [30]

    human: Detected that the page is loading

    click (accept cookies) Page Loading 1. human: Detected that the page is loading

  22. [31]

    ele_loc": {

    wait Network Disconnection human: Detected that the network is currently disconnected system-level disruptions such as login prompts (c, d), Captcha challenges (g–i), page loading delays (e), and network disconnections (f). Each of these anomalies can significantly interrupt t...

  23. [32]

    All data were collected via non-identifiable interaction traces from internal annotators performing routine tasks on public applications

    Institutional review board (IRB) approvals or equivalent for research with human subjects 23 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approv...

    2025

  24. [2000]

    ISBN 1581132662

    Association for Computing Machinery. ISBN 1581132662. doi: 10.1145/347324.348993. URLhttps://doi.org/10.1145/347324.348993

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.