Pith. sign in

REVIEW 9 cited by

ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.13108 v2 pith:B32ZJSBW submitted 2023-12-20 cs.CV

Difei Gao , Lei Ji , Zechen Bai , Mingyu Ouyang , Peiran Li , Dongxing Mao , Qinchen Wu , Weichen Zhang
show 5 more authors
Peiyi Wang Xiangwu Guo Hengxu Wang Luowei Zhou Mike Zheng Shou
This is my paper · ORCID
classification cs.CV
keywords tasksassistguiautomationbenchmarkexistinggraphicalinterfacemechanism
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Graphical User Interface (GUI) automation holds significant promise for assisting users with complex tasks, thereby boosting human productivity. Existing works leveraging Large Language Model (LLM) or LLM-based AI agents have shown capabilities in automating tasks on Android and Web platforms. However, these tasks are primarily aimed at simple device usage and entertainment operations. This paper presents a novel benchmark, AssistGUI, to evaluate whether models are capable of manipulating the mouse and keyboard on the Windows platform in response to user-requested tasks. We carefully collected a set of 100 tasks from nine widely-used software applications, such as, After Effects and MS Word, each accompanied by the necessary project files for better evaluation. Moreover, we propose an advanced Actor-Critic Embodied Agent framework, which incorporates a sophisticated GUI parser driven by an LLM-agent and an enhanced reasoning mechanism adept at handling lengthy procedural tasks. Our experimental results reveal that our GUI Parser and Reasoning mechanism outshine existing methods in performance. Nevertheless, the potential remains substantial, with the best model attaining only a 46% success rate on our benchmark. We conclude with a thorough analysis of the current methods' limitations, setting the stage for future breakthroughs in this domain.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Comprehensive Study of Implementation Bugs in Multi-modal Agents

    cs.SE 2026-07 accept novelty 7.0 of 10

    First systematic taxonomy of 158 multi-modal agent bugs plus a runtime analyzer that recovers most open issues and surfaces 31 new ones.

  2. R-VLM: Region-Aware Vision Language Model for Precise GUI Grounding

    cs.CV 2025-07 conditional novelty 7.0 of 10

    R-VLM improves GUI grounding by combining two-stage zoom-in proposals with an IoU-weighted training loss, raising accuracy by up to 13 absolute points over SeeClick.

  3. LogiDroid: Individual Functional Test Generation via Business Logic Extraction and Adaptation

    cs.SE 2026-02 conditional novelty 6.0 of 10

    LogiDroid generates functional Android test cases with verification assertions by retrieving similar test cases, fusing their business logic, and adapting it to the target app's real-time GUI state.

  4. MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents

    cs.CV 2025-07 conditional novelty 6.0 of 10

    MMBench-GUI provides a multi-platform, four-level benchmark and an efficiency-aware metric, and its experiments indicate visual grounding is the main bottleneck for current GUI agents.

  5. Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation

    cs.LG 2025-07 conditional novelty 6.0 of 10

    Screen2AX generates hierarchical macOS accessibility metadata from a screenshot and reports improved GPT-4 UI task success compared with native accessibility and OmniParser V2.

  6. PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalization

    cs.CL 2025-06 conditional novelty 6.0 of 10

    PersonaFeedback provides a human-labeled benchmark showing current LLMs, including strong reasoners, score only about 65-70 percent on hard personalization choices, and explicit persona information helps more than retrieval.

  7. OWMM-Agent: Open World Mobile Manipulation With Multi-modal Agentic Data Synthesis

    cs.RO 2025-06 conditional novelty 5.0 of 10

    A vision-language model fine-tuned on 572K synthetic simulation examples improves open-world mobile manipulation action decisions and object grounding over GPT-4o, with 21.9% full-task success in simulation and 90% ac...

  8. TextAtari: 100K Frames Game Playing with Language Agents

    cs.CL 2025-06 conditional novelty 5.0 of 10

    TextAtari is a text-based Atari benchmark for language agents; 7-8B LLMs stay below 10% of human scores in over 90% of tested conditions, and knowledge injection helps more than chain-of-thought.

  9. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools