REVIEW 19 cited by
GUI Agents with Foundation Models: A Comprehensive Survey
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent advances in foundation models, particularly Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), have facilitated the development of intelligent agents capable of performing complex tasks. By leveraging the ability of (M)LLMs to process and interpret Graphical User Interfaces (GUIs), these agents can autonomously execute user instructions, simulating human-like interactions such as clicking and typing. This survey consolidates recent research on (M)LLM-based GUI agents, highlighting key innovations in data resources, frameworks, and applications. We begin by reviewing representative datasets and benchmarks, followed by an overview of a generalized, unified framework that encapsulates the essential components of prior studies, supported by a detailed taxonomy. Additionally, we explore relevant commercial applications. Drawing insights from existing work, we identify key challenges and propose future research directions. We hope this survey will inspire further advancements in the field of (M)LLM-based GUI agents.
Forward citations
Cited by 19 Pith papers
-
ReGUIDE: Data Efficient GUI Grounding via Spatial Reasoning and Search
ReGUIDE reaches state-of-the-art GUI grounding accuracy using 0.2% of the usual training data by adding self-generated reasoning, spatial-consistency training, and test-time KDE coordinate search.
-
The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents
Gated Hindsight Distillation uses the next screenshot from a successful trajectory as privileged teacher information that corrects a student GUI agent's failed rollouts, improving Android task success over GRPO baselines.
-
GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
GUI-Lens improves GUI grounding by having a VLM iteratively select crops to zoom into, guided by OCR and UI detector references, and verifies its proposed clicks before outputting a coordinate.
-
SEE: Structure-aware Exploring \& Exploiting for Long-horizon GUI Agent Trajectory Synthesis
A graph-based pipeline automatically synthesizes long, multi-step mobile-app interaction trajectories, and fine-tuning GUI agents on them improves task success and cross-benchmark transfer.
-
GUI-AC: Enhancing Continual Learning in GUI Agents
GUI-AC stabilizes RFT for non-stationary GUI data by down-weighting noisy advantages and relaxing clipping bounds via a grounding certainty term.
-
Instruction Agent: Enhancing Agent with Expert Demonstration
With one human demonstration, the Instruction Agent extracts instructions and uses verifier/backtracker modules to execute them, reaching 60% success on 20 previously unsolved OSWorld tasks.
-
RoboBrain 2.0 Technical Report
RoboBrain 2.0, a 7B/32B embodied vision-language model built on Qwen2.5-VL, reports state-of-the-art or near-top scores on several spatial and temporal reasoning benchmarks for robotics.
-
DeepShop: A Benchmark for Deep Research Shopping Agents
DeepShop, a benchmark of 150 complex online shopping queries with fine-grained evaluation, shows that leading web agents and deep research systems achieve at most a 32% task success rate.
-
Robot Operation of Home Appliances by Reading User Manuals
A robot system that constructs a symbolic appliance model from a user manual and uses it to reliably execute natural language appliance operation tasks, outperforming direct VLM-based policies.
-
TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments
TransBench is a new benchmark of 1,459 screenshots and 22,000 grounding instructions for measuring how well GUI agents transfer across app versions, platforms, and applications.
-
Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents
A real-device-centric foundation GUI agent with hybrid GUI+CLI batched actions, AutoResearch data flywheel, online RL, and a proactive harness reaches SOTA mobile and competitive desktop/web scores.
-
MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment
MobileGUI-RL applies online RL with self-generated and filtered tasks plus trajectory-level rewards to mobile GUI agents, reporting improved success rates on AndroidWorld and AITW benchmarks.
-
Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System
AI phone agents succeed on single-step commands but fail on multi-step compositions; a new benchmark measures this gap and a scheduler that decomposes tasks recovers most of it.
-
GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents
GUI-G1-3B shows that for GUI grounding, removing reasoning chains, regularizing box size in the reward, and reweighting GRPO by difficulty yields 90.3% on ScreenSpot and 37.1% on ScreenSpot-Pro.
-
EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attacks
EVA evolves environmental injection payloads through a keyword-utility loop, achieving up to 80% attack success in the body's pop-up tests, while the abstract's stronger claims are not supported by the reported experiments.
-
SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control
A multi-agent RL workflow that interleaves single-agent updates, applied to mobile GUI control, achieves SOTA zero-shot performance and a +14.8 MATH500 gain.
-
Cybernaut: Towards Reliable Web Automation
A demonstration-to-SOP framework plus robust element identification and a trace similarity metric improves enterprise web automation success rates on an internal benchmark, with a fine-tuned consistency classifier rea...
-
Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinement
A GUI agent that trims UI input with a recommendation module and asks users when decisions are ambiguous reports state-of-the-art success rates, though the interaction module is not benchmarked.
-
Evaluation and Benchmarking of LLM Agents: A Survey
A review that proposes a two-dimensional taxonomy for evaluating LLM agents and highlights enterprise-specific evaluation gaps.
Discussion (0). Sign in to comment.