REVIEW 30 cited by
OmniParser for Pure Vision Based GUI Agent
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The recent success of large vision language models shows great potential in driving the agent system operating on user interfaces. However, we argue that the power multimodal models like GPT-4V as a general agent on multiple operating systems across different applications is largely underestimated due to the lack of a robust screen parsing technique capable of: 1) reliably identifying interactable icons within the user interface, and 2) understanding the semantics of various elements in a screenshot and accurately associate the intended action with the corresponding region on the screen. To fill these gaps, we introduce \textsc{OmniParser}, a comprehensive method for parsing user interface screenshots into structured elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface. We first curated an interactable icon detection dataset using popular webpages and an icon description dataset. These datasets were utilized to fine-tune specialized models: a detection model to parse interactable regions on the screen and a caption model to extract the functional semantics of the detected elements. \textsc{OmniParser} significantly improves GPT-4V's performance on ScreenSpot benchmark. And on Mind2Web and AITW benchmark, \textsc{OmniParser} with screenshot only input outperforms the GPT-4V baselines requiring additional information outside of screenshot.
Forward citations
Cited by 30 Pith papers
-
Do GUI Agents Believe Their Eyes? Diagnosing State-Belief Reliance on Pixels versus Structure
Across four open-weight models and three OpenAI models, textual state beliefs of GUI agents follow a conflicting structured value on 30–75% of probes where the same model reads the pixels correctly, so agent state bel...
-
GUI-Lens: Coarse-to-Fine Cropping for GUI Grounding with General-Purpose VLMs
GUI-Lens improves GUI grounding by having a VLM iteratively select crops to zoom into, guided by OCR and UI detector references, and verifies its proposed clicks before outputting a coordinate.
-
Design Theater: Evaluating the Gap Between User-Facing Design Reasoning and Implementation in Generative UI Tools
Generative UI tools' stated design rationales are not fully implemented in about 25% of cases, with functional requirements failing most often.
-
TargetFinder: Detecting Widgets from Pixels on Desktop Interfaces
A fine-tuned YOLO pipeline on a new 520-screenshot, 38,000-widget dataset detects desktop GUI widgets from pixels and drives system-wide Bubble Cursor and Semantic Pointing.
-
GUI-AC: Enhancing Continual Learning in GUI Agents
GUI-AC stabilizes RFT for non-stationary GUI data by down-weighting noisy advantages and relaxing clipping bounds via a grounding certainty term.
-
GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation
A training-free Video-RAG plus inverse-dynamics pipeline that injects dual-channel tutorial knowledge into GUI agents and lifts OSWorld scores by about 4.5–7.5%.
-
Generative Visual Code Mobile World Models
gWorld converts mobile GUI world modeling into renderable HTML generation, and its fine-tuned 8B and 32B VLMs outperform frontier open-weight models up to 50x larger on next-state accuracy.
-
Dark Patterns Meet GUI Agents: LLM Agent Susceptibility to Manipulative Interfaces and the Role of Human Oversight
GUI agents frequently fall for deceptive interface designs, often without recognizing them, and human supervision of agents improves avoidance only partially while introducing new attention and workload costs.
-
AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agents
AgentSentinel combines system-level tracing with LLM-based auditing to block 79.6% of attacks in the authors' 60-scenario computer-use agent benchmark.
-
MobiAgent: A Systematic Framework for Customizable Mobile Agents
A full-stack mobile agent framework reports state-of-the-art task completion on its own DAG-based benchmark and 2-3x speedups from replaying recorded trajectories.
-
Morae: Proactively Pausing UI Agents for User Choices
Morae, a UI agent that proactively pauses at ambiguous decision points, helps blind and low-vision users complete more tasks and express preferences better than fully autonomous agents.
-
PG-Agent: An Agent Powered by Page Graph
An MLLM GUI agent that stores past episodes as a page graph and retrieves action guidelines from it improves step success on three benchmarks.
-
Phi-Ground Tech Report: Advancing Perception in GUI Grounding
Phi-Ground models achieve state-of-the-art click accuracy on five GUI grounding benchmarks for models under 10B parameters using a 40M-sample training recipe with text-first inputs, random-resize augmentation, uniform...
-
Screen2AX: Vision-Based Approach for Automatic macOS Accessibility Generation
Screen2AX generates hierarchical macOS accessibility metadata from a screenshot and reports improved GPT-4 UI task success compared with native accessibility and OmniParser V2.
-
GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents
An attention-based action head with multi-patch supervision outperforms coordinate-generation baselines on GUI grounding, and a verifier further improves accuracy.
-
Agent-Environment Alignment via Automated Interface Generation
ALIGN automatically generates and verifies an interface wrapper that enriches environment observations, improving LLM agent success rates by up to 45.67 percentage points on ALFWorld without modifying the agent or env...
-
Robot Operation of Home Appliances by Reading User Manuals
A robot system that constructs a symbolic appliance model from a user manual and uses it to reliably execute natural language appliance operation tasks, outperforming direct VLM-based policies.
-
TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments
TransBench is a new benchmark of 1,459 screenshots and 22,000 grounding instructions for measuring how well GUI agents transfer across app versions, platforms, and applications.
-
FullFront: Benchmarking MLLMs Across the Full Front-End Engineering Workflow
FullFront adds a three-task benchmark for webpage design, perception, and code generation, and finds top MLLMs still fail at fine-grained layout and interaction implementation.
-
Tactile: Giving Computer-Using Agents Hands and Feet
Adding Tactile, an MCP tool layer that grounds agent actions in macOS accessibility semantics, OCR, and visual fallback, raised Codex Success@100 from 41.1% to 50.0% on macOSWorld-style tasks.
-
Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills
Mirage-1 combines a hierarchical multimodal skill memory with a skill-augmented Monte Carlo tree search to outperform prior GUI agents on Android and web online benchmarks.
-
Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System
AI phone agents succeed on single-step commands but fail on multi-step compositions; a new benchmark measures this gap and a scheduler that decomposes tasks recovers most of it.
-
GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents
GUI-G1-3B shows that for GUI grounding, removing reasoning chains, regularizing box size in the reward, and reweighting GRPO by difficulty yields 90.3% on ScreenSpot and 37.1% on ScreenSpot-Pro.
-
EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attacks
EVA evolves environmental injection payloads through a keyword-utility loop, achieving up to 80% attack success in the body's pop-up tests, while the abstract's stronger claims are not supported by the reported experiments.
-
Interfaze: The Future of AI is built on Task-Specific Small Models
Interfaze-Beta uses small specialist models and tools to build a compact context that a general-purpose LLM answers from, reporting competitive benchmark scores without reproducible evidence.
-
Cybernaut: Towards Reliable Web Automation
A demonstration-to-SOP framework plus robust element identification and a trace similarity metric improves enterprise web automation success rates on an internal benchmark, with a fine-tuned consistency classifier rea...
-
VLM-3D:End-to-End Vision-Language Models for Open-World 3D Perception
The paper promises VLM-3D but the body text is entirely the MVISU-Bench mobile-agent benchmark paper, so the stated result is unsupported.
-
GUI-G$^2$: Gaussian Reward Modeling for GUI Grounding
Modeling GUI elements as Gaussian distributions instead of binary targets yields 92.0% (ScreenSpot), 93.3% (ScreenSpot-v2), and 47.5% (ScreenSpot-Pro) for a 7B model, outperforming UI-TARS-72B by a relative 24.7% on t...
-
GenFlow: Interactive Modular System for Image Generation
GenFlow combines a node-based editor, retrieval-augmented workflow search, and web-exploration agents to simplify Stable Diffusion image-generation workflows, with a small user study reporting reduced task times and p...
-
DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning
Separating text and icon grounding with iterative zooming improves GUI-element localization accuracy of existing vision-language models without retraining.
Discussion (0). Sign in to comment.